How large of an XML string?

[ids]

How large can an XML string/file be for a Provision_Store function without timing out in some way?

Scott

[rbuelna]

How large can an XML string/file be for a Provision_Store function without timing out in some way?

Scott

Hi Scott,

I don't believe there is a "one size fits all" answer to this. While there are some default timeouts set on our servers they're not all the same across the board. Then you have some sites that use a CDN like Cloudflare that also plays a factor in timeout issues. Are you running into a timeout now on site? What is the exact error you're seeing and how long does the provision file run for before it times out?

Warm Regards

[Adam Christianson]

Scott,
One way you can get around request timeouts would be to run the import via a Scheduled Task and using the "Standard Scheduled Task: Import" operation with the import operation set to "Import Data and Settings in XML Provisioning Format". You'll need to upload your provisioning file via FTP and for the filename option provide the path to the file you uploaded relative to the "mm5" directory. So, for example, if you uploaded your provisioning file to the "private/mivadata/Merchant5/s01/import/" directory from the root of your server then the filename would be "../../private/mivadata/Merchant5/s01/import/provisioning.xml" (if you named the import file "provisioning.xml"). Hope this helps.

[ids]

Hey Rafael, Thanks.

I am trying to decide if I should be using XML instead JSON API function based.

The pre-requisite is that I am grabbing data from a 3rd party source. I can only grab 100 records at a time. Then I can process the data and combine into 1 string or run 100 record batches. Therefore, with 4000 records I need to MvCall 40 times. That results in 440 in the batch method because there is one call to json.mvc per 100 records.

It's a custom module. When I created a single JSON API string, it timed out. A support ticket response told me that 2k was roughly the max based on engineering dev benchmarking. 100 records is a natural breakpoint. This still times out after roughly 3800 records. I changed to running 1 record at a time. I have similar results.

The next phase of the project, is products. Look like there is roughly 8-10K.

Both phases are for updating or inserting new records in the Miva tables.

I need some guidance to overcome the time out issues. Maybe a way to reset the session(s)?

Also, I changed to using the standard API calls to category_insert and category_update. I am receiving a gateway timeout. Looks like after about 600 records. The only method I haven't tried yet is XML.

Scott

[ids]

Scott,
One way you can get around request timeouts would be to run the import via a Scheduled Task and using the "Standard Scheduled Task: Import" operation with the import operation set to "Import Data and Settings in XML Provisioning Format". You'll need to upload your provisioning file via FTP and for the filename option provide the path to the file you uploaded relative to the "mm5" directory. So, for example, if you uploaded your provisioning file to the "private/mivadata/Merchant5/s01/import/" directory from the root of your server then the filename would be "../../private/mivadata/Merchant5/s01/import/provisioning.xml" (if you named the import file "provisioning.xml"). Hope this helps.

Thanks Adam. Parts of this concept may be the solution. Scheduled Tasks are run in an Async wrapper. I've used this already manually/independently -- not in a scheduled task. It's worked for me in previous modules I've built. Until this one that is. There is much more data here. IMO it's probably that I need to use MANY MvCalls that eats up the timing. The caveat, in other modules where I've used scheduled tasks, I've been told even Async has a timeout.