Automating Template Data Feeds for Google Base using cron

[wcw]

You can run the Template Data Feed (TDF) module from a cron task without you having to log into admin. This is useful for running routine exports during slow hours at night.

Several have asked for instructions on how to do this. You must have access to the cron tab in your account so that you can create tasks which run at specific intervals. I suspect very few hosts give you access to this without having to ask for it. The host may not even allow it if you are on a shared server. What little I know about cron tasks came from David and Jen. So if you cannot get this to work, I can't help you any further than what I have posted below. The two commands below may be wrapped in the post. They are single line commands. You can also have a status of the task running sent to your email address.

This won't work on large stores because the wget command will not use the javascript needed to refresh the screen (which avoids timeouts). So before you setup the cron, you have to test the export from admin to see if it will finish before timeout. In the TDF admin there is an input "Override clock and increment every [ ] records" Set that number to be higher than the number of records in your store. The max is 9999. If it completes the export without having to refresh the screen, you are in luck.

This works for Merchant 4 and 5.

Example for data feed export:

/path/to/wget --post-data='Screen=SUTL&Action=SUTL&Store_Code=YOURSTORECODE&Module_Code=exprodf&UserName=ADMINLOGIN&Password=ADMINPASSWORD&Provision=yes&Submit=Login&ExProdF_report=REPORTNUMBER&ExProdF_Custom_Fields=1&ExProdF_UI=YOURUSERINTERFACEMODULE' https://www.YOURDOMAIN.com/MERCHANTDIRECTORY/admin.mvc --no-check-certificate

Change Store_Code, UserName, Password, REPORTNUMBER, user interface module, path to your admin.mvc.

ExProdF_UI will be either modules/ui/mmui.mvc or modules/ui/oui.mvc depending on your user interface module.

The report number can be found by going to admin to run the export. With the drop down showing your exports, eg Froogle, Google sitemap XML, Yahoo, etc, view the source in that frame. You will see a value in the options for the various exports. Select the number of the export you want for this cron task.



Example for FTP to Google Base:
/PATHTO/ncftpput -u GOOGLEBASEFTPLOGIN -p GOOGLEBASEFTPPASSWORD uploads.google.com / /PATHTO/YOURFILE.txt

You have to get a FTP login and password at Google Base. It is different from your Google Base screen uploads. Sign into your Google Base account and look in the right column for FTP information.

The path at the beginning of this command for the ncftpput program is specific for your server.

The path at the end of this command for the txt file in your root documents directory will be specific for your server.

[narmin]

Re: Automating Template Data Feeds for Google Base using cron

You Posted

wget --post-data='Screen=SUTL&Action=SUTL&Store_Code=YOURSTORE CODE&Module_Code=exprodf&UserName=ADMINLOGIN&Passw ord=ADMINPASSWORD&Provision=yes&Submit=Login&ExPro dF_report=REPORTNUMBER&ExProdF_Custom_Fields=1&ExP rodF_UI=YOURUSERINTERFACEMODULE' https://www.yourdomain.com/MERCHANTDIRECTORY/admin.mvc --no-check-certificate

I adjusted the line to meet my domains needs but then I didn't know what to do with it. Am I supposed to put it in a special named file in a special folder?

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

You setup a cron job in your domain. Your panel (not merchant) will prompt you to enter the command line when you do.

[John]

Re: Automating Template Data Feeds for Google Base using cron

How can I keep this from outputting the log file into the root of my domain's folder? I tried omitting the report variable but I guess that does something else. I've also tried 2>&1 to send the output to null which kills the email set up on the cron but I still keep getting the admin.mvc.x log files under /. I just need to shut them off so the server doesn't fill up. Otherwise it's working great!

My cron command:
wget --post-data='Screen=SUTL&Action=SUTL&Store_Code=****&Modu le_Code=exprodf&UserName=****&Password=****&Provis ion=yes&Submit=Login&ExProdF_Custom_Fields=1&ExPro dF_UI=modules/ui/oui.mvc' https://www.****.com/Merchant2/admin.mvc --no-check-certificate

What is ExProdF_report anyway?

Thanks,
John

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

You'll have to ask your host how to turn off the log.

Your cron command won't do anything because you have not defined which report to run.

As above, "The report number can be found by going to admin to run the export. With the drop down showing your exports, eg Froogle, Google sitemap XML, Yahoo, etc, view the source in that frame. You will see a value in the options for the various exports. Select the number of the export you want for this cron task."

In mine the google base is 7. In yours it might be 1 if that is the first export in the list.

[John]

Re: Automating Template Data Feeds for Google Base using cron

It must be defaulting to 1 then because at first I did follow your instructions and the report number is 1. Now, I have &ExProdF_report=1 removed and I am still making my froogle.txt under public_html of my domain.

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

Are you sure it is a new one and not the original one it created?

[John]

Re: Automating Template Data Feeds for Google Base using cron

Yes sir. I removed it to test that theory.

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

Then it is defaulting to the first one.

[John]

Re: Automating Template Data Feeds for Google Base using cron

I think so. I'm still on the hunt for a solution to turning off the admin.mvc log under /. I'm assuming it's created by wget where there are quite a few options. If it is made by wget I don't know where it gets the command to make a log with what we have here. I tried "-o --" without the qoutes and ended making two logs. Thre new one being named -- plus the admin.mvc. lol Have a few more ideas, I'll post back if and when I figure it out.

[John]

Re: Automating Template Data Feeds for Google Base using cron

YES! I went through almost all of the switches and `--spider` turns off the admin.mvc log file under / of the domain! `-a` by itself turned off both files - undesirable. Now I have what I need.

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

After you get it working, look at the end of the line and change it to read
--no-check-certificate >/dev/null 2>&1
This last part after the word "certificate" suppresses the sending of an email. The email that gets sent could contain your login and password info in the subject so this line kills the email.

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

If you are using PR7 or newer of Merchant you need to add another parameter to the url. Just after the REPORTNUMBER put this name value pair &include_status=A This will export only active records.

[ILoveHostasaurus]

Re: Automating Template Data Feeds for Google Base using cron

For CSSUI stores do you just need to change the parameters to modules/ui/cssui.mvc or will that not work?

I've got a store I set the job up for using this thread's instructions and when I run the job, I immediately get the following html code as the result instead of the export running:

<INPUT TYPE="hidden" NAME="next_cat" VALUE="1">
<INPUT TYPE="hidden" NAME="FlatLink_File" VALUE="">
<INPUT TYPE="hidden" NAME="total_products" VALUE="10">
<INPUT TYPE="hidden" NAME="total_checked" VALUE="11">
<INPUT TYPE="hidden" NAME="export_count" VALUE="10">
<INPUT TYPE="hidden" NAME="Screen" VALUE="SUTL">
<INPUT TYPE="hidden" NAME="Action" VALUE="SUTL">
<INPUT TYPE="hidden" NAME="Store_Code" VALUE="T">
<INPUT TYPE="hidden" NAME="Module_Code" VALUE="exprodf">
<INPUT TYPE="hidden" NAME="next_prod" VALUE="11">
<INPUT TYPE="hidden" NAME="ExProdF_report" VALUE="3">
<INPUT TYPE="hidden" NAME="ExProdF_Custom_Fields" VALUE="1">
<INPUT TYPE="hidden" NAME="ExProdF_CustomImage_Fields" VALUE="">
<INPUT TYPE="hidden" NAME="newest" VALUE="">
<INPUT TYPE="hidden" NAME="limit" VALUE="99999999">
<INPUT TYPE="hidden" NAME="first" VALUE="1">
<INPUT TYPE="hidden" NAME="last" VALUE="3536">
<INPUT TYPE="hidden" NAME="nextrow" VALUE="22">
<INPUT TYPE="hidden" NAME="firstid" VALUE="1">
<INPUT TYPE="hidden" NAME="lastid" VALUE="4071">
<INPUT TYPE="hidden" NAME="totcats" VALUE="0">
<INPUT TYPE="hidden" NAME="totall" VALUE="1611">
<INPUT TYPE="hidden" NAME="catsdone" VALUE="">
<INPUT TYPE="hidden" NAME="send2email" VALUE="">
<INPUT TYPE="hidden" NAME="include_status" VALUE="A">
</FORM><SCRIPT LANGUAGE = "JavaScript">
<!--
function Product_Continue()
{document.More_Products.submit();
}
//-->
</SCRIPT>
<script language="JavaScript">
<!--
var HaveLoadFinishedHandler = 1;

function LoadFinishedHandler()
{
        if ( parent.LeftNavigation )
        {
                parent.LeftNavigation.location.reload();
        }
}
//-->
</script>
</html>

The job I'm using is:

wget --post-data='Screen=SUTL&Action=SUTL&Store_Code=T&Module_Code=exprodf&UserName=USER&Password=PASS&Provision=yes&Submit=Login&ExProdF_report=1&include_status=A&ExProdF_Custom_Fields=1&ExProdF_UI=modules/ui/cssui.mvc'

So I assume one of those hidden values is something I need to account for?

[wcw]

Re: Automating Template Data Feeds for Google Base using cron

That is the javascript form which the wget cannot use. You have to set the override clock in the feed setup to be higher than the number of products in the store. Using the examples in this thread you are constrained to the number of products that can be exported in one pass. If it is a small store, e.g. 3000 products or less, you should be ok. But many are larger. Hence, I implemented the suggestion that you steered me towards and included it in the TDF this past weekend. The "how to" is at http://www.emporiumplus.com/v5/tdfcron.txt They will need the latest version of the TDF and also the Emporium Plus Tool Kit.