Announcement

#2 · 11-22-04, 09:20 PM

googlebot gone wild

It has been living on my dedicated server at Hostasaurus for almost the
entire month of November.

Unfortunately it's a catch 22. Block the IP or Not?

-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Eric Husby - HigherPowerSupplies.com
Sent: Monday, November 22, 2004 9:14 PM
To: [email protected]
Subject: [mru] googlebot gone wild

Anyone else having problems with googlebot slowing a server down by
spidering 3-6 pages per second?
<A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.ht">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.ht</A>
ml

It is really impacting our site performance over the past 2 days.

Eric

#3 · 11-22-04, 09:30 PM

googlebot gone wild

Supposedly, msn and yahoo support crawl delay through the robots.txt file:
User-Agent: msnbot
Crawl-Delay: 20

I wish google supported this right now. I just found this on google.com and
submitted my issue, I'll lyk what response I get:
8. Help! Googlebot is crawling my site too fast. What can I do?

Please send an email to [email protected] with the name of your site and
a detailed description of the problem. Please also include a portion of the
weblog that shows Google accesses, so we can track down the problem more
quickly on our end.

----- Original Message -----
From: "Jon Viscott" <[email protected]>
To: "'Eric Husby - HigherPowerSupplies.com'" <[email protected]>;
<[email protected]>
Sent: Monday, November 22, 2004 11:20 PM
Subject: RE: [mru] googlebot gone wild

> It has been living on my dedicated server at Hostasaurus for almost the
> entire month of November.
>
> Unfortunately it's a catch 22. Block the IP or Not?
>
>
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Eric Husby - HigherPowerSupplies.com
> Sent: Monday, November 22, 2004 9:14 PM
> To: [email protected]
> Subject: [mru] googlebot gone wild
>
> Anyone else having problems with googlebot slowing a server down by
> spidering 3-6 pages per second?
> <A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.ht">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.ht</A>
> ml
>
> It is really impacting our site performance over the past 2 days.
>
> Eric
>
>

#4 · 11-22-04, 09:31 PM

googlebot gone wild

--=====================_15533425==.ALT
Content-Type: text/plain; charset="us-ascii"

Send a message to Google (see "Googlebot is crawling my site too fast"):
<A HREF ="http://www.google.com.au/webmasters/3.html#B9">http://www.google.com.au/webmasters/3.html#B9</A>

Many sites are currently having problems, not just Miva stores.

HTH

Paul

At 04:14 PM 23/11/2004, you wrote:
>Anyone else having problems with googlebot slowing a server down by spidering 3-6 pages per second?
><A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html</A>
>
>It is really impacting our site performance over the past 2 days.

-----------------------------------------------------------------
How do your pages rate, for search engines?
<http://internetmarketingengine.com/analyzer/analyzer-lite.php>Analyze your pages and those of your competitors

Paul McMahon
Web: http://internetmarketingengine.com/

--=====================_15533425==.ALT--

#5 · 11-22-04, 11:01 PM

googlebot gone wild

You could place your merchant screens in a no robot zone and use a product such as
Merchant Optimizer (or others like SEK) to catch the search engine traffic.

www.phosphormedia.com/go.mv?ID=MORESPEED

-Bruce
......................................
PHOSPHOR Media
Featuring: StoreMan - A smarter way to manage Miva Merchant(tm) Store Content.
Download a hassle-free demo today
<A HREF ="http://www.phosphormedia.com/go.mv?ID=listsm">http://www.phosphormedia.com/go.mv?ID=listsm</A>
......................................

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of PMV McMahon
> Sent: Monday, November 22, 2004 9:31 PM
> To: [email protected]
> Subject: Re: [mru] googlebot gone wild
>
>
> Send a message to Google (see "Googlebot is crawling my site too fast"):
> <A HREF ="http://www.google.com.au/webmasters/3.html#B9">http://www.google.com.au/webmasters/3.html#B9</A>
>
> Many sites are currently having problems, not just Miva stores.
>
> HTH
>
>
>
> Paul
>
>
>
>
> At 04:14 PM 23/11/2004, you wrote:
> >Anyone else having problems with googlebot slowing a server down by spidering
> 3-6 pages per second?
> ><A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html</A>
> >
> >It is really impacting our site performance over the past 2 days.
>
>
>
> -----------------------------------------------------------------
> How do your pages rate, for search engines?
> <http://internetmarketingengine.com/analyzer/analyzer-lite.php>Analyze your
> pages and those of your competitors
>
> Paul McMahon
> Web: http://internetmarketingengine.com/
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (<A HREF =/"http://www.grisoft.com)/">http://www.grisoft.com)</A>.
> Version: 6.0.788 / Virus Database: 533 - Release Date: 11/1/04
>
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (<A HREF =/"http://www.grisoft.com)/">http://www.grisoft.com)</A>.
Version: 6.0.788 / Virus Database: 533 - Release Date: 11/1/04

#6 · 11-23-04, 12:17 AM

googlebot gone wild

--=====================_6484546==.ALT
Content-Type: text/plain; charset="us-ascii"

A Google rep has specifically asked for site owners/webmasters with problems with the current fast spidering to email Google at the address given here: <A HREF =/"http://www.google.com.au/webmasters/3.html#B9 and have indicated Google will take appropriate action. I should have said that in my previous post/">http://www.google.com.au/webmasters/3.html#B9 and have indicated Google will take appropriate action. I should have said that in my previous post</A>.

Paul

At 06:01 PM 23/11/2004, you wrote:
>You could place your merchant screens in a no robot zone and use a product such as
>Merchant Optimizer (or others like SEK) to catch the search engine traffic.
>
>www.phosphormedia.com/go.mv?ID=MORESPEED
>
>
>-Bruce
>......................................
>PHOSPHOR Media
>Featuring: StoreMan - A smarter way to manage Miva Merchant(tm) Store Content.
>Download a hassle-free demo today
><A HREF ="http://www.phosphormedia.com/go.mv?ID=listsm">http://www.phosphormedia.com/go.mv?ID=listsm</A>
>......................................
>
>
>
>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]]On Behalf Of PMV McMahon
>> Sent: Monday, November 22, 2004 9:31 PM
>> To: [email protected]
>> Subject: Re: [mru] googlebot gone wild
>>
>>
>> Send a message to Google (see "Googlebot is crawling my site too fast"):
>> <A HREF ="http://www.google.com.au/webmasters/3.html#B9">http://www.google.com.au/webmasters/3.html#B9</A>
>>
>> Many sites are currently having problems, not just Miva stores.
>>
>> HTH
>>
>>
>>
>> Paul
>>
>>
>>
>>
>> At 04:14 PM 23/11/2004, you wrote:
>> >Anyone else having problems with googlebot slowing a server down by spidering
>> 3-6 pages per second?
>> ><A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-say.html</A>
>> >
>> >It is really impacting our site performance over the past 2 days.
>>
>>
>>
>> -----------------------------------------------------------------
>> How do your pages rate, for search engines?
>> <http://internetmarketingengine.com/analyzer/analyzer-lite.php>Analyze your
>> pages and those of your competitors
>>
>> Paul McMahon
>> Web: http://internetmarketingengine.com/
>>
>> ---
>> Incoming mail is certified Virus Free.
>> Checked by AVG anti-virus system (<A HREF =/"http://www.grisoft.com)/">http://www.grisoft.com)</A>.
>> Version: 6.0.788 / Virus Database: 533 - Release Date: 11/1/04
>>
>---
>Outgoing mail is certified Virus Free.
>Checked by AVG anti-virus system (<A HREF =/"http://www.grisoft.com)/">http://www.grisoft.com)</A>.
>Version: 6.0.788 / Virus Database: 533 - Release Date: 11/1/04
>
>

#7 · 11-23-04, 12:20 PM

googlebot gone wild

Yep google did that to us on Saturday.
I love having our sites deep spidered... But the new googlebot spider is
pretty obnoxious.
If it weren't for the fact it was coming from the google ip block... I would
have suspected it was a fake googlebot.
However, I'd rather be spidered than not...

Kelly

-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of PMV McMahon
Sent: Monday, November 22, 2004 11:31 PM
To: [email protected]
Subject: Re: [mru] googlebot gone wild

Send a message to Google (see "Googlebot is crawling my site too fast"):
<A HREF ="http://www.google.com.au/webmasters/3.html#B9">http://www.google.com.au/webmasters/3.html#B9</A>

Many sites are currently having problems, not just Miva stores.

HTH

Paul

At 04:14 PM 23/11/2004, you wrote:
>Anyone else having problems with googlebot slowing a server down by
spidering 3-6 pages per second?
><A HREF ="http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-s">http://www.webstractions.com/news/2004/11/googlebot-hitting-too-heavy-s</A>
>ay.html
>
>It is really impacting our site performance over the past 2 days.

-----------------------------------------------------------------
How do your pages rate, for search engines?
<http://internetmarketingengine.com/analyzer/analyzer-lite.php>Analyze your
pages and those of your competitors

Paul McMahon
Web: http://internetmarketingengine.com/

Announcement

googlebot gone wild

googlebot gone wild

Comment

Comment

Comment

Comment

Comment

Comment