Webmaster Tools: Increase in server errors

Darter02 · June 27, 2014

I'm seeing a large spike in errors. About 1700 of them as of yesterday. 504 errors.

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

tomoscott · June 27, 2014

I got the same message - 200 errors in one day!

KQuinlan · June 27, 2014

Same here.. usually have 1 or 2 server errors a day from google bot... but today google detected 1336 server errors on my site while attempting to crawl.

richW · June 27, 2014

Could one or all of you post back one of the error messages you're receiving?

ChancyRat · June 27, 2014

I got it just now as well:

Increase in server errors

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.
Recommended action

Check the Crawl Errors page in Webmaster Tools.
Check your scripts and script permissions.
Examine the log files on the server for your site for scripts or pages that might be crashing.
Consider addressing the load on your server.

ChancyRat · June 27, 2014

Link takes me to:

https://support.google.com/webmasters/answer/2409437?hl=en

ChancyRat · June 27, 2014

Sorry, don't know what to look for so posting stuff I find:

504 Error

Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request.

ChancyRat · June 27, 2014

And the server errors went from 3 per day, to 35 on 6/26, to 1,950 so far today (6/27/14).

richW · June 27, 2014

richW wrote: »

Could one or all of you post back one of the error messages you're receiving?

We changed CDN providers and Google is crawling us a lot more aggressively. Over time it should calm down.
I just received the same email from Google. I ran the 'Fetch as Google' and received the same errors. After about 15 minutes I ran the 'Fetch as Google' again and received the Green Checkmark.
I went from HTTP/1.1 504 Gateway Time-out to HTTP/1.1 200 OK

I've contacted our Operations Team and they'll be looking into the Google crawl errors too.

ChancyRat · June 28, 2014

richW wrote: »

We changed CDN providers and Google is crawling us a lot more aggressively. Over time it should calm down.
I just received the same email from Google. I ran the 'Fetch as Google' and received the same errors. After about 15 minutes I ran the 'Fetch as Google' again and received the Green Checkmark.
I went from HTTP/1.1 504 Gateway Time-out to HTTP/1.1 200 OK

I've contacted our Operations Team and they'll be looking into the Google crawl errors too.

Rich, thanks for - er, whatever you said. lol, this is what I heard:

"Blah blah science jargon blah blah calm fetch green blah will check."

Are we supposed to do anything?

LPC · June 28, 2014

Me too -

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Recommended action
Check the Crawl Errors page in Webmaster Tools.
Check your scripts and script permissions.
Examine the log files on the server for your site for scripts or pages that might be crashing.
Consider addressing the load on your server.
Learn more in our Help Center.

AperturePlus · June 28, 2014

I received the same mail - 200 errors.

When I click on one of the errors, I get "Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request."

RobLoud · June 28, 2014

richW wrote: »

Could one or all of you post back one of the error messages you're receiving?

I have had the same notification..... My errors have jumped from around 25 (changed folder/gallery descriptions) to nearly 500 within the last week.

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Recommended action
Check the Crawl Errors page in Webmaster Tools.
Check your scripts and script permissions.
Examine the log files on the server for your site for scripts or pages that might be crashing.
Consider addressing the load on your server.
Learn more in our Help Center.

essex1970 · June 28, 2014

same here:

- 788 server errors
- 88 not found
- fetching is either redirected or unreachable
- /sitemap-galleryimages.xml got warning, some URLS are unreachable.

pabloconrad · June 28, 2014

Good to know Smug is working on this.

I had over 600. Usually have none. Every link I checked works and verified via Google.

I just marked all as fixed.

pabloconrad · June 28, 2014

On a good note:

Found a typo in one of my URLs and fixed it.

bwg · June 28, 2014

Like richW said, Google is hitting us really hard right now after changing CDN providers. We throttle the traffic to bots so they don't take down the rest of the site. This will resolve itself over time, there is nothing you need to do/worry about.

AperturePlus · June 29, 2014

Thanks bwg - good to know.

What does confuse me is that some are really strange errors, like: http://www.apertureplus.com/keyword/_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;1;havelock/

I am just trying to understand Webmaster Tools - where do these searches come from?

rainforest1155 · June 30, 2014

That looks more like a link formatting issue in Google. If it's indeed the link Google attempted to access, we don't have a way to tell where they got it from.

AperturePlus · June 30, 2014

Thanks Sebastian

Many, if not most of the errors, are in the form as per my above post, including many '1's... like

http://www.apertureplus.com/keyword/_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;2;agra/

Do we just ignore them?

shandrew · July 4, 2014

Google's bot/crawler problems were resolved on 6/30 8 PM Pacific. However, due to how google webmaster tools lags in how it reports traffic (especially when it hits very large traffic levels, as it did here), it will continue to report high error levels for several days until it has completed processing the data from prior to 6/30 8 PM Pacific--for some reason it records the data under the date it was processed rather than the date when the events occurred.

Unless you are very interested in google's bot/crawler activity itself, I recommend turning off Google Webmaster Tools' email notifications, as they simply aren't terribly useful--they are not designed well for large scale sites like SmugMug's. If you want free and simple external site monitoring, you can try a service like pingdom or siteuptime.

Darter02 · July 11, 2014

Six days after your post (07/10) I'm still getting almost 6,000 errors/day. How long should this take again?

thenickdude · July 11, 2014

Just got this warning from Google Webmaster Tools:

http://photos.chickensmoothie.com/: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 58 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 81.7%.

Our other subdomains show no DNS errors, so the error seems to be due to SmugMug's DNS.

mbonocore · July 12, 2014

Darter02 wrote: »

Six days after your post (07/10) I'm still getting almost 6,000 errors/day. How long should this take again?

Hard to say...I wish we knew all the details of how Google does what they do, but of course, it's not really public knowledge. It will calm down over time, but remember, this doesn't have any effect on actual viewers. They should not be experiencing anything different.

elizab · July 14, 2014

Same problem here. Still showing ca 1500 errors per day, two and a half weeks after the problem first started. While it doesn't affect people wanting to access our sites presumably it does affect google bots crawling our sites and therefore our search rankings?

www.madaboutgreys.com

RobLoud · July 14, 2014

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?
I initially had a high errors and even after the fix they peaked even more. I marked all of my errors as fixed and within a couple of days my errors had then gone back down to zero and I have had no issues since.

Robin

denisegoldberg · July 14, 2014

RobLoud wrote: »

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?
I initially had a high errors and even after the fix they peaked even more. I marked all of my errors as fixed and within a couple of days my errors had then gone back down to zero and I have had no issues since.

Thanks for that tip Robin - I've now done the same and will check the results in a few days.

--- Denise

RobLoud · July 14, 2014

denisegoldberg wrote: »

Thanks for that tip Robin - I've now done the same and will check the results in a few days.

--- Denise

I hope it works for you and others. Please post back in a few days to let us know whether it works or not.

Robin

AperturePlus · August 10, 2014

It seems to be happening again, to me at least.

I have just received a notification from Google of an increase in server errors.

Anyone else?
Clive

Darter02 · August 10, 2014

RobLoud wrote: »

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?

Robin

OK, I tried that. It took a few days of marking things fixed to get to zero. They only show the first 1000 errors, so you have to wait to see the next 1000. It looked fine until today. I'm upwards of 5K errors again. It looks like the same keyword URL errors as before.

chipj · August 10, 2014

Yes, I'm seeing the same. In fact this time I have more than twice the number of reported errors than the June 27th incident. They all appear to 504 errors.

AperturePlus wrote: »

It seems to be happening again, to me at least.

I have just received a notification from Google of an increase in server errors.

Anyone else?
Clive

Webmaster Tools: Increase in server errors

Comments