When your website gets indexed by Google it will be added to Google’s regular crawling. This means that Google will load pages from your site at a regular interval. If your website is popular (high traffic and many inbound links) it will be crawled more often than if it is seen as a less important site.
Google tries to adjust the crawling to hours to when it’s easier to access your site. If your site is seen as unreliable by Google you can be punished by losing positions in the search engine result pages. “Unreliable” can mean that your website is slow or that Google has had several issues with indexing your website.
If your site is completely down or responds with a server error page, Google will come back later and try again. After several attempts they will however give up and remove you from their index. In other words, you lose all the traffic you get from searches in Google.
Above: Graph showing how organic search traffic from Google can be affected if your site is dropped from the search index due to downtime.
Matt Cutts, a Google employee, and Vanessa Fox, a former Google employee, have said the following regarding Google indexing and downtime:
“Googlebot will try a few times before the pages drop from the index.” – Vanessa Fox
“As for how long it takes for a page to get back in once the site is back up, that really depends on a number of factors, such as how often the site is crawled in general.” – Vanessa Fox
“Sometimes temporarily down pages turn into truly-gone-forever pages, so we have to drop those pages at some point. But it’s also true that we go back and revisit those pages pretty often and try to recrawl them in case the site comes back up.” – Matt Cutts
It seems that depending on the size of your website, luck (!) and the type of issues you are having, anything can happen between nothing and being completely removed from Google’s index.
From all the discussions about this topic you can only come to one conclusion: uptime is very important if you don’t want to rely on pure luck.
What can I do?
-
- Monitor your uptime and quickly take action if your site goes down. You can use any uptime monitoring service that can send you alerts, for example website monitoring tools at Pingdom.
- Monitor and test your response time at regular intervals. In addition to monitoring services, you can use our free Pingdom Tools to test your site’s total load time.
- If you need to take down your site, use the right server error codes. Don’t just put up a regular page explaining the maintenance since this might get indexed instead of your real site.
- Use Google Webmaster Tools and check that no pages are listed in the crawl errors section.
- Beware of expiring domains, hosts that show informational pages if you exceed your bandwidth or any other occurrence that will alter your website in an unwanted way.
