Google Crawl times are getting faster and faster so while even a year ago it would be normal to wait a week or more for Google to index any part of your site now it takes just a few days that said it still can take several crawls before the whole site is indexed. Once a site has been crawled it is even quicker for re crawls.
To speed up the process the key is to prepare the site
- Make sure the site is crawlable, if you have used robots.txt or htaccess files validate them
- While there is some merits to using an xml sitemap for the first crawl, make sure it is correct and validated and mentioned in your robot.txt file.
- Key content is easily found through multiple internal links within navigation
- Navigation is crawlable and not some strange flash based menu
- Content that you do not wish crawled no-followed and is not included in the sitemap
- If RSS feed is available make sure it only includes pages you want crawled
Sitemap
The use of XML sitemaps is a bit contentious I use them as a matter of course but only for the first crawl after which I drop the file. Within the file I only include the first 100 key pages within the site, Google does appear to prioritise pages found in sitemaps within a crawl. Don’t rely on this but it’s useful to know.
Once you have the site ready its time to get search engines visiting, don’t submit a site via search submission simply select several locations to point inbound links to it, popular sites with high crawl rates such as forums are preferable. I tend to build links at about 10 at a time which is far slower then most. You may also wish to verify your site with Google Webmaster Console and submit your sitemap there.
Finally you could go all Bluehat and submit your sit through QUIT Eli rather handy tool it has lost some of its speed in recent months but is still a great little tool.
- More Information
- Did that answer your question? Get a personalised answer from Tim today for just £20
- More Questions
- Find and ask questions in Tims Question and answer section





