What is New with Google Search Spider?
Google’s Googlebot and other search engine spiders crawl websites frequently and add webpages to search index. You need to submit only the homepage of a website and the spider will index all pages by following the links provided in homepage.
Frequency of crawling a website depends on search engine and their infrastructure. Webmasters do not have an option to ping the search engines to tell that a webpage is changed and need to be re-indexed. Well, that was the past, at least for Google.
Google now provides a protocol for the webmasters to submit links of frequently changing webpages. This is called Google Sitemap. If you have dynamic content in your site, you can update the sitemap with new links and google will index them at the earliest.
I use Google sitemap.xml to let Google know of new pages added in news section or message boards in Ananthapuri.com website. Sometimes it is good idea to have our own program to update the XML when database contents changes. For example, I might schedule a 2-hourly job to check for newly added discussion forum threads or updated forums and ping Google to let know of changes in sitemap.xml.
Be the first to comment