Here are the basics of how website content is crawled and indexed, plus some great ways to get the Googlebot to your website or blog to index your content sooner than later. ... each page on your website. It tells search engines when new pages have been added and how often to check back for changes on specific pages.