The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It's also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also...