Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.
Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi
update After Google seemingly ignored restrictive search engine-blocking code built into the Obama administration's new stimulus-related site, the robots.txt code is removed.
The search giant injects a little Halloween humor into its robots.txt file's instructions for Web-crawling software.