Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.
Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi
Alleged action follows a lawsuit filed by the online bulletin board to prevent data harvester 3taps and apartment listings aggregator PadMapper from exporting its listings.
The search giant injects a little Halloween humor into its robots.txt file's instructions for Web-crawling software.
President Obama's new White House Web site has been lauded for being more open than former President Bush's. There's just one problem with that theory: it's wrong.
Whitehouse.gov's administrators silently triple the number of Web pages that it forbids Google and other search engines from accessing. Is this a bad omen or much ado about nothing?
update After Google seemingly ignored restrictive search engine-blocking code built into the Obama administration's new stimulus-related site, the robots.txt code is removed.
On Thursday, the White House ceased to block search engines from its Iraq-related directories. Now the National Intelligence director is doing the same thing in response to queries from News.com.
In response to questions from News.com, the Bush administration says blocking search engines from indexing Iraq-related documents was a simple mistake and alters the White House Web site.