4 Results for

robots.txt

Article

Analyze, create robots.txt files in Google

Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

By March 29, 2008

Article

Feds use robots.txt files to stay invisible online. Lame.

Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi

By August 24, 2007

Article

Recovery.gov blocked search engine tracking

update After Google seemingly ignored restrictive search engine-blocking code built into the Obama administration's new stimulus-related site, the robots.txt code is removed.

By February 19, 2009

Article

Google jokesters ward off zombies

The search giant injects a little Halloween humor into its robots.txt file's instructions for Web-crawling software.

By October 31, 2008