26 Results for

robots.txt

Article

Analyze, create robots.txt files in Google

Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

By March 29, 2008

Article

Feds use robots.txt files to stay invisible online. Lame.

Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi

By August 24, 2007

Article

Phew! Google prevents Terminators from killing Larry and Sergey

If you see a killer robot (including one that looks like a certain former California governor) making threatening movements toward Google's founders, there's no need to worry.

By July 3, 2014

Article

Google jokesters ward off zombies

The search giant injects a little Halloween humor into its robots.txt file's instructions for Web-crawling software.

By October 31, 2008

Article

Recovery.gov blocked search engine tracking

update After Google seemingly ignored restrictive search engine-blocking code built into the Obama administration's new stimulus-related site, the robots.txt code is removed.

By February 19, 2009

Article

Craigslist reportedly asks search engines to stop indexing its posts

Alleged action follows a lawsuit filed by the online bulletin board to prevent data harvester 3taps and apartment listings aggregator PadMapper from exporting its listings.

By August 7, 2012

Article

Much ado about Whitehouse.gov's new openness

President Obama's new White House Web site has been lauded for being more open than former President Bush's. There's just one problem with that theory: it's wrong.

By January 21, 2009

Article

White House expands use of search-blocking code

Whitehouse.gov's administrators silently triple the number of Web pages that it forbids Google and other search engines from accessing. Is this a bad omen or much ado about nothing?

By January 30, 2009

Article

National Intelligence Web site no longer invisible to search engines

On Thursday, the White House ceased to block search engines from its Iraq-related directories. Now the National Intelligence director is doing the same thing in response to queries from News.com.

By September 7, 2007

Article

White House says blocking Iraq Web documents was 'mistake'

In response to questions from News.com, the Bush administration says blocking search engines from indexing Iraq-related documents was a simple mistake and alters the White House Web site.

By September 7, 2007