30 Results for

robots.txt

Article

Analyze, create robots.txt files in Google

Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

By Mar. 29, 2008

Article

Feds use robots.txt files to stay invisible online. Lame.

Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi

By Aug. 24, 2007

Article

Craigslist reportedly asks search engines to stop indexing its posts

Alleged action follows a lawsuit filed by the online bulletin board to prevent data harvester 3taps and apartment listings aggregator PadMapper from exporting its listings.

By Aug. 7, 2012

Article

Google jokesters ward off zombies

The search giant injects a little Halloween humor into its robots.txt file's instructions for Web-crawling software.

By Oct. 31, 2008

Article

Much ado about Whitehouse.gov's new openness

President Obama's new White House Web site has been lauded for being more open than former President Bush's. There's just one problem with that theory: it's wrong.

By Jan. 21, 2009

Article

White House expands use of search-blocking code

Whitehouse.gov's administrators silently triple the number of Web pages that it forbids Google and other search engines from accessing. Is this a bad omen or much ado about nothing?

By Jan. 30, 2009

Article

Recovery.gov blocked search engine tracking

update After Google seemingly ignored restrictive search engine-blocking code built into the Obama administration's new stimulus-related site, the robots.txt code is removed.

By Feb. 19, 2009

Article

National Intelligence Web site no longer invisible to search engines

On Thursday, the White House ceased to block search engines from its Iraq-related directories. Now the National Intelligence director is doing the same thing in response to queries from News.com.

By Sep. 7, 2007

Article

White House says blocking Iraq Web documents was 'mistake'

In response to questions from News.com, the Bush administration says blocking search engines from indexing Iraq-related documents was a simple mistake and alters the White House Web site.

By Sep. 7, 2007

Article

Chinese social networks block Baidu indexing

User privacy concerns on Chinese social networking sites have led the biggest players to block indexing by Baidu, China's leading search engine.

By Sep. 14, 2008