26 Results for

robots.txt

Article

Phew! Google prevents Terminators from killing Larry and Sergey

If you see a killer robot (including one that looks like a certain former California governor) making threatening movements toward Google's founders, there's no need to worry.

By July 3, 2014

Article

Craigslist reportedly asks search engines to stop indexing its posts

Alleged action follows a lawsuit filed by the online bulletin board to prevent data harvester 3taps and apartment listings aggregator PadMapper from exporting its listings.

By August 7, 2012

Article

Kogan Australia website missing from Bing

Is Microsoft biting back after Kogan set penalties for IE7 users? Is it a mad publicity stunt? Either way, the Kogan Australia website is mysteriously missing from Bing search results.

By July 9, 2012

Article

Analyze, create robots.txt files in Google

Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

By March 29, 2008

Article

Feds use robots.txt files to stay invisible online. Lame.

Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi

By August 24, 2007

Article

Google First Click Free gets you under paywalls -- but only five times

Google has added an option to its First Click Free feature that will allow publishers to place a limit on how many free stories you can read before you're asked to cough up

By December 2, 2009

Article

Report: Microsoft may help News Corp. delist sites

The media empire is talking with the tech giant about having its Web sites removed from Google search results, according to a Financial Times report.

By November 22, 2009

Article

Buzz Out Loud Podcast 1105: The New Testament of Torrents

I'm very pleased that LegalTorrents.com has come along to prove that I'm not crazy when I say there are plenty of legal reasons to use torrents. And they even have a torrent of Bible 2.0. Probably with new commandments from the MPAA. We also chat about AM

By November 12, 2009

Article

Execs reveal why newspapers don't block Google

Forbes.com CEO rips Google but tells CNET it needs search engine to stay competitive. Former Washington Post editor says time to play by Google's rules.

By May 7, 2009

Article

Google: We're good for journalism

Under fire for being a 'parasite' that's hurting the media business, Google offers a rebuttal in Senate testimony.

By May 6, 2009