10 Results for

robots.txt

download

Robots.txt Editor

Log onto your FTP or local network server to select documents, which are not searchable, generate robots.txt files, and track spiders.

download now
619 total downloads
1 last week
download

Robots.Txt Generator Software

Block bad Web robots and stop others from scanning and indexing your Web site.

download now
788 total downloads
1 last week
download

Blasphemous robots.txt for iPad and iPhone

The Robot Exclusion Protocol is a convention for instructing web search engine spiders about which portions of your websites they should access....

download now
9 total downloads
1 last week
download

SiteMap Generator

Extract all the links from your site recursively and generate Google or Yahoo Sitemap.

3.5 stars Users' rating out of 8 reviews
download now
18,795 total downloads
31 last week
download

A1 Sitemap Generator

Create sitemaps for your websites including visual HTML/CSS, XML, image, video, and text.

4.5 stars Users' rating out of 8 reviews
download now
17,174 total downloads
21 last week
download

Robotto

Monitor robots.txt files and get alerts when they change.

download now
26 total downloads
2 last week
download

Robot-Manager Professional

Manage and track robots, spiders, and crawlers that visit your Web site.

download now
2,362 total downloads
1 last week
download

Robot-Manager Standard

Manage and track robots, spiders, and crawlers that visit your Web site.

5 stars Users' rating out of 1 reviews
download now
1,257 total downloads
1 last week
download

Keyword Crawler

Optimize, explore, and view your site like Web engines.

2.5 stars Users' rating out of 2 reviews
download now
6,205 total downloads
1 last week
download

LinkAssistant

Increase Web sites popularity and exchange cross-links with partner sites.

4 stars Users' rating out of 18 reviews
download now
5,768 total downloads
1 last week
Article

Analyze, create robots.txt files in Google

Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

By March 29, 2008

Article

Feds use robots.txt files to stay invisible online. Lame.

Some federal government Web sites, including the Office of the Director of National Intelligence, are trying to remain hidden online by blocking search engines from indexing them. Not only is this lame, but it's a good reason to ignore their robots.txt fi

By August 24, 2007

Article

Phew! Google prevents Terminators from killing Larry and Sergey

If you see a killer robot (including one that looks like a certain former California governor) making threatening movements toward Google's founders, there's no need to worry.

By July 3, 2014

Article

Craigslist reportedly asks search engines to stop indexing its posts

Alleged action follows a lawsuit filed by the online bulletin board to prevent data harvester 3taps and apartment listings aggregator PadMapper from exporting its listings.

By August 7, 2012

Article

Kogan Australia website missing from Bing

Is Microsoft biting back after Kogan set penalties for IE7 users? Is it a mad publicity stunt? Either way, the Kogan Australia website is mysteriously missing from Bing search results.

By July 9, 2012

Article

Google First Click Free gets you under paywalls -- but only five times

Google has added an option to its First Click Free feature that will allow publishers to place a limit on how many free stories you can read before you're asked to cough up

By December 2, 2009

Article

Report: Microsoft may help News Corp. delist sites

The media empire is talking with the tech giant about having its Web sites removed from Google search results, according to a Financial Times report.

By November 22, 2009

Article

Buzz Out Loud Podcast 1105: The New Testament of Torrents

I'm very pleased that LegalTorrents.com has come along to prove that I'm not crazy when I say there are plenty of legal reasons to use torrents. And they even have a torrent of Bible 2.0. Probably with new commandments from the MPAA. We also chat about AM

By November 12, 2009

Article

Execs reveal why newspapers don't block Google

Forbes.com CEO rips Google but tells CNET it needs search engine to stay competitive. Former Washington Post editor says time to play by Google's rules.

By May 7, 2009

Article

Google: We're good for journalism

Under fire for being a 'parasite' that's hurting the media business, Google offers a rebuttal in Senate testimony.

By May 6, 2009