X

Working with spiders

Site owners and SEOs have the opportunity to work with the search engines to identify crawl issues, check backlinks, and get access to even more useful information.

Stephan Spencer
Search engine optimization expert Stephan Spencer shares late-breaking SEO tools, tips, trends, resources, news and insights. Stephan is the founder and president of Netconcepts, a web agency specializing in search engine optimized ecommerce. Clients include Discovery Channel, AOL, Home Shopping Network, Verizon SuperPages.com, and REI, to name a few. Stephan is a frequent speaker at Internet conferences around the globe. He is also a Senior Contributor to MarketingProfs.com, a monthly columnist for Practical Ecommerce, and he's been a contributor to DM News, Multichannel Merchant, Catalog Success, Catalog Age, and others.
Stephan Spencer
3 min read

Web site owners and SEOs alike often feel at odds with the search engines, but times are changing. This was often the case in the past when the engines made updates and changes to their algorithms that seemed to send Web sites into a SERP tailspin, leaving everyone scrambling to regain their precious page-one positions. The engines were also a lot less forthcoming with information and guidance, perhaps taking the view that giving this information gave too much power to the spammers and phishers.

While this view was understandable on the surface, it didn't float all that well in reality. In the real world, those who are out to game and manipulate the engines may have as many or more resources to keep up with the engines than "the rest of us." So over the last few years, the search engines have continued to be more open with what they consider important as well as what abuses may get sites into trouble, perhaps realizing that there are also a lot of sites that may not have been purposely trying to mislead the engines, but were just victims of bad advice. And of course the algorithms have become far more powerful and fine-tuned than they once were.

By openly helping everyone, they are really just helping to raise the bar of quality for all sites, and maybe even making it even harder for bad sites to game the engines. Along with providing more detailed information and answering more and more questions publicly, the greatest advancement they have made has been in creating tools to actually give site owners (who have validated their sites) more information about their sites than they've ever experienced before.

Webmaster Central and Webmaster Tools

Google introduced Webmaster Central, which continues to add more and more features for site owners. Not surprising, Webmaster Central is leading the pack in delivering great information and tools to Webmasters. At the center, literally, is Webmaster Tools, which provides site owners with fairly detailed information on site crawling, queries, considerably more backlink information than can be queried outside of Tools, and much more. The query information in particular provides an unprecedented view of the search phrases that a site is showing up for, including those terms that aren't actually delivering traffic to the site.

Site Explorer

Yahoo's Site Explorer is still lacking in a few areas compared with Google's Webmaster Tools, but they almost make up for that with their powerful link information. Through simple drop-down menus, it is quick and easy to tailor results based on links to a specific page or the entire site, to include all links or to exclude links from the site and focus on external links only. Yahoo added a new feature that may give even more control to site owners. The Dynamic URLs tab gives site owners the ability to inform Yahoo of their site's dynamic URL patterns to help eliminate duplicate content issues, better handling of multiparameter URLs, addressing session IDs, and even presenting "cleaner" URLs in search results. Ideally, it would be best to address as many of these issues through rewrites and the robots.txt file on the server, but this is a great addition as a backup or for when that isn't possible.

Webmaster Portal

Trying not to be outdone, the Live Search team at MSN recently announced their entry into the mix with the Webmaster Portal, currently in beta and by request only. Little detailed information is available, but their tool also is claiming to help troubleshoot crawl issues, assist with sitemaps, and provide site statistics, including a replacement to the "link:" operator query that was decommissioned back in March. The portal is slated to be fully available to the public by late fall, but it may be worthwhile to request an invite to participate in the beta now.

The advancement in all of these tools is great news to Webmasters and SEOs alike. They continue to put more information and control into our hands. Not wanting to be outdone by the others, hopefully each of the engines will add each other's additions to their own toolsets. As each of these is free, there is no reason for site owners not to take a few minutes to validate their sites and start spending a little time each month putting these tools to work for them. This is one invitation from the spiders you don't want to turn down.