Search Engine Optimization Services - Spider Analysis

* By Robin Nobles, Director of Training for the Academy of Web Specialists

More on spider analysis - Intro to robots.txt files

Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format, and this will help in performing search engine optimization services.

• Has your site been spidered?
• If so, by which engines?
• When did the spiders visit?
• Which directories and pages did they visit?
• Are certain pages getting respidered more often, signaling their importance to the search engines?
• Are certain pages not getting spidered at all?
• Are the spiders indexing inappropriate content?
• Are the spiders getting everything they want and need, or are they receiving error messages?
• Was your site spidered within the specified time agreed upon in the pay inclusion programs you're participating in?
• Is your site getting respidered on a regular basis, as agreed upon in your participating pay inclusion programs?

Though the object of search engine optimization servcies and marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages. You can do this with a robots.txt file.

A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.

Let's say that you're creating a new page for your site, and you've placed the page online while you tweak and edit it. The page isn't ready for visitors, so you wouldn't want it indexed yet. Or, let's say you've placed some employee guidelines on your site. The guidelines are of interest only to your employees, and there's no reason for the public to view them, so this does not need search engine optimization services.

Using a robots.txt file, you can keep the spiders out of those pages.

Therefore, when working with spiders or robots, you want to be able to:

• Create a robots.txt file quickly and easily
• Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant pages and direct robots from international search engines to the localized content areas of your site
• Send e-mail harvesting programs away from your site to keep your e-mail spam down
• View highlighted pages requested in error by the spiders
• Direct search engine spiders to relevant areas of your Web site

More on robots.txt files...