How do you create a robots.txt file?

Search Engine Optimization Services - Creating a robots.txt file

* By Robin Nobles, Director of Training for the Academy of Web Specialists

How do you create a robots.txt file?

Creating a robots.txt file manually is tricky at best. One little mistake will make the entire file invalid, and you'reng your Web site up to spiders operating on their own free will. Plus, through search engine optimization services, if you've created engine-specific pages with similar content, the spiders may discover those nearly duplicate pages, and you could find yourself in trouble for spamming.

While there have been software programs on the market to create robots.txt files and help with other search engine optimization services, that's been their sole function. Spider analysis hasn't been a part of their features, until now.

Robot-Manager Professional Edition is a software program that concentrates totally on spider visits to your site.

Recently, I had the good fortune of reviewing a new software program that performs two major functions: spider analysis and robots.txt creation.

In a nutshell, I'm amazed, and I have to predict that the software, Robot-Manager Professional Edition, will rapidly become a "must have" tool for search engine marketers.

To begin with, the program is easy to use. I don't know about you, but I don't have time to learn a complicated software program. Though the program is intuitive and doesn't require the reading of a complicated user's guide, you'll still find detailed help topics that can answer any questions.

The program begins with Step 1, where you choose which spiders you want to work with. In Step 2, you download your file directory tree, which is where you begin to tell the program which files you want to keep spiders out of. The robots.txt file is automatically created in Step 3, and you can instantly upload the file to your directory. The program allows you to edit the file manually as well.

In Step 4, the program gets down to the real "meat" of the analysis: analyzing your log files for spider visits. When downloading and analyzing the log files from the domain I worked with, it automatically added 30 new spiders to the spider list.