Source Editor Overview


Source Editor Overview	Contents > Dialog Reference >Source Editor Dialog

The source editor dialog is used to preview and edit the robot exclusion source generated by RoboGen before it is saved. It is accessed by choosing Verify Source from the File menu.

The robot exclusion syntax is fairly straight forward. Each section has a user-agent tag and the associated disallow tags and comments are inserted in the file by preceding pound signs (#). The generated source for a simple robot exclusion file might look something like this:

# Rietta Solutions, Inc. -- http://www.rietta.com
# Robot Exclusion File -- robots.txt
# Author: Frank Rietta
# Last Updated: 05/23/99 21:27:40
User-agent: *
Disallow: /cgi-bin/
Disallow: /temp/ # these files change a lot
Disallow: /search.html # search engines don't need to index this
Disallow: /about.html # same as above

This exclusion code is fairly straight forward; it consists of two major parts, the optional header which is the group of comments at the top of the file and the disallow section. Notice that an asterisk (*) is used as the user-agent -- that means that this disallow section will apply to all robots.

After you have made all the changes you want to make, simply click the Save As button to save the file.

Note: Even though RoboGen lets you save your exclusion file under any name on your computer, it must be called robots.txt when uploaded to you server.

This HTML file is part of the documentation for Rietta Solutions's RoboGen application. Click here for more information about RoboGen.