RoboGen has been designed to make managing the robot exclusion
files which can control how search engines visit websites as easy as
possible. Here are some screen shots of RoboGen in action.
Most of the operations of
RoboGen center around the main window.
(Click to Enlarge)
The Ftp Dialog is where
you manage your FTP servers and passwords. It can store the information
for multiple sites.
The Robot Manager provides
is where you view and edit the user-agent information in the robot database.
RoboGen comes with information on over 180 known web robots and the robot
information for 10 of the most popular search engines.
The header editor is used
to edit the header that is placed at the top of all the robot exclusion
files you create.
The options dialog lets
you set general FTP and help file options.
The source editor is used
to edit the generated robot exclusion code before saving it to the disk.
If you wish to enter documents
or directories manually, use the Add Relative Path dialog.
The Advanced Add dialog
allows multiple documents and paths to be entered at once. (One per
line.)
RoboTag is a small companion
program used to create Robot Exclusion META tags. Click
here for more information on RoboTag.
