If you want robots (or spiders) to stop indexing your site, there
are a couple of different ways you can go about this. You can also stop them from indexing
individual Web pages or subdirectories and you stop robots from specific search engines.
You can stop robots in one of two ways. You can do it with either
a txt file or a meta tag. Let's use Google as an example. One of Google's more popular robots
is called "Googlebot". To stop Googlebot from indexing a particular page you would add the
<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">. If you want
to stop all robots from indexing a page the meta tag would look like this:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">. If you would want a
robot or robots excluded from an entire site you would have to put this tag on every page.
That would be a serious pain.
The best way to exclude a robot or robots from an entire site
is to use a robots.txt file. You place the robots.txt file on your server and it instructs
the robots what not to index on your web site. You create can create this file in Dreamweaver,
NotePad, or any application that will allow you to create a text file. Search engines will
request a robots.txt file before indexing your site. As an example, if you didn't want
Google to index your site, put the following into a robots.txt file:
Copyright 2000 - 2004 Barnett Computer Services All rights reserved.