Excluding Robots

Barnett Computer Services

Home > Search Engines > Excluding Robots

Excluding Robots

If you want robots (or spiders) to stop indexing your site, there are a couple of different ways you can go about this. You can also stop them from indexing individual Web pages or subdirectories and you stop robots from specific search engines.

You can stop robots in one of two ways. You can do it with either a txt file or a meta tag. Let’s use Google as an example. One of Google’s more popular robots is called “Googlebot”. To stop Googlebot from indexing a particular page you would add the meta tag <META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">. If you want to stop all robots from indexing a page the meta tag would look like this: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">. If you would want a robot or robots excluded from an entire site you would have to put this tag on every page. That would be a serious pain.

The best way to exclude a robot or robots from an entire site is to use a robots.txt file. You place the robots.txt file on your server and it instructs the robots what not to index on your web site. You create can create this file in Dreamweaver, NotePad, or any application that will allow you to create a text file. Search engines will request a robots.txt file before indexing your site. As an example, if you didn’t want Google to index your site, put the following into a robots.txt file:

User-agent: Googlebot
Disallow: /