Overview: The Robots Tag declares to search engines what content to index and spider

Robots, also known as spiders, are automated mechanisms that spider your site, or search your site on how to categorize the information you submitted to the search engine. Typically, a website owner would submit the main page and the robots would visit your site and collect all subpages and related links from your main page. However, this tag enables you to control which pages you would like spidered, and which to ignore. For instance, certain webpages and directories (ie: CGI Scripts) you may not want indexed in the search engines. Using the robots tag, you can define which pages to follow, which to index and which to ignore completely.

Search Engines Usage

The Robots META Tag is used by search engines as a means to indicate the level of spidering a search engine should do. Most search engines look for this META tag and will only index and/or spider the pages you want to be indexed.

Recommended Usage: Suggested