| All About Search Indexing Robots and Spiders: Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites. |
| ASAP Consulting: Robots/Crawlers DB: Searchable list of robot agent strings with their description and links to home pages. |
| Bots vs Browsers: This large database lists user agents in categories and distinguishes between robots and browsers. |
| HTTP User Agent Index: An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg. |
| List of Robot Agent Strings: A list from PGTS of Web robots with the identifying data they leave in Web site logs. |
| Psychedelix List of User-Agents: Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators. |
| Robot IP Address: Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only. |
| Robots, Spiders and Other User Agents: a Resource for WebMasters: José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web. |
| Robotstxt.org: Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots. |
| Search Engine IP Addresses: Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders. |
| Search Engine Robots and Other User Agents: John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses. |
| Spider Track: Tracks and displays real time hits and IP addresses of MSNbot, Yahoo's Slurp, and Googlebot. Provides aggregate hits of each spider over the last 30 days. |
Classical
