HotForBot

Everybody has their opinion of what a web page should ‘look’ like. The problem with this is that everyone has their own opinion of what the web page should look like. Fortunately, search bots are much more discerning and less opinionated, and they look for specific elements which are either there, or not there. If it’s there, it’s good, and if not, it’s not. Simple.
(more…)

It might not be a TOTALLY complete list (there may be some obscure Canadian hacker on a boat off the shores of Newfoundland who has a new bot), but the list supplied by RobotsTXT.org is a fairly reliable list of bots, with data on who they are, and a whole lot more.

Here’s an example data return on number 282 of 298 - “WebSpider”:
(more…)