HotForBot

So, you’ve probably wondered, “do these guys even have a robot.txt file on their site?”…

Yer damn right we do! (our robots.txt file) And we don’t need to keep downloading and uploading every time we want to make and addition or change because we use a great plugin from Adam Brown called KB Robots.txt Plugin for WordPress .

This plugin lets you have direct access to your robots.txt file right from the “Options” menu of your WordPress dashboard.

Got WordPress? - Get this plugin. We use it.

It might not be a TOTALLY complete list (there may be some obscure Canadian hacker on a boat off the shores of Newfoundland who has a new bot), but the list supplied by RobotsTXT.org is a fairly reliable list of bots, with data on who they are, and a whole lot more.

Here’s an example data return on number 282 of 298 - “WebSpider”:
(more…)