HotForBot

So, you’ve probably wondered, “do these guys even have a robot.txt file on their site?”…

Yer damn right we do! (our robots.txt file) And we don’t need to keep downloading and uploading every time we want to make and addition or change because we use a great plugin from Adam Brown called KB Robots.txt Plugin for WordPress .

This plugin lets you have direct access to your robots.txt file right from the “Options” menu of your WordPress dashboard.

Got WordPress? - Get this plugin. We use it.

Everybody has their opinion of what a web page should ‘look’ like. The problem with this is that everyone has their own opinion of what the web page should look like. Fortunately, search bots are much more discerning and less opinionated, and they look for specific elements which are either there, or not there. If it’s there, it’s good, and if not, it’s not. Simple.
(more…)

It might not be a TOTALLY complete list (there may be some obscure Canadian hacker on a boat off the shores of Newfoundland who has a new bot), but the list supplied by RobotsTXT.org is a fairly reliable list of bots, with data on who they are, and a whole lot more.

Here’s an example data return on number 282 of 298 - “WebSpider”:
(more…)