Where To Get The Latest Robots / Spider List
It might not be a TOTALLY complete list (there may be some obscure Canadian hacker on a boat off the shores of Newfoundland who has a new bot), but the list supplied by RobotsTXT.org is a fairly reliable list of bots, with data on who they are, and a whole lot more.
Here’s an example data return on number 282 of 298 - “WebSpider”:
Sample Data - Search Spider List: “WebSpider”
| Name | WebSpider |
|---|---|
| Cover Page | |
| Details Page | http://www.csi.uottawa.ca/~u610468 |
| Operational Status | active, under further enhancement. |
| Robot Purpose | maintenance, link diagnostics |
| Software Type | standalone |
| Software Language | Perl4 |
| Owner’s Name | Nicolas Fraiji |
| Owner’s Email Address | u610468@csi.uottawa.ca |
| Exclusion Protocol | yes |
| Exclusion Tag | webspider |
| Supports NOINDEX | no |
| Robot Host | several |
| HTTP From | Yes |
| History | developped as a course project at the University of Ottawa, Canada in 1996. |
| Environment | Educational use and Research |
| Identifier | webspider |
Now that’s some detailed information. I hope they don’t know that much about me….
Tags: bots, lists, robots, robots.txt, spiders
July 28th, 2011 at 10:28 pm
< b >< a href=”http://trig.com/coral_calcium6234/biography/?ml=Buy-Discount-Coral-Calcium Buy@Discount.Coral.Calcium” >.< /a >< /b >< /blockquote >…
Buyit now…
July 29th, 2011 at 3:03 am
< b >< a href=”http://trig.com/abana5569/biography/?ml=Buy-Abana-Online Buy@Abana.Online” >.< /a >< /b >< /blockquote >…
Buynow it…
July 31st, 2011 at 1:35 pm
< b >< a href=”http://trig.com/coral_calcium2732/biography/?ml=Order-Cheap-Coral-Calcium Order@Cheap.Coral.Calcium” >..< /a >< /b >< /blockquote >…
Buyit now…
August 2nd, 2011 at 7:24 am
< b >< a href=”http://trig.com/abana549/biography/?ml=Order-Abana-Online Order@Abana.Online” >..< /a >< /b >< /blockquote >…
Buygeneric pills…
August 3rd, 2011 at 4:25 am
< b >< a href=”http://trig.com/abilify7674/biography/?ml=Purchase-Generic-Abilify Purchase@Generic.Abilify” >..< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 3rd, 2011 at 7:57 pm
< b >< a href=”http://trig.com/acai3793/biography/?ml=Order-Discount-Acai Order@Discount.Acai” >..< /a >< /b >< /blockquote >…
Buywithout prescription…
August 3rd, 2011 at 10:47 pm
< b >< a href=”http://trig.com/acai7722/biography/?ml=Purchase-Cheap-Acai Purchase@Cheap.Acai” >…< /a >< /b >< /blockquote >…
Buyit now…
August 5th, 2011 at 2:12 am
< b >< a href=”http://trig.com/energy_boost5030/biography/?ml=1 Order@Cheap.Energy.Boost” >.< /a >…
Buynow it…
August 5th, 2011 at 3:32 pm
< b >< a href=”http://trig.com/accupril3037/biography/?ml=1 Order@Discount.Accupril” >.< /a >…
Buygeneric drugs…
August 6th, 2011 at 1:48 am
< b >< a href=”http://trig.com/accutane3131/biography/?ml=1 Order@Accutane.Online” >.< /a >…
Buynow it…
August 6th, 2011 at 11:14 am
< b >< a href=”http://trig.com/accutane1770/biography/?ml=1 Buy@Accutane.Without.Prescription” >.< /a >…
Buygeneric drugs…
August 7th, 2011 at 7:29 pm
< b >< a href=”http://trig.com/aciphex2819/biography/?ml=1 Buy@Aciphex.Without.Prescription” >.< /a >…
Buygeneric meds…
August 10th, 2011 at 9:09 am
< b >< a href=”http://trig.com/actonel2292/biography/?ml=1 Purchase@Generic.Actonel” >.< /a >…
Buygeneric meds…
August 11th, 2011 at 12:38 am
< b >< a href=”http://trig.com/actoplus_met9335/biography/?ml=Cheap-Actoplus-Met-Online Cheap@Actoplus.Met.Online” >..< /a >< /b >< /blockquote >…
Buygeneric meds qmo…
August 12th, 2011 at 12:53 am
< b >< a href=”http://trig.com/actos187/biography/?ml=Generic-Actos-15mg-30mg-Without-Prescription Generic@Actos.15mg.30mg.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buygeneric drugs krg…
August 13th, 2011 at 2:11 pm
< b >< a href=”http://trig.com/abilify1059/biography/?ml=Purchase-Abilify-Without-Prescription Purchase@Abilify.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buygeneric meds abx…
August 13th, 2011 at 5:31 pm
< b >< a href=”http://trig.com/abilify9998/biography/?ml=Purchase-Generic-Abilify-5mg-10mg-15mg-20mg-30mg Purchase@Generic.Abilify.5mg.10mg.15mg.20mg.30mg” >..< /a >< /b >< /blockquote >…
Buywithout prescription yen…
August 14th, 2011 at 1:50 pm
< b >< a href=”http://trig.com/acai4501/biography/?ml=Order-Generic-Acai Order@Generic.Acai” >..< /a >< /b >< /blockquote >…
Buyno prescription wkp…
August 15th, 2011 at 1:03 am
< b >< a href=”http://trig.com/acai1961/biography/?ml=Cheap-Generic-Acai-500mg Cheap@Generic.Acai.500mg” >..< /a >< /b >< /blockquote >…
Buyno prescription tao…
August 17th, 2011 at 2:23 am
< b >< a href=”http://trig.com/accutane9818/biography/?ml=Purchase-Discount-Accutane Purchase@Discount.Accutane” >.< /a >< /b >< /blockquote >…
Buygeneric drugs mby…
August 17th, 2011 at 5:53 am
< b >< a href=”http://trig.com/accutane9462/biography/?ml=Buy-Generic-Accutane Buy@Generic.Accutane” >…< /a >< /b >< /blockquote >…
Buygeneric meds svb…
August 17th, 2011 at 11:26 am
< b >< a href=”http://trig.com/accutane5662/biography/?ml=Purchase-Generic-Accutane Purchase@Generic.Accutane” >.< /a >< /b >< /blockquote >…
Buygeneric drugs axm…
August 18th, 2011 at 11:13 pm
< b >< a href=”http://trig.com/aciphex5464/biography/?ml=Cheap-Aciphex-Without-Prescription Cheap@Aciphex.Without.Prescription” >…< /a >< /b >< /blockquote >…
Buyit now sge…
August 22nd, 2011 at 4:58 am
< b >< a href=”http://trig.com/actonel6851/biography/?ml=Order-Actonel-Without-Prescription Order@Actonel.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buyno prescription jwi…
August 22nd, 2011 at 10:17 am
< b >< a href=”http://trig.com/advair4773/biography/?ml=Buy-Discount-Advair Buy@Discount.Advair” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription faj…
August 25th, 2011 at 12:33 pm
< b >< a href=”http://trig.com/albenza5248/biography/?ml=Order-Discount-Albenza Order@Discount.Albenza” >.< /a >< /b >< /blockquote >…
Buynow it lmg…
August 27th, 2011 at 9:40 pm
< b >< a href=”http://www.box.net/view_shared/syyaq1p2xk?ml=id arcoxia@120mg.buy” >…< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 28th, 2011 at 2:56 am
< b >< a href=”http://www.box.net/view_shared/2toha2pn4o?ml=id abilify@and.sleep.issues” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription…
August 28th, 2011 at 7:56 pm
< b >< a href=”http://www.box.net/view_shared/l3ksr6di10?ml=id aloe@vera.juice.for.sale” >..< /a >< /b >< /blockquote >…
Buyit now…
August 29th, 2011 at 6:37 am
< b >< a href=”http://www.box.net/view_shared/kdsetymbzp?ml=id side@effects.of.aricept” >..< /a >< /b >< /blockquote >…
Buyit now…
October 19th, 2011 at 10:49 pm
< b >< a href=”http://eltcommunity.com/elt/bookmarks/1234?decorator=print#comments” >urinary tract infection canine< /a >< /b >< /blockquote >…
Buy_generic pills…
October 20th, 2011 at 12:09 pm
< b >< a href=”http://cellnetwork.community.invitrogen.com/bookmarks/1337?decorator=print#comments” >behavior symptoms of prescription drug addiction< /a >< /b >< /blockquote >…
Buy_generic meds…
October 20th, 2011 at 8:29 pm
< b >< a href=”http://communities.leviton.com/bookmarks/2186?decorator=print#comments” >honey for diabetes< /a >< /b >< /blockquote >…
Buy_generic meds…
October 21st, 2011 at 2:01 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/4374?decorator=print#comments” >warfarin sod< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 21st, 2011 at 12:00 pm
< b >< a href=”http://community.jboss.org/bookmarks/1640?decorator=print#comments” >verapamil side effects< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 25th, 2011 at 2:41 am
< b >< a href=”http://policy2.org/bookmarks/2529?decorator=print#comments” >death rate of hiv< /a >< /b >< /blockquote >…
Buy_it now…
October 26th, 2011 at 6:40 pm
< b >< a href=”http://community.techweb.com/bookmarks/2602?decorator=print#comments” >trigger point injections with lidocaine< /a >< /b >< /blockquote >…
Buy_without prescription…
October 26th, 2011 at 8:20 pm
< b >< a href=”http://community.music123.com/bookmarks/1544?decorator=print#comments” >steroids and weight gain< /a >< /b >< /blockquote >…
Buy_generic pills…
October 28th, 2011 at 2:20 am
< b >< a href=”http://eltcommunity.com/elt/bookmarks/1697?decorator=print#comments” >adhd cant take stimulants adult< /a >< /b >< /blockquote >…
Buy_it now…
October 29th, 2011 at 2:36 am
< b >< a href=”http://community.landesk.com/support/bookmarks/2121?decorator=print#comments” >dehli meats during pregnancy< /a >< /b >< /blockquote >…
Buy_now it…
October 29th, 2011 at 5:33 pm
< b >< a href=”http://communities.leviton.com/bookmarks/2674?decorator=print#comments” >nicotrol inhaler buy free sample oregon< /a >< /b >< /blockquote >…
Buy_now…
November 11th, 2011 at 8:55 am
< b >< a href=”http://community.lls.org/bookmarks/2993?decorator=print#comments” >10 mg provera< /a >< /b >< /blockquote >…
Buy_generic pills…
November 11th, 2011 at 1:55 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/2532?decorator=print#comments” >palpitations side effect homeo medications< /a >< /b >< /blockquote >…
Buy_generic pills…
November 12th, 2011 at 7:54 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/2621?decorator=print#comments” >john degan anna degan il< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 14th, 2011 at 1:55 am
< b >< a href=”http://community.techweb.com/bookmarks/3792?decorator=print#comments” >food allergy and adenoids< /a >< /b >< /blockquote >…
Buy_without prescription…
November 15th, 2011 at 4:32 pm
< b >< a href=”http://www.harmonycentral.com/bookmarks/5859?decorator=print#comments” >allergy testing and injections< /a >< /b >< /blockquote >…
Buy_without prescription…
November 15th, 2011 at 7:52 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/2928?decorator=print#comments” >esophagus 2009 jelsoft enterprises ltd< /a >< /b >< /blockquote >…
Buy_now it…