加载中…
正文 字体大小:

全球搜索引擎蜘蛛/爬虫列表

(2011-12-26 15:27:13)
标签:

搜索引擎

爬虫

win32

蜘蛛

小智

分类: 网站日志

全球搜索引擎蜘蛛/爬虫列表

这里列出了全球搜索引擎蜘蛛/爬虫列表,如果有没有被我们收录的 请您联系我们 我们修正它.



Crawler User agent Owner
192.com 192.comAgent 192.com
4anything 4anything.com LinkChecker v2.0 4anything
A-Online A-Online Search Aon
ABCdatos ABCdatos BotLink/5.xx.xxx#BBL ABCdatos
AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) AOL
ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) ASAHA
ASPseek ASPSeek/1.2.xa ASPseek
ASPSeek/1.2.xxpre
ASPSeek/1.2.x
ASPSeek/1.2.5
ASPseek/1.2.9d
ASPseek/1.2.xx
AVSearch AVSearch-1.0(peter.turney@nrc.ca) National Research Council Canada
AbachoBot AbachoBOT Abacho
AbachoBOT (Mozilla compatible)
Aberja Checkoma Aberja Checkoma Aberja
Abot abot/0.1 (abot; http://www.abot.com; abot@abot.com) Abot.com
About About/0.1libwww-perl/5.47 About
AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) AboutUs
Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) Accelovation
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com)
Accoona Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) Accoona
accoona
Acoi AcoiRobot Acoi
Acoon Robot Acoon Robot v1.50.001 Acoon
Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) Acoon
Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o
rg)
Isara
Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) Activtourist
Aesop AESOP_com_SpiderMan Aesop
Agada Mozilla/4.0 (agadine3.0) www.agada.de Agada
agadine/1.x.x (+http://www.agada.de)
Mozilla/4.0 (agadine3.0)
AgentName AgentName/0.1 libwww-perl/5.48 Linkomatic
Aibot AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C
hina)
21seek
Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Accoona
Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) Aipbot
Alacra PortalBSpider/2.0 (spider@portalb.com) Alacra
Aladin.de Aladin/3.324 Abacho
Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/) Aleksika
Alexa ia_archiver Alexa
AlkalineBOT AlkalineBOT/1.4 (1.4.0326.0 RTM) Vestris
AlkalineBOT/1.3
Allesklar.de Allesklar/0.1 libwww-perl/5.46 Allesklar
Allrati Allrati/1.1 (+) Unknown
Almaden http://www.almaden.ibm.com/cs/crawler [hc5] IBM
http://www.almaden.ibm.com/cs/crawler
Altavista Scooter-W3-1.0 Altavista
Scooter2_Mercator_x-x.0
scooter-venus-3.0.vns
Scooter-W3.1.2
Scooter-3.2.NIV
Scooter-3.2.JT
Scooter-3.2.EX
Scooter/2.0 G.R.A.B V1.0
Scooter_trk3-3.0.3
Scooter-3.2.SF0
Scooter/3.3.QA.pczukor
Scooter/3.3
Scooter/1.1 (custom)
Scooter-3.2.DIL
Scooter/1.0 scooter@pa.dec.com
Scooter-3.0.FS
Scooter-3.0.EU
Scooter/2.0 G.R.A.B. X2.0
Scooter/2.0 G.R.A.B. V1.1.0
Scooter-3.0.HD
Scooter-3.0.VNS
Scooter-3.2.BT
Scooter-3.2
Scooter-3.0QI
Scooter/3.3_SF
Scooter-3.2.snippet
Scooter-ARS-1.1-ih
Scooter/3.3.vscooter
Scooter-3.3dev
Scooter-ARS-1.1
Scooter_bh0-3.0.3
Scooter/1.0
Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) Amfibi
Amidalla libwww-perl/5.65 Amidalla
Annomille AnnoMille spider 0.1 alpha - http://www.annomille.it Annomille
AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) Ansearch
AnswerBus AnswerBus (http://www.answerbus.com/) AnswerBus
Answerchase PROve AnswerBot 4.0 Answerchase
Antibot antibot-V1.3.3.1/debian-linux-sarge Antidot
Any Search Info Mozilla/4.0 (Sleek Spider/1.2) Search-Info
Anzwers Australia AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Anzwers Australia
Apexoo Spider Apexoo Spider 1.0 Apexoo
Aport Aport Aport
Appie appie 1.1 (www.walhello.com) Walhello
appie 1.1 (www.walhello.com)
Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) Arabul
ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby
.com;)
Araby
Arachnoidea Arachnoidea (arachnoidea@euroseek.com) Euroseek
ArchitextSpider ArchitextSpider Excite
Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht
tp://www.archive-it.org)
Archive.org
Arexera TECOMAC-Crawler/0.x Arexera
X-Crawler
Arianna www.arianna.it Libero
Arikus_Spider Arikus_Spider Arikus
Asahina Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) Asahina
Asahina-Antenna/1.x
Ask 24x Info ask.24x.info Ask 24x
Ask Jeeves/Teoma Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Ask Jeeves
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr
awling.html)
Asked asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) Asked
Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) Askpeter
Asterias asterias/2.0 Singing Fish
Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis
ion: 3.11
Singingfish
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi
on: 3.10
Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) Seeq
Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) @Local
Attentio Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@att
entio.com)
Attentio
Augurnet Swiss augurfind Augurnet Swiss
augurnfind V-1.x
Axada axadine/ (Axadine Crawler; http://www.axada.de/; ) Axada
Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) Axandra
Axmo AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine
.
Axmo
Ay-Up FastBug http://www.ay-up.com Ay-up
BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) Rawgrunt
Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be) BA
BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us) Baboum
BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) Babaloo
Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html) Backlink-Check
Baiduspider Baiduspider+(+http://www.baidu.com/search/spider.htm) Baidu.com
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots
meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Balihoo
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a
spx; tgautier at balihoo dot com)
BanBot BanBots/1.2 (spider@banbots.com) Banbot
BeamMachine BeamMachine/0.5 (dead link remover of www.beammachine.net) BeamMachine
Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) uCHOOSE
BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) Apassion4jazz
BecomeBot Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become
.com/site_owners.html)
BecomeBot
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.
com/site_owners.html)
BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom
e.co.jp/site_owners.html)
Become
BeijingCrawler BeijingCrawler Unknown
BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) BigClique
Biglotron BIGLOTRON (Beta 2;GNU/Linux) Biglotron
Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
Bigsearch
Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk) Bilinfo
BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html
; nutch-agent@lucene.apache.org)
Bilgi
BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Bilgi
Bisnisseek Custom Spider www.bisnisseek.com /1.0 Bisnisseek
Bitacle Robot Bitacle bot/1.1 Bitacle
Bitacle Robot (V:1.0;) (http://www.bitacle.com)
Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net) Blaiz Enterprises
Blaiz-Bee Blaiz-Bee/2.00.5502 (+http://www.blaiz.net) Blaiz
Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
Blitzsuche BlitzBOT@tricus.net RP ONLINE
Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)
BlitzBOT@tricus.net (Mozilla compatible)
BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers
)
BlogRefs
BlogSearch BlogSearch/1.0 +http://www.icerocket.com/ IceRocket
BlogSearch/1.x +http://www.icerocket.com/
BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) Okumura Group
Blogbot Naamah 1.0a/Blogbot (http://blogbot.de/) Blogbot
Naamah 1.0.1/Blogbot (http://blogbot.de/)
Blogdex BlogBot/1.x Massachusetts Institute of Technology
Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com) Blogdimension
Bloglines Bloglines Title Fetch/1.0 (http://www.bloglines.com) Bloglines
Bloglines/3.1 (http://www.bloglines.com)
Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com) Bloglines
BlogzIce BlogzIce/1.0 +http://www.icerocket.com/ IceRocket
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
Boitho boitho.com-robot/1.x (http://www.boitho.com/bot.html) Boitho
boitho.com-dc
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-robot/1.x
Bot bot/1.0 (bot; http://; bot@bot.bot) Unknown
BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) Penn State College of Information Sciences and Technology
Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi
nd.mobi/bot.html abuse@mtld.mobi)
Find.mobi
BravoBrian bSTOP BStop.BravoBrian.it Agent Detector BravoBrian
BravoBrian SpiderEngine MarcoPolo
BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) Brightcloud
Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) University of California
Btbot BTbot/0.x (+http://www.btbot.com/btbot.html) Btbot
BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler) BuildCMS
BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) BuiltWith
BullsEye/Intelliseek BullsEye Intelliseek
BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl
er@burstfind.com)
Burstfind
Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Buscaplus
CEA larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) CEA
CMP libwww-perl/5.41 CMP United Business Media
CUPS PrivacyFinder Cache Bot v1.0 PrivacyBird
Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html) Sensation Internet Services
CanadianContent Search RoboCrawl (http://www.canadiancontent.net) CanadianContent
RoboCrawl (www.canadiancontent.net)
Carleson carleson/1.0 Cosmix
Catall Spider Catall Spider Catall
Catall-Spider Catall-Spider/3.3.3(www.Catall.de) Catall
CazoodleBot CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.co
m)
Cazoodle
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood
lebot; cazoodlebot@cazoodle.com)
Ccubee ccubee/x.0 Empyreum
Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect
ion.com/bot.html )
Changedetection
Charlotte Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) Searchme
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)
Christcentral Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com) Christcentral
ChristCRAWLER 2.0
CipinetBot CipinetBot (http://www.cipinet.com/bot.html) Cipinet
CipinetBot/1.0 (http://www.cipinet.com/bot.html)
CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) CjLog
Claymont Search Claymont.com Claymont Search
CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/) Seznam
Clushbot Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) Clush
Clushbot/2.x (+http://www.clush.com/bot.html)
Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)
 
Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/) Search.com
CoBITSProbe CoBITSProbe Academia Sinica
Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Cobion
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)
oBot ((compatible;Win32))
Combine Combine/2.0 http://combine.it.lth.se/ Combine
Combine/2.0
Combine/3 http://combine.it.lth.se/
Cometrics-bot cometrics-bot, http://www.cometrics.de Cometrics
Cometsystems Crawler (cometsearch@cometsystems.com) Cometsystems
Crawler (cometsearch@cometsystems.com)
Comperio FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Comperio
Compete.com larbin_2.2.0 (crawl@compete.com) Compete Inc
Computerorgs htdig/3.1.6 (http://computerorgs.com) Computerorgs.com
Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen
e.apache.org)
Comrite
ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl) Convera
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Converas RetrievalWare CrawlConvera0.1 (CrawlConvera@yahoo.com) Convera
ConveraCrawler/0.2
Convera Internet Spider V6.x
ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
CoolBot CoolBot SuchMaschine21
Cortina Vision Research Lab image spider at vision.ece.ucsb.edu Vision Research Lab
CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) CougarSearch
Cowbot Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver
Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
CrawlerBoy CrawlerBoy Pinpoint.com Motricity
Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) Yamana Laboratory - Waseda University Japan
Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Crawlly
Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Croccrawler.com
CsCrawler Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo
glespam/crawler.html RPT-HTTPClient/0.3-3
University of Kassel
Csci_b659/0.13 csci_b659/0.13 Indiana University School of Informatics
Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Cuasar
CurryGuide CurryGuide SiteScan 1.1 CurryGuide
CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) CyberAlerts
Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) Cydral
CydralSpider CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com) Cydral
CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com)
DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net
/aboutkr.html)
Daum
DAUM Web Robot Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Daum
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu
nications Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi
cations Corp., Korea; +http://ws.daum.net/aboutkr.html)
DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Dnsdigger
DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
DataFountains DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents
.shtml)
University of California
DataFountains/DMOZ Downloader
DataSpear Spider Bot DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co
m)
DataSpear
DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html
; dssb@dataspear.com)
DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/) dpSearch
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DaviesBot DaviesBot/1.7 (www.wholeweb.net) Wholeweb
Daypop daypopbot/0.x Daypop
DbDig dbDig(http://www.prairielandconsulting.com) Connections
De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) De.com
DeepIndexer DeepIndexer.ca Deepindex
Deepak-USC/ISI deepak-USC/ISI University of Southern California
Deepindex DeepIndex (www.en.deepindex.com) Deepindex
DeepIndex
Deepindex V2
Denmex Websearch Denmex websearch (http://search.denmex.com) Denmex Websearch
DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) DepSpid
Dev-spider2 dev-spider2.searchpsider.com/1.3b Searchspider
DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) DiaGem Japan
Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de Die Kraehe
Diggit Digger/1.0 JDK/1.3.0rc3 Diggit
Direct Hit Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) Teoma
Disco-crawl disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@
discoveryengine.com)
Discoveryengine
disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc
overyengine.com)
Ditto DittoSpyder Ditto
DoCoMo DoCoMo/1.0/Nxxxi/c10 NTT DoCoMo
DoCoMo/1.0/Nxxxi/c10/TB
DoCoMo/2.0 P900iV(c100;TB;W24H11)
Dodgebot dodgebot/experimental Agmlab
DotBot Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne
tdotcom.org)
Dotnetdotcom
DotBot/1.0.1
Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com) Douban
Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/) Download-Tipp
EyeCatcher (Download-tipp.de)/1.0
Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html) Drecom Japan
DtSearchSpider dtSearchSpider dtSearch
Dumbot Dumbot(version 0.1 beta) DumbFind.com
Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)
Dumbot(version 0.1 beta - dumbfind.com)
E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Yamana Laboratory
E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) e-StyleISP
EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at
global-opto dot com)
EApollo
EMPAS_ROBOT EMPAS_ROBOT Empas
ESISmartSpider ESISmartSpider smart-spider.com
Earthcom EARTHCOM.info/1.xbeta [www.earthcom.info] Earthcom.info
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
EARTHCOM.info/1.x [www.earthcom.info]
Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)
EasyDL EasyDL/3.04 http://keywen.com/Encyclopedia/Bot Keywen
EasyDL/3.xx
EasyDL/3.xx http://keywen.com/Encyclopedia/Bot
Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Echo.com
Echo.fr EchO!/2.0 Echo.fr
Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) Charles University in Prague
Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm) Egoto.com
Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) uCHOOSE
Elsop LinkScan/9.0g Unix LinkScan
LinkScan/x.x Unix
LinkScan/11.0beta2 Unix
EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) Emeraldshield
Enfish Tracker Enfish Tracker Enfish
Enoola enoola (http://www.enoola.com) Enoola
Enterprise Search Enterprise_Search/1.0.xxx Innerprise
Search/1.0 (http://www.innerprise.net/es-spider.asp)
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
Enterprise_Search/1.0
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
Entireweb WorldLight Entireweb
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Speedy Spider (Beta/x.x; speedy@entireweb.com)
Speedy_Spider (http://www.entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Envolk
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
EroCrawler EroCrawler EroCrawler
Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com) Eruvo
EuripBot EuripBot/0.4 (+http://www.eurip.com) PreCheck Eurip.com
EuripBot/0.2 (+http://www.eurip.com) GetRobots
EuripBot/0.4 (+http://www.eurip.com) GetFile
EuripBot/0.5 (+http://www.eurip.com) PreCheck
Euro-spider Euro-Spider Shopping 1.0 Euro-spider
Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) Evaal
EvaalSE EvaalSE - bot@evaal.com Evaal
Eventax eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Eventax
Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra
wlerhelp)
Vulcan
Exabot Exabot/2.0 Exalead
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
Exalead NG/MimeLive Client (convert/http/0.120)
Exabot/3.0
ExaBotTest/3.0
ExaBotTest/2.0
NG/2.0
Exabot-Test/1.0
Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko)
Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) Exalead
Exabot-Images/1.0
NG/4.0.1229
ExactSEEK ExactSeek Crawler/0.1 ExactSEEK
exactseek.com
eseek-larbin_2.6.2 (crawler@exactseek.com)
exactseek-pagereaper-2.63 (crawler@exactseek.com)
exactseek-crawler-2.63 (crawler@exactseek.com)
ExactSeek_Spider ExactSeek_Spider ExactSeek
Excalibur Excalibur Internet Spider V6.5.4 Convera
Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) Execrawl
FAST-WebCrawler FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support
/crawler.asp)
FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch
/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/
faqfastwebcrawler.html)
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net)
FAST Enterprise Crawler/6.4.18 (crawler@fast.no)
FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw
ebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)
FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa
qfastwebsearch/faqfastwebcrawler.html)
fastlwspider/1.0
FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net)
FDSE Mozilla/4.0 (compatible; FDSE robot) Abadoor
FaXobot Faxobot/1.0 Faxo
Factbot factbot : http://www.factbites.com/robots Factbites
Factbot 1.09 (see http://www.factbites.com/webmasters.php)
Fast Search PycURL FAST
Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de) Fastbot
Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu) Favo
Feed24 Feed24.com Feed24
FeedChecker FeedChecker/0.01 University of Tokyo
Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Google
Feedster Crawler Feedster Crawler/3.0; Feedster, Inc. Feedster
Felix Felix - Mixcat Crawler (+http://mixcat.com) MixCat
Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi
langy-agent@filangy.com)
Filangy
FindLinks http://wortschatz.uni-leipzig.de/findlinks/ University of Leipzig
findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) Findexa
FineBot FineBot Finesearch
Firefly Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Fireball
Firefly/1.0
FirstGov FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov U.S.Government
Firstsbot firstsbot Firstsfind
Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio
d com)
Flaptor
Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.
flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
Flatland Industries
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan
dindustries.com/flatlandbot.php; jason@flatlandindustries.com)
flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu
stries.com/flatlandbot.php; jason@flatlandindustries.com)
FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3 DivX.com
Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se
archhippo.com)
Searchhippo
FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) Fnoole
Folkd.com Spider Folkd.com Spider/0.1 beta 1 (www.folkd.com) Folkd
ForAll.pl-Crawler ForAll.pl-Crawler/1.0 ForAll
Francis Francis/1.0 (francis@neomo.de http://www.neomo.de/) Neomo
FreshNotes crawler FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com FreshNotes
FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com
Freshmeat freshmeat.net URL validator/1.1 Freshmeat
FuchsBot FuchsBot +http://www.fuchsbot.tld FuchsBot
FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn
.furlbot@looksmart.net)
Furl
FuseBulb FuseBulb.Com FuseBulb
FyberSpider FyberSpider (+http://www.fybersearch.com/fyberspider.php) FyberSearch
GAIS Robot GAIS Robot/1.0B2 Seed
GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co
m)
Unknown
GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin
e.php)
GPU
GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) GSiteCrawler
Gaaz gazz/x.x (gazz@nttrd.com) Infobee
Gaisbot Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gais
Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot Mozilla/4.0 (compatible; www.galaxy.com) Galaxy
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) Uchoose
GammaSpider GammaSpider/1.0 Gammasite
GenieKnows geniebot wgao@genieknows.com GenieKnows
larbin_2.6.3 (wgao@genieknows.com)
Mozilla/5.0 wgao@genieknows.com
Mozilla/5.0 (wgao@genieknows.com)
GeonaBot GeonaBot 1.x; http://www.geona.com/ Geona
Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu) Georgia Institute of Technology
Geourl Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot) Geourl
GigaBaz Brainbot gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) Gigabaz
MicroBaz
Gigabot Gigabot/2.0; http://www.gigablast.com/spider.html Gigablast
Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gigabot/2.0att
Gigabot/2.0/gigablast.com/spider.html
Gigabot/2.0
Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf
a dot com; http://www.girafa.com)
Girafa
GlobalQueue Look.com Multi-mode
GnodSpider GNODSPIDER (www.gnod.net) Gnod
GoForIt GOFORITBOT ( http://www.goforit.com/about/ ) GoForIt
GoForIt.com
Goblin Goblin/0.9 (http://www.goguides.org/) GoGuides
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)
Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp T-info
Gonzo2 gonzo2[P] mailto:crawleradmin.t-info@telekom.de T-info
gonzo1[P] mailto:crawleradmin.t-info@telekom.de
gonzo2[P] +http://www.suchen.de/faq.html
Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
 
 
Google-Adsense Mediapartners-Google Google
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
Mediapartners-Google/2.1
Google-Image Googlebot-Image/1.0 Google
Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Google-Sitemaps Google-Sitemaps/1.0 Google
Google-WAP Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Google
Google WAP Proxy/1.0
GoogleBot Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Google
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible;
Googlebot/2.1; http://www.google.com/bot.html)
Googlebot/Test ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog
lebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Googlebot/1.0 (googlebot@googlebot.com)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
Googlebot/1.0
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/2.0 beta (googlebot@googlebot.com)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b
ot.html)
Google
KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google
bot-Mobile/2.1; +http://www.google.com/bot.html)
Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) Greaterera
GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html) Grigor
Gromit Gromit/1.0 Australasian Legal Information Institute
Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru
b.org)
Grub
Gsa-crawler gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) IBM
gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw)
Gulliver Gulliver/1.2 Northernlight
Gulliver/1.3
GulperBot Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim
/cgi-bin/Link/GulperBot)
University of New-York
Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) Gungho
GurujiBot GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) Guruji
GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
Harvest-NG Harvest-NG/1.0.2 Harvest-NG
Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Hatena Antenna
HatenaScreenshot HatenaScreenshot/1.0 (checker) Hatena
HatenaScreenshot/1.0 (checker)
Hbtronix.spider hbtronix.spider.2 -- http://hbtronix.de/spider.php Hbtronix
HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Mirago
Helix Helix/1.x (+http://www.sitesearch.ca/helix/) SiteSearch
HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Mirago
HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) Mirago
Heritrix mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org) University of Washington
archive.org_bot
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs
.washington.edu/research/networking/websys/)
Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec
ts/crawling/)
L3S Research Center
Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) Chepi
Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) Hermits Search
Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider Hiigle
Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com Hitwise
Holmes holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl) Szukaj.onet
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/x.x
holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
HolmesBot HolmesBot (http://holmes.ge) Holmes
HomePageSearch HomePageSearch(hpsearch.uni-trier.de) HomePageSearch
Homerbot Homerbot: www.homerweb.com Homerweb
Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda
-search.com)
Honda-Search
Hoowwwer HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info
hiit.fi)
NGIR
HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:
crawler-infohiit.fi)
Htdig htdig/3.1.x (root@localhost) ht:/dig
Htdig/3.1.6 htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) Acad?mie de Toulouse
I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search
.com)
I1search
ICC-Crawler ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con
tact(at)ml(dot)nict(dot)go(dot)jp)
NICT
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a
t)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c
ontact(at)ml(dot)nict(dot)go(dot)jp)
ICCrawler ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) ICCenter
ICRA_Label_spider ICRA_label_spider/x.0 Icra
IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Id-search
IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a
t iiit dot ac dot in)
Webkhoj
INGRID Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster.
ilse.nl/jsp/webmaster.jsp)
Ilse
IP2MapBot IP2MapBot/1.1 http://www.ip2map.com Ip2Map
IPiumBot IPiumBot laurion(dot)com Laurions
IRLbot IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) Texas A&M University
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
IWAgent IWAgent/ 1.0 - www.brandprotect.com Brandprotect
Iaskspider2 iaskspider2 (iask@staff.sina.com.cn) Sina
Ichiro ichiro/1.0 (ichiro@nttr.co.jp) Goo
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/1.0 (ichiro@nttr.co.jp)
IconSurf IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) IconSurf
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
Icsbot icsbot-0.1 International Christian school of Seoul
Ideare ideare - SignSite/1.x Ideare
IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator
e.it)
IlTrovatore
Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo
r more information please visit http://www.ilial.com/crawler; http://www.ilial.c
om/crawler; crawl@ilial.com)
Ilial
ilial/Nutch-0.9-dev
Ilse Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld
en.ilse.nl/?aanmeld_mode=webhints)
Ilse
Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.
ilse.nl/?aanmeld_mode=webhints)
ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com) Bdbrandprotect
IncyWincy NetResearchServer/x.x(loopimprovements.com/robot.html) LoopImprovements
IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement
s.com/robot.html)
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy/2.1(loopimprovements.com/robot.html)
IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen
ts.com/robot.html)
IncyWincy (Look) IncyWincy(http://www.look.com) Look
IndexTheWeb IndexTheWeb.com Crawler7 IndexTheWeb
Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia
Interactive
Indonesia Interactive
InelaBot InelaBot/0.2 (+http://inelegant.org/bot) Inelegant
Inet Library Inet library Inet Library
InfoFly InfoFly/1.0 (http://www.versions-project.org/) Versions-project
InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Stanford University
InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht
ml; lhall@telegenetic.net)
Telegenetic
Infoseek InfoSeek Sidewinder/0.9 Go
Inria larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Inria
Insitor Search robot Insitor.com search and find world wide! Insitor
Insitornaut Insitornaut Insitor
Internet Ninja Internet Ninja x.0 Dream Train Internet
Internetseer InternetSeer.com Internetseer
Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) Iprospect
IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) Ipselon
Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot
com)
Iseek
Ishida Lab larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Kyoto University
It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt
rovatore.it/bot.html; bot@iltrovatore.it)
IlTrovatore
Jabot Jabot/7.x.x (http://odin.ingrid.org/) ODIN Directory
Jabot/6.x (http://odin.ingrid.org/)
Jambot Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot
; crawler@jambot.com)
Jambot
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Jayde Crawler Jayde Crawler. http://www.jayde.com Jayde
Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) Sidedc
Jetbot Jetbot/1.0 JetEye
Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de)
( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge
n/search/robot.html )
Neofonie
Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za; info@jonga.co.za) Jonga
Jyxobot Jyxobot/1 Jyxo
Jyxobot/x
K2 Spider k2spider Verity
KAIST AITrc Crawler KAIST AITrc Crawler AITrc
KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de) KFSW
KIT_Fireball KIT_Fireball/2.0 Dino-online
KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte
nt/crawler/index.html; crawleradmin@knowledgestorm.com)
Knowledgestorm
KakleBot KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bo
t.html; support@kakle.com)
akle
KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw
ler@kalooga.com)
kalooga
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com)
Kasparek Firefox_1.0.6 (kasparek@naparek.cz) Czech Technical University Prague
Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) Keegee
Kenjin Spider Kenjin Spider Kenjin
Kevin Kevin http://dznet.com/kevin/ Dznet.com
Kevin http://websitealert.net/kevin/
KicktooBot kicktooBotV1.1 kictooBot@kictoo.com Kicktoo
Kinja-imagebot kinja-imagebot (http://www.kinja.com/) Kinja
Kinjabot kinjabot (http://www.kinja.com) Kinja
KnowItAll KnowItAll(knowitall@cs.washington.edu) University of Washington
Knowledge.com Knowledge.com/0.x knowledge.com
Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info
.html; webcrawler@krugle.com)
Krugle
Kulokobot kulokobot www.kuloko.com kuloko@backweave.com Kuloko
kuloko-bot/0.x
Kulturarw kulturarw3/0.1 National Library of Sweden
Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez) Sanoma
Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k
yluka.com)
Kyluka
LECodeChecker LECodeChecker/3.0 libgetdoc/1.0 Linkexchange
LNSpiderguy LNSpiderguy Lexis-Nexis
LapozzBot LapozzBot/1.5 (+http://robot.lapozz.hu) Lapozz
LapozzBot/1.4 (+http://robot.lapozz.hu)
LapozzBot/1.4 ( http://robot.lapozz.com)
Larbin_2.6.3 larbin_2.6.3 larbin2.6.3@unspecified.mail Unknown
larbin_2.6.3 marzia.polito@intel.com
Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo
.com; webmaster@lawinfo.com)
Lawinfo
Lemur Consulting larbin_2.6.2 (tom@lemurconsulting.com) Lemur Consulting
Lexibot Mata Hari/2.00 BrightPlanet
LexiBot/1.00
Liafa larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Liafa
LibWeb libWeb/clsHTTP -- hiongun@kt.co.kr Korea Telecom
LibertyW LibertyW (+http://www.lw01.com) LibertyW
LibertyW (+http://www.libertyw.eu)
LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co
m)
Lijit
LinkWalker LinkWalker Seven TwentyFour
Linknzbot linknzbot LinkNZ
Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Links2Go
Links4US-Crawler Links4US-Crawler, (+http://links4us.com/) Links4US
LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck
er.html)
Unknown
Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) Universitat de les Illes Balears
Lmspider lmspider (lmspider@scansoft.com) Nuance
LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/) LocalBot
Lockstep Spider Lockstep Spider/1.0 Lockstep
Look.com NetResearchServer(http://www.look.com)  
LookdirBot LookdirBot Lookdir
Lovel Lovel as 1.0 ( +http://www.everatom.com) Everatom
Ltaa_web_crawler larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Ecole Polytechnique F?d?rale de Lausanne
Luchs.at URL checker luchs.at URL checker Luchs
Lycos_Spider Lycos_Spider_(modspider) Lycos
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide
r_(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_
Spider_Beta2(T-Rex) )
Lycos_Spider_(T-Rex)
MJ12bot MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Majestic 12
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.ui
uc.edu)
University of Illinois
MSN Bot msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) MSN
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot/0.9 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSN
MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot) Microsoft
MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/) Microsoft
MSRBOT (http://research.microsoft.com/research/sv/msrbot/
MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Sagool
Mail.Ru Mail.Ru/1.0 Mail.Ru
Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT) Mainseek
Mammoth Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 SLI Systems
mammoth/1.0 (+http://www.sli-systems.com/)
Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
MantraAgent MantraAgent LookSmart
Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Kolibri
Martini Martini LookSmart
MARTINI
Marvin Marvin v0.3 Health On the Net Fondation
Masterseek MasterSeek Masterseek
Maxbot Spider/maxbot.com admin@maxbot.com Maxbot
Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html;
maxomobot@maxomo.com)
Maxomo
MediaCrawler MediaCrawler-1.0 (Experimental) Media Find
MediaSearch MediaSearch/0.1 WWW.FI
Mediater Rechercher libwww/5.3.2 Mediater
MegaSheep MegaSheep v1.0 (www.searchuk.com internet sheep) Search UK
Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) Megaglobe
Melbot WebSpider Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.) Melbot
Mercator Mercator-1.x Altavista
Mercator-Scrub-1.1
Mercator-2.0
Merl.com larbin_2.1.1 larbin2.1.1@somewhere.com Mitsubishi Electrical Research Lab
MetaGer_PreChecker MetaGer_PreChecker0.1 MetaGer
Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) Metacarta
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)
Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta
euro.com; crawler at metaeuro dot com)
Metaeuro
Metager-Linkchecker MetaGer-LinkChecker Metager
MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Metager
Metaquerier MQbot http://metaquerier.cs.uiuc.edu/crawler University of Illinois
MQbot metaquerier.cs.uiuc.edu/crawler
Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner
.de/)
Metaspinner
Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/) Metatagsdir
Microsoft Small Business Indexer Microsoft Small Business Indexer Microsoft
Microsoft URL Control Microsoft URL Control - 6.00.8862 Unknown
Microsoft URL Control - 6.01.9782
Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/)
miggibot/2006.01.24
Micro-sys
Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s
ee getmirar.com)
Mirar
Missigua Locator 1.9 Missigua Locator 1.9 Unknown
Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot
.fr)
Misterbot
Miva Miva (AlgoFeedback@miva.com) Miva
MnoGoSearch MnogoSearch/3.2.xx MnoGoSearch
Mo College 1.9 Mo College 1.9 Unknown
Moget moget/x.x (moget@goo.ne.jp) Goo
Mogimogi mogimogi/1.0 Goo
MojeekBot Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) Mojeek
MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)
Monogol http://www.monogol.de Monogol
Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com) Moreover
Morris Morris - Mixcat Crawler (+http://mixcat.com) MixCat
Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.
com/bot)
Mowser
MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) MozDex
Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com) MozDex
MultiText MultiText/0.1 Virginia Polytechnic Institute and State University
Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) SWSE
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) Euro Ferret
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)
Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)
MusicWalker MusicWalker2.0 (+http://www.somusical.com) SoMusical!
My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info) My-byte
MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx
; SearchBot@MyFamilyInc.com)
Ancestry
Mylinea Mylinea.com Crawler 2.0 Mylinea
NASA Search 1.0 NASA Search 1.0 Unknown
NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) National Center for Supercomputing Applications
NG-SearchBot NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) NG-Search
NG-Search/0.9.8 (http://www.ng-search.com)
NII larbin_2.6.2 (hamasaki@grad.nii.ac.jp) National Institut of Informatics (Japan)
NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.
89 (like Gecko)
Northern Light
NMG Spider NMG Spider/0.3 (szukanko.com) Szukanko
NTT Directory nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) NTT Directory
NWSpider NWSpider 0.9 NWSpider
Nabot nabot_1.0 Naver
NABOT/5.0
Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x) Najdi
Nameprotect NPBot (http://www.nameprotect.com/botinfo.html) Nameprotect
NPBot-1/2.0
NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)
Nationaldirectory NationalDirectoryAddURL/1.0 Nationaldirectory
NationalDirectory-WebSpider/1.3
NatzanBot NatzanBot, http://www.natzan.com Natzan
NaverBot NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver
NaverBot_dloader/1.5
dloader(NaverRobot)/1.0
NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
NavissoBot NavissoBot/1.7 (+http://navisso.com/) Navisso
NavissoBot
Nebulla Nebullabot/2.2 (http://bot.nebulla.info) Nebulla
Nebullabot/2.2 (http://bot.nebulla.de)
Nebulla/V1.0 Spider (http://spider.nebulla.de)
Nelian Pty Ltd - Spider Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com ) Pcaccessoriesparts
NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience:
http://www.netnose.com)
NetNose
NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html) Loopimprovements
NetSprint -- 2.0 NetSprint -- 2.0 Wirtualna Polska
NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com
; support@netwhat.com)
Netwhat
NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) NetinfoBot
Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) Netluchs
Netprospector Netprospector JavaCrawler Netprospector
Netscape Robozilla/1.0 DMOZ
NextGenSearchBot NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe
nSearchBot.aspx)
Zoominfo
NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot
)
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB
ot)
NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1 Nextopia
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x
NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Healthline
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob
lems contact: crawler@healthline.com
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe
ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Noago Spider Noago Spider Noago
NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm) Nokodo
Norbert Norbert the Spider(Burf.com) Burf
Noxtrumbot noxtrumbot/1.0 (crawler@noxtrum.com) Noxtrum
Noyona noyona_0_1 Noyona
Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) Nsyght
NuSearch Spider nuSearch Spider www.nusearch.com (compatible
; MSIE 4.01)
NuSearch
Nutch Nutch Lucene
NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis
ts.sourceforge.net)
NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy
crawl@cs.washington.edu)
University of Washington
NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o
rg/nutch/bot.html; ec2test at lucene.com)
Amazon
Obidos-bot obidos-bot (just looking for books.) Weblog bookwatch
Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3
E
 
Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup
port@thesoftwareobjects.com)
SAIC
Ocelli Ocelli/1.x (http://www.globalspec.com/Ocelli) GlobalSpec
Ocelli/1.3 (http://www.globalspec.com/Ocelli)
Octora Octora Beta - www.octora.com Octora
Octora Beta Bot Octora Beta Bot - www.octora.com Octora
OmniExplorer OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer OmniExplorer
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler
OnetSzukaj Onet.pl SA, http://szukaj.onet.pl Szukaj
Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)
OpenISearch OpenISearch/1.x (www.openisearch.com) OpenISearch
OpenPortal4U DigOut4U Arisem
OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) OpenTagger
OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2 OpenText
OpenWebSpider OpenWebSpider/x OpenWebSpider
Openfind Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht
ml)
Openfind
Openfind Robot/1.1A2
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.
openfind.com.tw/robot.html)
OpidooBot OpidooBOT (larbin2.6.3@unspecified.mail) Opidoo
Oracle Oracle Ultra Search Oracle
OrangeSpider OrangeSpider Orangeslicer
Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor
t/crawler.asp)
FAST
OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) OzMonitor
Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat
.de)
Flying-cat
PJspider PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Nextopia
PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) Programacionweb
Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Page-store
PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/) PageBites
Page_verifier page_verifier http://www.securecomputing.com/goto/pv Securecomputing
Pagebull Pagebull http://www.pagebull.com/ Pagebull
Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com) Pages Jaunes
FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr)
PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi
on 0.001
Links24h
Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/
crawl.html)
National Library of Australia
ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/) Ianett
Patwebbot Patwebbot (http://www.herz-power.de/technik.html) Patsearch
Peerbot PEERbot www.peerbot.com Peerbot
Petitsage www.petitsage.fr site detector 0.4 Petitsage
PicoSearch PicoSearch/1.0 Pico Search
PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org) Malfunction
Piffany Piffany_Web_Spider_v0.x Piffany
Piffany_Web_Scraper_v0.x
Pilot Hitlist Marketwave Hit List Pilot Hitlist
Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com) Pingdom
PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Pipeline
Pita Pita Stanford University
PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 dhkang@plantynet.com PlantyNet
PlantyNet_WebRobot_V1.9 yangsam@plantynet.com
PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1
subscribers)
Pluck
Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin
fo.htm)
Pmoz
Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) Podtech
Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x) Pogodak
Polybot polybot 1.0 (http://cis.poly.edu/polybot/) Polytechnic University Brooklyn
Pompos Pompos Iliad (Free)
Pompos/1.3 http://dir.com/pompos.html
Pompos/1.x pompos@iliad.fr
Pompos/1.1 http://dir.com/pompos.html
Pompos/1.2 http://dir.com/pompos.html
Pompos/1.x http://dir.com/pompos.html
PopJapanSearch Robot/www.pj-search.com PopJapanSearch
Popdexter Popdexter/1.0 Popdex
PrivacyFinder/1.1 PrivacyFinder/1.1 AT&T Privacy Bird Privacy Preferences
ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com) ProWebguide
Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi
a.com)
Probert Encyclopaedia
Psbot psbot/0.1 (+http://www.picsearch.com/bot.html) Picsearch
Psycheclone psycheclone Unknown
Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca Pubblisito
Pumpkin blogsearchbot-pumpkin-3 Artofcomputing
Python-urllib Python-urllib/1.16 Unknown
QPCreep QPCreep Test Rig ( We are not indexing, just testing ) Quepasa
QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n
et)
Qihoo
Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) Quantcast
Qube qube/2.0 (+http://qube.qelix.com/v2) Qube
QuepasaCreep QuepasaCreep v0.9.1x Quepasa
QuepasaCreep ( crawler@quepasacorp.com )
QueryN Metasearch QueryN Metasearch QueryN
Qweerybot QweeryBot/3.01 ( http://qweerybot.qweery.nl) Qweery
Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)
QweeryBot/3.02 ( http://qweerybot.qweery.nl)
R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6
R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6
RAMPyBot RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu
cene.apache.org)
GiveRamp
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveram
p.com/bot.html; support@giveRAMP.com)
RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com) RSS One
Rambot ramBot xtreme x.x Intersearch
RankMeter Web Snooper SearchUtilities
Rational SiteCheck Rational SiteCheck (Windows NT) GSInnova
Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper) Marty Anstey
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) Rediff
RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Pronto
RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) Telegenetic
RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) RedKernel
RoboPal RoboPal (http://www.findpal.com/) FindPal
Rotondo Rotondo/3.1 libwww/5.3.1 QualiGo
RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) Webaroo
RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html) Aktienbetreuer
SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Samsung
SBIder SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site
sell.com/contact-support.html)
Sitesell
SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell
.com/contact-support.html)
SandCrawler SandCrawler - Compatibility Testing Microsoft
Savvybot savvybot/0.2 WebSavvy
ScanWeb ScanWeb Eserver
Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/
/www.schibstedsok.no/bot/)
Schibsted
ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho
laruniverse.com)
ScholarUniverse
Scirus-crawler FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com
/srsapp/contactus/
Scirus
ScollSpider ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Webwobot
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots Fooky
ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ Ant
Scrubby Scrubby/2.x (http://www.scrubtheweb.com/) Scrub the web
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache
.org/nutch/bot.html; researchbot@shopping.com)
Shopping
Search-Engine-Studio Search-Engine-Studio Xtreeme
Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Search-Info
Search.ch search.ch V1.4 Search CH
search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)
Search4free lwp-trivial/1.34 Search4free
SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs
a.com)
Search4USA
SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
 
SearchEngineWorlds
SearchExpress Spider SearchExpress Spider0.99 SearchExpress
SearchScout ClariaBot/1.0 SearchScout
Diamond/1.0
DiamondBot
SearchSight SearchSight/2.0 (http://SearchSight.com/) SearchSight
SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi
der.com)
SearchSpider
SearchSpider.com/1.1
SearchdayBot SearchdayBot Searchday
Searchguild SearchGuild_DMOZ_Experiment (chris@searchguild.com) SearchGuild
Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Searchit-Now
Searchmee! Spider Searchmee! Spider v0.98a Searchmee!
Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 SeekPort
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seeker.lookseek Seeker.lookseek.com Lookseek
Semager Semager/1.0 (http://www.semager.de) Semager
Semager/1.1 (http://www.semager.de/blog/semager-bots/)
Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) Sensis
SeznamBot SeznamBot/2.0-test (+http://fulltext.sblog.cz/) Seznam
SeznamBot/1.0 (+http://fulltext.seznam.cz/)
SeznamBot/1.0
SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) SharewarePlaza
Sherlock sherlock/1.0 Indiana University School of Informatics
Shim-Crawler Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl
er/; crawler@logos.ic.i.u-tokyo.ac.jp)
Chikayama-Taura laboratory
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c
rawl@logos.ic.i.u-tokyo.ac.jp)
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra
wl@logos.ic.i.u-tokyo.ac.jp)
ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/) Littlewiki
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Shoula Shoula.com Crawler 2.0 Shoula! Search
SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz) Siets
Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com) Siigle
Silk/1.0 silk/1.0 Slider
silk/1.0 (+http://www.slider.com/silk.htm)/3.7
Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Sirketce
Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR American College of Radiology
SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) SiteBar
SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru) SiteBase
SiteSpider SiteSpider +(http://www.SiteSpider.com/) SiteSpider
SiteTruth SiteTruth.com site rating system SiteTruth
SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) SitiDi.net
Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] Skaffe
Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE
.com)
Skizzle
SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) SkreemR
Slider Slider_Search_v1-de Slider
Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)
Yahoo
Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
 
Yahoo
Slurp Inktomi Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Hotbot-Lycos,NBCi etc.
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp Inktomi (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu
rp)
Yahoo
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071
3 Firefox/2.0.0.5
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch
/slurp)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE
cho/2.0.0.4
Slurpy Verifier Slurpy Verifier/1.0 Yahoo
SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0 Smiffysplace
Snapbot Snapbot/1.0 Hotlinks
Snoopy Snoopy v1.2 Unknown
SnykeBot SnykeBot/0.6 (http://www.snyke.com) Snyke
SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) SoftHypermarket
Sogou web spider Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou
Sohu-search sohu-search Sohu
Somewhere Mozilla (Mozilla@somewhere.com) Somewhere
Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Unknown
Speedfind speedfind ramBot xtreme 8.1 Lotse
Speedy Spider Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Entireweb
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Sphere Scout Sphere Scout&v4.0 - scout at sphere dot com Sphere
Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg
)
Singapore Press Holdings
SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) SpiderMonkey
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
SplatSearch libwww-perl/5.45 Splat
Spock Crawler Spock Crawler (http://www.spock.com/crawler) Spock
Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr
oose.com)
Sproose
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c
om)
StackRambler StackRambler/2.0 Rambler
StackRambler/x.x
StackRambler/2.0 (MSIE incompatible)
Steeler Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) University of Tokyo
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com) Strategic Board
Suchbaer suchbaer.de (CrawlerAgent v0.103) Suchbaer
suchbaer.de
Suchpad suchpad/1.0 (+http://www.suchpad.de) Suchpad
SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Summize
SuperSnooper Robot@SuperSnooper.Com SuperSnooper
SurveyBot SurveyBot/2.3 (Whois Source) Domaintools
Susie !Susie (http://www.sync2it.com/susie) Sync2it
Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) University of Maryland
SycaBoT SycaBoT/1.0 SycaroX
SycaBoT-Audio SycaBoT-Audio SycaroX
SycaBoT-Image SycaBoT-Image SycaroX
SycaBoT-Programme SycaBoT-Programme SycaroX
SycaBoT-Video SycaBoT-Video SycaroX
Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr
awl@cs.washington.edu)
University of Washington
Sygol http://www.sygol.com Sygol
SygolBot SygolBot http://www.sygol.com Sygol
SygolBot http://www.sygol.net
SynooBot SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.
html)
Synoo
Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)
Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Syntryx
Szukacz Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) Szukacz
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl)
TAGword Tagword (http://tagword.com/dmoz_survey.php) TAGword DMOZ survey
TCDBOT TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot
IE)"
Trinity College Dublin
Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/) Tags2dir
TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta
ilrank.com/robot) Gecko/20021130
TailRank
Talkro Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w
ebshot)
DaumSoft
TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/
TargetSeek.html)
Targetgroups
Technoratibot Technoratibot/0.7 Technorati
Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm) Tecomi
Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w
ww.netseer.com/crawler.html; crawler@netseer.com)
Netseer
TelenetDigger TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be) Telenet
Teoma teoma_agent1 Teoma
Teoma MP
teomaagent crawler-admin@teoma.com
teomaagent1 [crawler-admin@teoma.com]
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Teradex Mapper Teradex Mapper; mapper@teradex.com; http://www.teradex.com Teradex
TeragramCrawler TeragramCrawler Teragram
TeragramWebcrawler TeragramWebcrawler/1.0 Teragram
TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Terrawiz
TheSuBot TheSuBot/0.1 (www.thesubot.de) TheSuBot
Theme Spider Theme Spider (+http://www.themespider.com/spider.html) Themespider
Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th
eophrastus.php)
N.A.Smith
Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) Thumbshots
Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Thunderstones
TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html Breakingblogs
TivraSpider tivraSpider/1.0 (crawler@tivra.com) AT&T
Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html) Tkensaku
ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 La Toile du Qu?bec
Topodia Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia
.com/; support@topodia.com)
Topodia



Traazibot traazibot/testengine (+http://www.traazi.de) Traazi
Trampelpfad Trampelpfad-Spider Trampelpfad
Trampelpfad-Spider-v0.1
TrendTech http://www.trendtech.dk/spider.asp) TrendTech
Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) Truveo
Turnitin TurnitinBot Turnitin
TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) Turnitin
Turnpike Emporium Turnpike Emporium LinkChecker/0.1 Turnpike Emporium Directory
TutorGigBot TutorGig/1.5 (+http://www.tutorgig.com/crawler) TutorGig
Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)
Twiceler Twiceler www.cuill.com/twiceler/robot.html Cuill
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
Twiceler-0.9 http://www.cuil.com/twiceler/robot.html
Twiceler www.cuil.com/twiceler/robot.html
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
TygoBot TygoProwler Tygo
TygoBot
TÜzilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez
illa.de/t_st-odp-entries-agent.html)
TÜzilla
UKWizz Mackster( http://www.ukwizz.com ) UKWizz
UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
UN OCHA FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA) UN OCHA
UOLCrawler UOLCrawler (soscrawler@uol.com.br) UOL
URL-Spider Mozilla/5.0 URL-Spider URL-Spider
URL_Spider_Pro URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) Innerprise
URL_Spider_Pro/x.x
URL Spider Pro/x.xx (innerprise.net)
USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) University of Sydney
Ultraseek Ultraseek Go
UnChaosBot UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc
haos.com
(info@unchaos.com)
UnCHAOS
UnChaos Bot, Hybrid Web Search Engine. (
vadim_gonchar@unchaos.com)
UnChaos , From Chaos To Order, Hybrid Web
Search Engine. (vadim_gonchar@unchaos.com)
Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt University of Koblenz
Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html University of Dortmund
Updated updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) Updated
Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
 
Updated
Uptimebot Uptimebot Uptimebot
UptimeBot(www.uptimebot.com)
Urlchecker1.0 Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) Euro Directory
Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html Urlfan
VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com) VBSEO
VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; info@vipsolutions.hu) Vipsolutions
VMBot VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) VerticalMatch
VMBot/0.9 (VMBot; http://www.verticalmatch.com; vmbot@tradedot.com)
VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu
c.edu
University of Illinois
Vacobot Vacobot; (+http://vaco.ws/bot.html) Vaco
Vagabondo Vagabondo/3.0 (webagent at wise-guys dot nl) WiseGuys
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http
://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w
ebagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt
p://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://
webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/
)/1.0 Profile
WiseGuys
Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) Vakes
Verizon fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt
ers@fastsearch.com)
Verizon Superpages
FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad
min@superpages.com)
Vermut Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com) AOL
mozilla/5.0 (compatible; vermut +http://www.aol.com)
Mozilla/5.0 (compatible; vermut +http://vermut.aol.com)
Versus versus 0.2 (+http://versus.integis.ch) Hochschule f?r Technik Rapperswil
Versus Crawler versus crawler eda.baykan@epfl.ch Ecole Polytechnique F?d?rale de Lausanne
VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs VeryGoodSearch
Verzamelgids verzamelgids.nl - Networking4all Bot/x.x Verzamelgids
Verzamelgids/2.2 (http://www.verzamelgids.nl)
Vespa Crawler (Yahoo) Vespa Crawler Yahoo! Norway
Visbot Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com) Visvo
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) Freshnotes
VoilaBot KE_1.0/2.0 libwww/5.2.8 Voila.fr
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor
t.voilabot@orange-ftgroup.com)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/
/www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi
la.com/)
Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) Vonna
Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) Marty Anstey
Voyager cfetch/1.0 Kosmix
voyager-hc/1.0
voyager/1.0
Vspider vspider Verity
vspider/3.x
W3C-Validator W3C_Validator/1.432.2.10 W3C
W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803 W3C
W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de W3SiteSearch
W8net Mozilla/5.0 usww.com-Spider-for-w8.net usww
WEP Search WEP Search 00 Unknown
WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) CWR
WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)
WISEbot WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Koreawisenut
WWW.fi crawler www.fi crawler, contact crawler@www.fi WWW.FI
WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de) World Wide Weasel
Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/) Wadaino
Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) Wavefire
Waypath Waypath development crawler - info at waypath dot com Waypath
Waypath Scout v2.x - info at waypath dot com
WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer
elate.org)
WeRelate
WeatherBot WeatherBot v1.4 http://www.ezweather.net EZ Weather
WebAlta Crawler WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT
5.1; ru-RU)
WebAlta
WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot
/index.html
Carnegie Mellon University
WebCorp WebCorp/1.0 WebCorp
WebFindBot WebFindBot(http://www.web-find.com) web-find
WebGobbler webGobbler/1.x.x WebGobbler
WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) WebRankSpider
WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S
earch@WebSearch.COM.AU)
WebSearch
WebSearchBench WebSearchBench WebCrawler v0.1(Experimental) University of Dortmund
WSB, http://websearchbench.cs.uni-dortmund.de
WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsität Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebStat WebStat/1.0 (Unix; beta; 20040314) University of South Carolina
WebVac WebVac (webmaster@pita.stanford.edu) Stanford University
WebarooBot WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) Webaroo
WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)
Webbandit webbandit/4.xx.0 SoftwareSolutions
Webbot Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html) Webbot
webbot(+http://webbot.com/bot.htm)
Webclipping Webclipping.com Webclipping
Webcrawl webcrawl.net Webcrawl
webcrawl.net
Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) Webdunia
Webglimpse Webglimpse 2.xx.x (http://webglimpse.net) Webglimpse
Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0 Los Alamos National Laboratory
Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de University of Leipzig
Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ University of Dortmund
WebsiteWorth WebsiteWorth v1.0 Sootle
Webspinne Webspinne/1.0 webmaster@webspinne.de Webspinne
Websquash Websquash.com (Add url robot) Websquash
Webster Webster v0.3 ( http://webster.healeys.net/ ) Webster.healeys
Webverzeichnis Webverzeichnis.de - Telefon: 01908 / 26005 Webverzeichnis
Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/] Axxus
Willow Willow Internet Crawler by Twotrees V2.1 Twotrees
WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ?
WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB
ot; winkbot@wink.com)
Wink
WiseGuys Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi
se-guys.nl/)
WiseGuys
Vagabondo/2.0 MT
Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.
nl/)
Vagabondo/1.x MT (webagent@wise-guys.nl)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Worio Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) Worio
Worio bot Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com) Worio
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Worio heritrix bot worio heritrix bot (+http://worio.com/) Worio
WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) WorldWideWeb
Wotbox Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 Wotbox
Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)
Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Lehigh University
Wwlib/Linux Wwlib/Linux Wolverhampton Web Library
Wwwster wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) Centrum f?r Informations- und Sprachverarbeitung
XP5 Project XP5 [2.03.07-111203] Marty Anstey
Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html) Xdefine
Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) Xirq
Xyleme SA France cosmos/0.9_(robot@xyleme.com) Xyleme
cosmos/0.8_(robot@xyleme.com)
Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Yahoo Japan
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yacy yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/
en)
Yacy
Yacybot yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net Yacy
yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net
yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net
Yahoo Japan robot DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Yahoo Japan
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Y!J-SRD/1.0
Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo Japan
Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) Yahoo
YahooYSMcm/2.0.0
Yahoo! Mindset Yahoo! Mindset Yahoo! Mindset
Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/
us/ysearch/crawling/crawling-02.html )
Yahoo
Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co
m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo
Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Yahoo
Yahoo-Test/4.0 Yahoo-Test/4.0 Yahoo
Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com;
http://www.alltheweb.com/
Yahoo
YahooFeedSeeker YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c
om/rssguide)
Yahoo
YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu
blishers.html)
YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web
master@yahoo-inc.com ) (KHTML, like Gecko)
Yahoo
YahooSeeker/M1A1-R2D2 SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i
nc dot com)
Yahoo
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile
-search-customer-care AT yahoo-inc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-
01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in
c dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo.
com/help/us/ysearch/crawling/crawling-01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/
crawling/crawling-01.html)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot
com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch
/crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo
.com/help/us/ysearch/crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust
omer-care AT yahoo-inc dot com)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli
ng/crawling-01.html)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http:
//help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo
-inc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com)
Yandex Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) Yandex
Yandex/1.01.001 (compatible; Win16; I)
Yarienavoir yarienavoir.net/0.2 Yarienavoir
Yellopet spider.yellopet.com - www.yellopet.com Yellopet
Yeti Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) Naver
Yeti
Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)
Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m
inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres
den dot de)
GoPubMed
YodaoBot Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide
r/; )
Yodao
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu) YooW!
Yoogli yoogliFetchAgent/0.1 Yoogli
Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) Yoono
Yoono web-crawler yoono/1.0 web-crawler/1.0 Yoono
YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine YottaCars
YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine YottaShopping
Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) Z-add
Zao Crawler Zao-Crawler Kototoi
Zao-Crawler 0.2b
Zao/0.1 (http://www.kototoi.org/zao/)
ZeBot ZeBot_www.ze.bz (ze.bz@hotmail.com) ZE.BZ
ZeBot_lseek.net (bot@ze.bz)
Zealbot Mozilla/4.0 (compatible; Zealbot 1.0) LookSmart
Zearchit Zearchit Zearchit
Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/) Zedzo
Zerxbot zerxbot/Version 0.6 libwww-perl/5.79 Zerx
Zetbot DeepIndex ( http://www.zetbot.com ) Zetbot
Zeus ZBot/1.00 (icaulfield@zeus.com)  
Zeus xxxxx Webster Pro V2.9 Win32
Zeus ThemeSite Viewer Webster Pro V2.9 Win32
ZipppBot ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) Zipp
ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net)
Zippy Zippy v2.0 - Zippyfinder.com Zippyfinder
ZoomSpider ZoomSpider - wrensoft.com WrenSoft
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR
1.1.4322)
Zspider zspider/0.9-dev http://feedback.redkolibri.com/ Redkolibri
ZyBorg (LookSmart) ZyBorg LookSmart
ZyBorg (Wisenut) Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnu
tbot.com)
Wisenut
ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com)

0

阅读 评论 收藏 转载 喜欢 打印举报
  • 评论加载中,请稍候...
发评论

    发评论

    以上网友发言只代表其个人观点,不代表新浪网的观点或立场。

      

    新浪BLOG意见反馈留言板 电话:4006900000 提示音后按1键(按当地市话标准计费) 欢迎批评指正

    新浪简介 | About Sina | 广告服务 | 联系我们 | 招聘信息 | 网站律师 | SINA English | 会员注册 | 产品答疑

    新浪公司 版权所有