skip to content

System: Searchable Directory of User Agents: Search Engine Spiders

 Tweet0 Shares0 Tweets

The following is a directory of user agents, including their source and general purpose as far as we can determine. Most entries link to an "official" site containing more detailed information. You can also paste a UA from your logs into the form below, hit [Go!] and see a list the relevant agents.

We currently have 946 distinct user agents in our database representing everything from search engines to software components and spambots. These have been collected from our log files over a number of years and researched manually.

Search for User Agent

To use this form just copy and paste an entire User-Agent string from your server log file into the input box and then submit the form. The search is case-sensitive so "nokia" will not match "Nokia".

Most user agent strings now contain a number of separate components so the search will return a list of everything that has a match in the database.

Analyse UA:

Search Engine Spiders

View Category:

These agents conduct Internet-wide indexing for various search engines.

1) 123spider-Bot - www.123spider.de/
2) abot - abot.com/
3) acont.de - hilfe.acont.de/eintrag/
4) ActiveTouristBot - www.activetourist.com/
ActiveTouristBot is a web crawler that automatically crawls the web looking for tourist information.
5) AdsBot - www.google.com/adsbot.html
The AdWords system will visit and evaluate all pages specified by your ad Destination URLs. It will also follow redirect URLs. To fully understand the quality of your specified page, the system may follow other links on the page.
6) agadine - www.agada.de/
7) Alteray - production.artxite.com/
requests XXXLINEXXXXrobots.txt
8) Amfibibot - www.amfibi.com/
9) AnsearchBot - www.ansearch.com.au/
Ansearch is designed around the concept of delivering quality results based on user behaviour and popularity
10) AnswerBus - www.answerbus.com/
11) antibot - www.antidot.net/Welcome/jsp/robots.html
Le robot utilisé par AFS est nommé AntiBot
12) Aport - www.aport.ru/
13) appie - www.walhello.com/aboutgl.html
The walhello spider (appie) follows links to web pages on the Internet and stores every new page found in the Walhello database
14) Argus - www.simpy.com/bot.html
Argus is Simpy's web-crawling robot. It fetches documents from the web to build searchable indices for users of Simpy service.
15) Art-Online - www.art-online.com/
16) Ask Jeeves/Teoma - static.wc.ask.com/docs/addjeeves/Submit.html
Search engine crawler for ask.com
17) asterias - www.singingfish.com/help/spider.html
18) athenusbot - www.athenus.com/botinfo.html
AthenusBot is the crawler and indexer used by Athenus to build it's database and provide users with the most up-to-date information possible on internet engineering and science resources.
19) Aussie Golf Search - www.aussiegolfsearch.com/
20) BaiduImagespider - help.baidu.jp/system/05.html
21) Baiduspider - www.baidu.com/search/spider.htm
Chinese search engine.
22) BecomeBot - www.become.com/webmasters.html
BecomeBot is the user-agent for Become's new web crawler.
23) BigCliqueBOT - www.bigclique.com/
BigClique Search Engine - Just Search... Nothing Else!
24) Bigsearch - www.bigsearch.ca/
25) Blaiz-Bee - www.rawgrunt.com/
26) boitho.com - www.boitho.com/dcbot.html
27) BTbot - www.btbot.com/btbot.html
BTbot is a new efficient and fast search engine for bittorrent files.
28) btbot - www.btbot.com/btbot.html
BitTorrent Search Engine
29) Btsearch - www.baotongsoft.com/search.html
信息产业部ICP/IP地址信息备案管理系统
30) BurstFind - www.burstfind.com/
31) Buscaplus Robi - www.buscaplus.com/
32) Businessjet - www.businessjet.com/
33) Cazoodle - www.cazoodle.com/
Coming soon!
34) Charlotte - charlotte.tinami.com/robot.html
35) CipinetBot - www.cipinet.com/bot.html
36) Climate Change Spider - www.climateark.org/
37) Clushbot - www.clush.com/bot.html
38) Clustered-Search-Bot - www.clush.com/
39) Cowbot - www.naver.com/
40) CrawlWave - www.crawlwave.com/
41) CreativeCommons - search.creativecommons.org/
This search helps you find photos, music, text, and other works whose authors want you to re-use it for some uses -- without having to pay or ask permission.
42) DeepIndex - www.deepindex.com/
43) DesertRealm.com - www.desertrealm.com/
44) DIE-KRAEHE - www.die-kraehe.com/
45) Dumbot - www.dumbfind.com/
"the greatest search engine in the history of everything or something"
46) EARTHCOM - www.earthcom.info/
47) EasyDL - keywen.com/Encyclopedia/Bot/
48) ebingbong - www.ebingbong.com/help/about.php
eBingBong has created a search engine which is fun, interactive and personal.
49) Eco-Portal Spider - www.eco-portal.com/
50) ejupiter.com - search.ejupiter.com/
51) EMPAS_ROBOT - www.empas.com/
52) Environmental Sustainability Spider - www.environmentalsustainability.info/
53) envolk - www.envolk.com/envolkspider.html
The envolk spider tracks current states of internet index pages listed in the envolk public internet search database.
54) eseek-crawler - www.exactseek.com/about.html
55) Eurobot - www.ayell.eu/
56) exactseek-crawler - www.exactseek.com/about.html
57) Exalead - www.exalead.com/
58) Factbot - www.factbites.com/webmasters.php
FactBites provides full sentence results, rather than excerpts like other search engines
59) FAST - fast.no/support/crawler.asp
Crawler for alltheweb.com
60) FastBug - www.ay-up.com/
61) favicon - iconsurf.com/
Click on any icon to visit the webpage that hosts the icon
62) Faxobot - www.faxo.com/
63) Feedster Crawler - www.feedster.com/press/overview_tech.php
64) FindelioBot - www.findelio.com/
65) FlickBot - www.divx.com/movies/searchfaq.php
66) Fluffy the spider - www.searchhippo.com/faq.php
67) Forest Conservation Spider - forests.org/
68) Forex - www.netforex.org/
69) Francis - www.neomo.de/
70) FreeFind - www.freefind.com/
Let your visitors Search Your Website
71) Gaisbot - gais.cs.ccu.edu.tw/robot.php
Gaisbot is the agent software of GAIS which crawls web sites all over the world, in order to build a search engine like google or altavista.
72) GalaxyBot - www.galaxy.com/galaxybot.html
73) genieBot - 64.5.245.11/faq/faq.html
GenieBot is a web-indexing robot of GenieKnows Local Search Engine.
74) geometabot - www.geometa.info/geometabot/
GeometaBot ist der Name der Webspider-Komponente von geometa.info
75) GeonaBot - www.geona.com/
76) GeorgeTheTouristBot - www.touristdirectory.co.uk/about/
77) GETRAX - www.getrax.com/
78) Gigabot - www.gigablast.com/spider.html
Gigabot is the name of Gigablast's indexing agent, also known as a spider. Gigabot is like a thousand internet users busily surfing the web. But it moves from page to page indexing the content it finds.
79) Girafabot - www.girafa.com/
Girafa is a FREE web navigation service that works alongside your browser providing you with visualization capabilities when searching and navigating the web
80) GOFORITBOT - www.goforit.com/about/
A search engine that queries other search engines and then combines the results that are received from all
81) GoForIt.com - www.goforit.com/
82) goliatspider
83) Googlebot - www.googlebot.com/bot.html
Google's web-crawling robot
84) Gromit - www2.austlii.edu.au/~dan/gromit/
Gromit is a specialist web robot designed and implemented by programmers at the Australasian Legal Information Institute
85) GurujiBot - www.guruji.com/en/WebmasterFAQ.html
Our goal is to make Guruji a complete India related search engine.
86) Helix - www.sitesearch.ca/helix/
Helix crawls the web in a considerate manner looking for content in order to build a large searchable index of websites.
87) HelpSpy - helpspy.com/spider/
88) HenryTheMiragoRobot - www.miragorobot.com/scripts/mrinfo.asp
Mirago is a Search Engine aimed specifically at UK users.
89) Homerbot - www.homerweb.com/
90) ia_archiver - pages.alexa.com/help/webmasters/
The "ia_archiver" robot drives the archive.org and alexa.com web sites
91) icsbot - icseoul.org/
92) igougocrawler - www.igougo.com/search/web.asp
We are working on a vertical Travel Search Engine, which is a Web search engine that is restricted to a travel domain.
93) ilial - www.ilial.com/crawler/
Ilial is still in stealth mode
94) IlTrovatore-Setaccio - www.iltrovatore.it/aiuto/faq.html
95) infomine.ucr.edu - infomine.ucr.edu/
Scholarly Internet Resource Collections
96) IpselonBot - www.ipselon.com/
97) Jayde - www.jayde.com/
98) Jetbot - www.jeteye.com/jetbot.html
JetEye's Web crawler retrieves Web documents to build a searchable index for the JetEye search engine.
99) JIST3
Joint Information for Systems Technology, Test and Training
100) Jumble - www.jumblefox.com.au/
Australian Search Engine
101) jumblefox - www.jumblefox.com.au/
Australian Search Engine.
102) Jyxobot - jyxo.cz/
Czech search engine.
103) KaloogaBot - www.kalooga.com/
Kalooga is currently in private-beta
104) Kevin - www.dznet.com/kevin/
105) kinja-imagebot - kinja.com/aboutsite.knj
Kinja is a weblog portal, collecting news and commentary from some of the best sites on the web.
106) kuloko-bot - www.kuloko.com/
107) libWeb - lists.webjunction.org/libweb/
(best guess)
108) LNSpiderguy - www.lexisnexis.com/
109) Look.com - www.look.com/
110) Looker - www.lookerbot.com/robot.html
Lookerbot is Looker's web-crawling robot. It searches sites on the web for relevant content and provides it for use by the Looker Search Engine.
111) Lycos-News-Xml-Fetcher
112) Mackster - www.click4choice.com/
113) MarcoPolo - www.marcopolo-education.org/
114) MARTINI - www.looksmart.com/
115) Mavicanet - www.mavicanet.ru/directory/eng/
Multilingual Search Catalog
116) MojeekBot - www.mojeek.com/bot.html
MojeekBot and formerly Citenikbot is the web crawler for the Mojeek search engine
117) mozDex - www.mozdex.com/en/bot.html
118) msnbot - search.msn.com/msnbot.htm
119) msnbot-media - search.msn.com/msnbot.htm
120) multicrawler - sw.deri.org/2006/04/multicrawler/robots.html
The amount of available formal data is growing steadily, but a means to find and thus utilize this data is still missing. What is needed is a service which explores and indexes the Semantic Web.
121) n4p_bot - www.n4p.com/
122) NationalDirectory - www.nationaldirectory.com/
123) NaverBot - www.naver.com/
124) NCSA - vias.ncsa.uiuc.edu/viasarchivinginformation.html
125) NetNose - www.netnose.com/
126) NetResearchServer - loopimprovements.com/robot.html
127) NextGenSearchBot - www.zoominfo.com/About/misc/NextGenSearchBot.aspx
NextGenSearchBot is an indexing robot for a web search engine.
128) NG - www.exalead.com/
Exalead NG/MimeLive Client
129) ObjectsSearch - www.objectssearch.com/
Objects Search aims to provide the users with best search results, free website submission and unbiased website ranking
130) Ocelli - www.globalspec.com/Ocelli
Ocelli is a Web crawler owned and operated by GlobalSpec
131) OmniExplorer_Bot - www.omni-explorer.com/
So far, we've identified a "Jobs Crawler", "Cars Crawler", "Books Crawler" and now an "Internet Categorizer"
132) Openbot - www.openfind.com.tw/robot.html
Openbot is the agent software of Openfind which crawls web sites all over the world, in order to build a search engine like google or altavista.
133) Panopy Bot
134) Patwebbot - www.herz-power.de/technik.html
135) PEERbot - www.peerbot.com/
seerch different
136) PhpDig - www.phpdig.net/robot.php
PHP and MySQL Web Spider and Search Engine. PhpDig is released under GNU GPL.
137) Piffany - www.piffany.com/spider.html
In a few months, you will be able to test the alpha-release of our search engine for kids.
138) pipeLiner - www.pipeline-search.com/webmaster.html
The pipeLiner spider uses the websites in the DMOZ Directory to populate it's crawl list.
139) Pompos - dir.com/pompos.html
140) Popdexter - www.popdex.com/
Popdex crawls to determine the most popular links on the Internet
141) psbot - www.picsearch.com/bot.html
Picsearch is indexing pictures from the web
142) QPCreep - www.quepasa.com/
143) QuepasaCreep - www.quepasa.com/
144) RedBot - www.rediff.com/
Rediff.com is an online providers of news, information, communication, entertainment and shopping services.
145) RixBot - babelserver.org/rix
The index comprises 19715 pages containing the word «rebol».
146) RoboCrawl - www.canadiancontent.net/corp/spider.html
147) Robozilla - www.dmoz.org/
148) RufusBot - 64.124.122.252/feedback.html
We crawl the web towards the goal of developing a new kind of index/search tool.
149) Scooter - www.altavista.com/sites/help/search/faq_web
Search engine crawler for Altavista
150) search.ch - www.search.ch/
151) Search-Channel - www.search-channel.com/fr/
Adult Search Channel
152) SearchIt.Bot - www.searchit.com/
153) searchme - www.searchme.com/support/pages/spider.php
Charlotte is a spider created by Searchme, Inc.
154) Searchspider - www.searchspider.com/
155) Seekbot - www.seekbot.net/bot.html
Der Seekport Roboter heißt Seekbot
156) Sensis - www.sensis.com.au/
157) sensis - www.sensis.com.au/
The search engine for Australians.
158) SeznamBot - fulltext.seznam.cz/
159) ShopWiki - www.shopwiki.com/
ShopWiki is a comparison shopping engine that gets all its data from crawling online stores.
160) SideWinder - www.infoseek.com/
161) SiteSpider - www.sitespider.com/
162) SKIZZLE - www.skizzle.com/
163) Slurp - www.inktomi.com/slurp.html
Slurp is Inktomi Corporation's web-indexing robot. It collects documents from the web to build a searchable index for search services using the Inktomi search engine, including Microsoft and HotBot
164) SMEALSearch-Bot - smealsearch.psu.edu/
165) snap.com - snap.com/about/about.php
This unique new search engine makes it easier than ever to find information on the Web.
166) sohu-search - www.sohu.com/about/English/
167) Sosospider - help.soso.com/webspider.htm
Tencent, Inc. has grown into China's largest and most used Internet service portal.
168) Speedy Spider - www.entireweb.com/
169) Spider_Monkey - www.spidermonkey.ca/add_site.html
170) Spinne - www.webauskunft.at/search/
171) SplatSearch - www.splatsearch.com/
172) sproose - www.sproose.com/bot.html
173) StackRambler - home.rambler.ru/
174) Steeler - www.tkl.iis.u-tokyo.ac.jp/~crawler/crawler.html.en
Steeler is being developed and operated at Kitsuregawa Laboratory, The University of Tokyo
175) sygol - www.sygol.com/
176) Szukacz - www.szukacz.pl/html/RobotEnglishVersion.html
177) Technoratibot - www.technorati.com/
Searches weblogs by keyword and for links. Also provides news from general news services and blogs.
178) Teradex Mapper - www.teradex.com/
179) timboBot - www.breakingblogs.com/timbo_bot.php
timboBot is a bot that scans recently updated weblogs to be included in the BreakingBlogs.com database.
180) Tkensaku - www.tkensaku.com/q.html
181) TravelSpyder - www.travelspyder.com/about-travel-spyder-search.php
TravelSpyder.com is a SPAM FREE co-operative search engine
182) TutorGigBot - www.tutorgig.com/crawler/
TutorGigBot collects content from the web for use by TutorGig's Search.
183) Tutorial Crawler - www.tutorgig.com/crawler/
TutorGig's Tutorial Crawler collects content from the web for use by TutorGig's Search.
184) TygoBot - www.tygo.com/AboutTygo/FAQ.aspx
185) unchaos_crawler - www.unchaos.com/
UnChaos is engaged in the development of the next generation of Web Search Engine.
186) Unitek UniEngine - www.unitek-systems.co.uk/
187) updated.com - shop.updated.com/
188) UTSE - utse.list-team.com/
The UTSE search engine covers Performing Arts and supporting industries worldwide.
189) Vagabondo - webagent.wise-guys.nl/
190) Vakes - www.vakes.com/
191) VoilaBot - www.voila.com/
192) WebAlta - www.webalta.net/ru/about_webmaster.html
Russian search engine.
193) webcrawl.net - www.webcrawl.net/
194) WebGo - www.webgo.com/
195) Web-Robot - web-robot.com/policy.html
196) WebSearch - websearch.com.au/
197) WebSpider - www.webspider.com/
198) WhatchaBot - www.whatchaseek.com/
199) wikia-robot - www.wikia.com/
200) Wine-Searcher - www.wine-searcher.com/
The resource for locating and pricing wines.
201) worio - www.worio.com/
WORIO is an Internet search engine created especially for computer scientists and programmers.
202) WorldLight - www.worldlight.com/
WorldLight.com is dedicated to setting the standard for search technology.
203) Wotbox - www.wotbox.com/about/
204) www.business-socket.com - www.business-socket.com/
205) www.galaxy.com - www.galaxy.com/info/crawler.html
206) www.twi.gs - www.twi.gs/
207) www.webwombat.com.au - www.webwombat.com.au/
208) yacy - www.yacy.net/yacy/
p2p-based distributed Web Search Engine
209) Yahoo-MMCrawler
Crawler for Yahoo! paid results supplied by Overture(?)
210) YahooSeeker - help.yahoo.com/help/us/shop/merchant/
Yahoo! crawls hundreds of thousands of web sites for product information to include within Yahoo! Shopping. We extract product information like product names, prices, images, and more and store them within our Yahoo! Product Search index.
211) Yahoo-VerticalCrawler - www.alltheweb.com/help/webmaster/crawler
212) YANDEX - www.yandex.com/
Yandex is the leading Russian web-resource. Yandex sells indexing and search toolkit applicable to a wide range of search and retrieval applications.
213) Yandex - www.yandex.com/
214) Y!J-BSC - help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html
215) YowedoBot - yowedo.com/en/partners.html
216) Zealbot - listings.looksmart.com/help/faq.jhtml?showFaq=tracking
LookSmart software, called Zealbot, visits all Web sites listed with LookSmart each week to ensure that they are still active, responsive sites
217) zibber - www.zibb.com/CrawlerInformation.aspx
Zibb.com is actively investigating the web from the strict perspective of business professionals.
218) ZipppBot - www.zippp.net/
219) Zoopta - www.zoopta.com.au/
Let your children search with the knowledge that this search engine is providing only sites that are Australian based and safe for search.
220) ZyBorg - www.WISEnutbot.com/
LookSmart link checker

For more information on the user agents listed you can click on the associated link. If you think any of the information here is incorrect or misleading please let us know using the Feedback link below.

Please be aware that we do not add user agents to the database on request, but rather wait to see them in our log files.

Browse User Agents by Category

Browser Extensions (42)
Browser extensions are programs that change or enhnace your web browser. Some of them also collect data by sending information on your browsing habits back to a central server.
Content Management (13)
Data Collection - Commercial (47)
These are sites that collect information for commercial benefit. As far as we are aware no useful information or reports are provided to the public.
Data Collection - Research (29)
These agents are conducting research on the WWW. They may also offer commercial services.
Devices (23)
Mobile phones and other gadgets with browser technology.
Download Managers (39)
Programs that enable users to download or extract information from a website or web server.
Indexing Tools (50)
This is software that enables local or remote indexing of web pages and other content for the purposes of setting up a search engine.
Link Checking Utilities (41)
This is software that conducts remote or local link checking.
Media Players (5)
Applications for playing music, video and other media over the Internet.
Other Resources (12)
Links to online resources relating to robots and spiders.
Proxies (7)
If several clients request the same content, the proxy can deliver that content from its cache, rather than requesting it from the origin server each time.
RSS/Atom Aggregators (43)
These are browser extensions or search spiders that focus on indexing or aggregating RSS and Atom feeds.
Search Engine Spiders (220)
These agents conduct Internet-wide indexing for various search engines.
Server Platforms (6)
Server Software (31)
Site Monitoring Services (15)
Software Components (58)
These are code libraries or application development packages that can be used to build Internet-related applications. How they are used depends on the developer.
Spambots? (45)
These are programs that are used predominately to harvest email addresses, find open guestbooks to post to, etc. They may also have legitimate uses.
Unclassified (174)
The following user agents have either not been identified or do not fit neatly into other categories. New agents appear every day that have limited lifespans. Most (but not all) legitimate user agents identify themselves with a URI or email address.
Validation Tools (10)
These are programs and sites that can be used to validate various aspects of your site: HTML, CSS, META tags, etc.
Web Browsers (36)

< System

Send a message to The Art of Web:


used only for us to reply, and to display your gravatar.

CAPTCHA

<- copy the digits from the image into this box

press <Esc> or click outside this box to close

Post your comment or question
top