Nutch Based Crawler User Agents

This is a listing of only some of our most popular Nutch Based Crawler user agents. Click on the user agent to view detailed information about it.

To decode any user agent, use our API. To get our entire categorised listing of 2,920,985+ user agents, download our database.

User agent Version OS Hardware Type Popularity
Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0/Nutch-1.12 1.12 Linux Computer Common
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0/Nutch-1.12 1.12 Linux Computer Common
Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0/Nutch-1.9 1.9 Linux Computer Common
IRP_edu_bot/Nutch-1.0-dev 1.0 -- Common
DataCrawler/Nutch-1.11 1.11 -- Common
rdfbot/Nutch-1.0-dev 1.0 -- Common
SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) 1.0 -- Common
InsideView/Nutch-1.5.1 1.5 -- Common
Spiderbot/Nutch-1.7 1.7 -- Common
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) -- Common
nutch-1.4/Nutch-1.4 1.4 -- Common
tbot-nutch/Nutch-1.10 -- Common
Telefoonboek.nl/Nutch-2.3 2.3 -- Common
MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com) 1.12 -- Average
GeoHasher/Nutch-1.0 (GeoHasher Web Search Engine; geohasher.gotdns.org; geo_hasher at yahoo * com) 1.0 -- Server Average
NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) -- Average
spider-lh/Nutch-1.10 1.10 -- Average
Mozilla/5.0/nutch (contactbigdatafr at gmail.com) -- Average
CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) 0.9 -- Average
CRMNLCrawlAgent/Nutch-1.15-SNAPSHOT 1.15 -- Average
AA_Crawler/Nutch-1.12 1.12 -- Average
ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com) 0.9 -- Average
Mozilla/5.0 (compatible; TMMBot/0.2; +http://www.dslab.ch)/Nutch-1.15 1.15 -- Average
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) -- Average
IRP_edu_bot/Nutch-1.0-dev (michael dot p dot ryan at intel dot com) 1.0 -- Average
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) -- Average
cshopper/Nutch-1.0-dev 1.0 -- Average
crawler.datenmarkt.de/Nutch-1.11 1.11 -- Average
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36/Nutch-1.13 1.13 -- Average
Insideview/Nutch-1.13-SNAPSHOT 1.13 -- Average
disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) 1.0 -- Average
Mozilla/5.0 (compatible; enolystBot/0.2; +http://www.enolyst.com)/Nutch-1.14 1.14 -- Average
SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html) 1.0 -- Average
SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com) 1.0 -- Average
SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com) 1.9 -- Average
NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) -- Average
CB/Nutch-1.7 1.7 -- Average
MXT/Nutch-1.10 1.10 -- Average
CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) 0.9 -- Average
Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://corp.krugle.com/crawler/info.html; webcrawler@krugle.com) 0.8 -- Average
BlackCatB/Nutch-1.10-SNAPSHOT 1.10 -- Average
NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) -- Average
SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) 1.0 -- Average
Olbico agent/Nutch-1.15-SNAPSHOT 1.15 -- Average
MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com) 1.10 -- Average
HouxouCrawler/Nutch-0.9 (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.houxou.com/crawler; crawler at houxou dot com) 0.9 -- Average
LoocalCrawler/Nutch-1.7 (v2014a by weluse; https://weluse.de/) 1.7 -- Average
MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; mqbot@cs.uiuc.edu) 0.9 -- Average
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) -- Average
Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) 1.0 -- Average

Download our database of user agents

Do you want to easily download all our 2,920,985+ user agents? It's available with an API subscription. Find out more about our User agent database.