roughtrax4x4.com
robots.txt

Groups

*

fast enterprise crawler*

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

petalbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

barkrowler

wget

grub-client

k2spider

turnitinbot

slysearch

findlinks

magpie-crawler

pixray-seeker

mj12bot

ezooms

ahrefsbot

lb-spider

wbsearchbot

psbot

huaweisymantecspider

sistrix

yandexbot

ec2linkfinder

htdig

semrushbot

discobot

linkdex.com

seznambot

edisterbot

swebot

picmole

yeti

yeti-mobile

pagepeeker

catchbot

yacybot

netestate ne crawler

surveybot

comodo ssl checker

comodo-certificates-spider

gonzo

schrein

backlinkcrawler

afilias web mining tool

seokicks

seokicks-robot

suggybot

bdbrandprotect

bpimagewalker

bpimagewalker*

updownerbot

lex

content crawler

dcpbot

kaloogabot

mlbot

wget

libwww-perl

curl

java

icjobs

obot

webmastercoffee

qualidator*

webinator

scooter

larbin

opidoobot

ips-agent

tineye

unisterbot

unister*

reverseget

Comments

• Moz
• User-agent: rogerbot
• Disallow: /
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Sorry, wget in its recursive mode is a frequent problem.
• Please read the man page and use it properly; there is a
• --wait option you can use to set the delay between hits,
• for instance.
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• "TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)"
• "findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)"
• "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
• http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
• http://www.80legs.com/webcrawler.html - if 008 is crawling your website, it
• means that one or more 80legs users created a web crawl
• "Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)"
• "lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider;http://www.linkbutler.de/spider; spider@linkbutler.de)"
• "Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)"
• "psbot/0.1 (+http://www.picsearch.com/bot.html)"
• "HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152;.NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)"
• "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
• "EC2LinkFinder"
• "http://SiteIntel.net Bot"
• "htdig"
• "SemrushBot/0.91" - http://de.semrush.com/? -Professionelle Software für SEO & SEM Professionals?
• "Mozilla/5.0 (compatible; discobot/2.0;+http://discoveryengine.com/discobot.html)" - we sell no wine before its time!= trustworthy
• "linkdex.com/v2.0" - SEO
• "SeznamBot/3.0 (+http://fulltext.sblog.cz/)" - sz-SEO
• "EdisterBot (http://www.edister.com/bot.html)"
• "Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)" - versuchtauf posting im forum zu replien
• ab hier noch in htaccess eintragen
• "Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)"
• "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)"
• "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible;Yeti-Mobile/0.1; +http://help.naver.com/robots/)"
• "PagePeeker.com (info: http://pagepeeker.com/robots)"
• "CatchBot/1.0; +http://www.catchbot.com"
• "yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24;Europe/de) http://yacy.net/bot.html"
• "netEstate NE Crawler (+http://www.sengine.info/)"
• "Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)"
• "COMODO SSL Checker"
• "Comodo-Certificates-Spider"
• "gonzo2[p] (+http://www.suchen.de/faq.html)" (Geschäftesuche)
• "crawler schrein, crawler@schrein.nl id-4"
• "BacklinkCrawler (http://www.backlinktest.com/crawler.html)"
• "Mozilla/5.0 (compatible; Afilias Web Mining Tool 1.0;+http://www.afilias.info; awmt@afilias.info)"
• "Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)"
• "Mozilla/5.0 (compatible; suggybot v0.01a,http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)"
• "http://www.bdbrandprotect.com""Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
• "Updownerbot (+http://www.updowner.com/bot)"
• "lex/1.0"
• "Content Crawler"
• "Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)"
• "Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)"
• "MLBot (www.metadatalabs.com/mlbot)"
• "Wget/1.9"
• "libwww-perl/5.837"
• "curl/7.21.3 (amd64-portbld-freebsd7.2) libcurl/7.21.3 OpenSSL/0.9.8ezlib/1.2.3"
• "Java/1.6.0_29"
• "Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3"
• "Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)"
• "Mozilla/5.0 (compatible; WebmasterCoffee/0.7;+http://webmastercoffee.com/about)"
• "Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)"(http://www.qualidator.com/Web/de/Support/robotstxt_Hinweise.htm)
• "Mozilla/4.0 (compatible;http://search.thunderstone.com/texis/websearch/about.html)"(http://www.thunderstone.com/site/gw25man/page_exclusion_and_robots_txt.html)
• "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)(larbin2.6.3@unspecified.mail)"
• "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent)Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24"
• "TinEye/1.1 (http://tineye.com/crawler.html)"
• "Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)"
• "Mozilla/5.0 (compatible; en-US; ReverseGet/1.0; http://reverseget.com/;robot@reverseget.com)"

Warnings

• 4 invalid lines.