Log In Sign Up

fun-flyfishing.com
robots.txt

Robots Exclusion Standard data for fun-flyfishing.com

Resource Scan

Scan Details

Site Domain fun-flyfishing.com
Base Domain fun-flyfishing.com
Scan Status Ok
Last Scan2024-06-08T10:41:17+00:00
Next Scan 2024-07-08T10:41:17+00:00

Last Scan

Scanned2024-06-08T10:41:17+00:00
URL https://fun-flyfishing.com/robots.txt
Domain IPs 166.0.234.33
Response IP 166.0.234.33
Found Yes
Hash 0cde59a1fb1c4344c01764448e6960c81f52c047f3aa9c87663f8abf919c46c7
SimHash 5abe2f107d53

Groups

googlebot

Rule Path
Allow /forum

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot-sa

Rule Path Comment
Disallow / BotDoku: de.wetena.com/bot

garlikcrawler

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

slurp

Rule Path
Disallow /forum/adm/
Disallow /cgi-bin/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ltx71

Rule Path
Disallow /

researchbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

red

Rule Path
Disallow /

pocketparser

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

publiclibraryarchive

Rule Path
Disallow /

publiclibraryarchive.org

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

domaintunocrawler

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

pagespeedbot

Rule Path
Disallow /

infohelfer

Rule Path
Disallow /

pixray*

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

u

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

spiderlytics

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

alexa

Rule Path
Disallow /

ssearch_bot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

exb

Rule Path
Disallow /

macinroy

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

icjobs

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mxt

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

domaintuno

Rule Path
Disallow /

dubaiindex

Rule Path
Disallow /

abonti

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

ncbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

downloadbot

Rule Path
Disallow /

testcrawler

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

it2media-domain-crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

psbot

Rule Path
Disallow /

woriobot

Rule Path
Disallow /

ssearch

Rule Path
Disallow /

waybackarchive.org

Rule Path
Disallow /

netestate

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

obot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

mediabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

seobility

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

monobot

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

ca-crawler

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

it2media-domain-crawler

Rule Path
Disallow /

xxx

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

sitedomain-bot

Rule Path
Disallow /

wonderbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

dbot

Rule Path
Disallow /

medialbot

Rule Path
Disallow /

dl2bot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

*

Rule Path
Disallow /forum/adm/
Disallow /cgi-bin/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1
Back to top

Comments

  • Doku: www.robotstxt.org
  • erlaubte robots
  • google.com Googlebot
  • bing.com bingbot
  • msn.com
  • MSIE
  • info@netcraft.com
  • qwant.com
  • TODO: Noch testen ob robot.txt ausgewertet wird
  • OK: Web security - Kommerzielles Tool schuetzt Verbraucher in UK und US vor online criminals
  • IP 185.26.92.4
  • "GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)"
  • ignoriert robots.txt
  • IP 106.120.173.103, 218.30.103.125
  • "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
  • Check
  • IP 46.165.197.142
  • "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)"
  • Crawl-Delay: 5
  • Liste Download Einschraenkungen
  • Yahoo! Slurp
  • === Crawl-delay ===
  • No crawl delay set Normal
  • 1 Slow
  • 5 Very slow
  • 10 Extremely slow
  • ToDo: noch checken ob Sperre funktioniert
  • IP 52.3.127.144
  • "ltx71 - (http://ltx71.com/)"
  • ToDo: noch checken ob Sperre funktioniert
  • IP 84.19.180.46
  • "Mozilla/5.0 (compatible; ResearchBot/0.7)"
  • ToDo: noch checken ob Sperre funktioniert - liest Index ein
  • IP 117.78.13.x
  • "nutch-1.4/Nutch-1.4"
  • ToDo: NEW noch checken ob Sperre funktioniert - greift vor Lesen von robots.txt zu
  • deny 199.101.132.0/22
  • IP 199.101.132.161, 54.205.175.165
  • "Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)"
  • Liste von gesperrten analyse utility
  • IP 119.9.43.241
  • "RED/1 (https://redbot.org/)"
  • Zugriff OHNE robots.txt -
  • IP 54.158.98.57 403 54.144.0.0/12
  • "PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)"
  • Hier ein Liste der unerwuenschten robots - bots ohne Kennung!
  • Yandex
  • ignoriert robot.txt!!! 403 5.102.173.64/28
  • IP 5.102.173.71
  • "Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)"
  • ignoriert robots.txt
  • IP 62.138.0.25, 85.25.
  • "Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)"
  • ignoriert robot.txt!!!
  • IP 136.243.83.16
  • "Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)"
  • ignoriert robots.txt
  • IP 81.30.151.220, 85.114.139.54
  • "Mozilla/5.0 (compatible; publiclibraryarchive.org/1.0; +crawl@publiclibraryarchive.org)"
  • ignoriert robots.txt!!!
  • IP 192.99.39.68, 192.99.107.208
  • "Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)"
  • ignoriert robots.txt!!!
  • "Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)"
  • ignoriert robots.txt!!!
  • IP 192.99.40.137
  • "Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)"
  • ignoriert robots.txt!!!
  • IP 69.58.178.57, 69.58.178.58, 69.58.178.59
  • "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1"
  • ignoriert robots.txt!!!
  • IP 207.241.237. 102,103,105,229 (abwechselnd!) + 207.241.226.234
  • "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
  • ignoriert robots.txt!!!
  • IP 130.185.109.243, 130.185.104.89
  • "PagesInventory (robot http://www.pagesinvenotry.com)"
  • ignoriert robots.txt!!!
  • 64.246.165.160, 64.246.187.42
  • "Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)"
  • ignoriert robots.txt!!!
  • 65.208.151.112 - 65.208.151.119
  • 63.110.148.104 - 120
  • 63.110.158.48 - 63.110.158.55
  • 65.200.47.0 - 65.200.47.7
  • 65.208.189.24 - 65.208.189.31
  • 65.208.185.96 - 65.208.185.103
  • 65.211.195.16 - 65.211.195.23
  • "GET / HTTP/1.1" 200 4755 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
  • Kintiskton LLC
  • ignoriert robots.txt!!!
  • IP 104.131.211.106
  • "Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)"
  • ignoriert robots.txt!!!
  • IP 69.50.224.0/19, 198.89.96.0/19
  • "Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)"
  • Bots, welche robots.txt respektieren
  • IP 138.201.30.66, 46.4.68.142
  • "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
  • IP 63.249.66.212
  • "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )"
  • metadatalabs.com
  • IP 5.10.83.36, 151.80.31.162
  • "Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)"
  • "http://www.youdao.com/help/webmaster/spider/"
  • Disallow: jobs.de-Robot
  • IP 167.114.172.225
  • "Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)"
  • ezooms.bot
  • Pagespped Crawler
  • www.infohelfer.de
  • www.pixray.com
  • http://warebay.com/bot.html
  • aihit.com https://www.aihitdata.com/about
  • IP 141.8.147.17
  • "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
  • U
  • http://www.seodiver.com/bot
  • http://www.brandwatch.com/magpie-crawler/
  • http://www.cloudservermarket.com
  • http://www.trendiction.de/de/publisher/bot
  • http://www.career-x.de/bot.html
  • IP 5.199.136.130
  • "Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)"
  • IP 5.9.6.51
  • "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"
  • http://help.coccoc.com/
  • ubermetrics-technologies.com
  • datenbutler.de
  • http://searchgears.de/uber-uns/crawling-faq.html
  • http://linkfluence.net/
  • http://www.botje.com/plukkie.htm
  • http://www.haosou.com/help/help_3_2.html
  • http://www.haosou.com/help/help_3_2.html
  • http://www.moz.com/dp/rogerbot
  • http://www.openhose.org/bot.html
  • http://www.screamingfrog.co.uk/seo-spider/
  • http://thumbsniper.com
  • http://www.radian6.com/crawler
  • Unknown
  • IP 207.241.226.239
  • "ia_archiver(OS-Wayback)"
  • IP 204.236.235.245
  • "ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)"
  • IP 148.251.44.138
  • "ssearch_bot (sSearch Crawler; http://www.semantissimo.de)"
  • IP 217.69.133.253, 217.69.143.62
  • "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
  • IP 78.46.154.40
  • "ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)"
  • IP 85.25.137.24
  • "MacInroy Privacy Auditors. See xyz.org's privacy violation report: http://xyz.org.macinroy.com/xyz.org"
  • IP 46.229.164.100, 46.229.164.102, 46.229.164.112,
  • "Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)"
  • IP 52.91.150.203
  • "CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)"
  • IP 85.25.71.40
  • "Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3"
  • IP 77.75.77.32, 77.75.73.17
  • "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
  • IP 139.18.2.209
  • "findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)"
  • IP 108.178.53.146
  • "Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)"
  • IP 208.43.225.84, 208.43.225.85
  • "Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)"
  • IP 54.242.123.170, 23.22.229.75, 54.225.52.217 23.20.126.233
  • "Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)"
  • IP 94.199.151.22
  • "Wotbox/2.01 (+http://www.wotbox.com/bot/)"
  • wertet bei Anfrage (ohne www.) von robots.txt kein 301 aus => 403
  • IP 64.79.85.205, 64.79.76.50
  • "Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)"
  • IP 192.96.204.42
  • "http://www.domaintuno.com/whois/jarnold.org" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)"
  • unknown addressendeutschland.de
  • IP 86.109.249.174
  • "http://arnold-soft.de/" "dubaiindex (addressendeutschland.de)"
  • IP 77.233.225.115
  • "Mozilla/5.0 (compatible; Abonti/0.91 - http://www.abonti.com)"
  • IP 46.4.100.231
  • "BacklinkCrawler (http://www.backlinktest.com/crawler.html)"
  • IP 54.227.175.17
  • "NCBot http://netcomber.com?st=ba2Tool for finding all their domain names."
  • IP 89.145.95.2
  • "Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)"
  • IP 188.40.249.87
  • "Mozilla/5.0 (compatible; DownloadBot/2.0; +http://overx50.com/)"
  • IP 104.130.201.84
  • "Mozilla/5.0 (compatible; TestCrawler)"
  • IP 217.69.143.66
  • "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)"
  • IP 64.125.222.16
  • "Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html;) Gecko/2008032620"
  • IP 86.109.249.169
  • "it2media-domain-crawler/1.0 on crawler-prod.it2media.de"
  • IP 176.9.148.197, IP 176.9.155.226, 5.9.112.66
  • "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
  • IP 217.212.224.183
  • "psbot/0.1 (+http://www.picsearch.com/bot.html)"
  • IP 107.22.250.59
  • "Mozilla/5.0 (compatible; woriobot +http://worio.com)"
  • IP 88.198.24.173
  • "ssearch_bot (sSearch Crawler; http://www.semantissimo.de)"
  • IP 5.199.136.130
  • "Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)"
  • IP 81.209.177.145
  • "netEstate NE Crawler (+http://www.website-datenbank.de/)"
  • IP 68.47.129.55
  • "Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)"
  • www.seoprofiler.com/bot
  • IP 198.199.89.149, 162.243.203.202, 104.131.217.194, 107.170.40.178
  • "Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )"
  • IP 206.253.226.18
  • "Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)"
  • 183.60.243.187, 123.125.71.60
  • "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0"
  • "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
  • Fake oder Baidu OHNE Kennung??? 180.76.15.xx
  • "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"
  • IP 178.255.215.x
  • crawled trotzdem "Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)" 403 178.255.215.
  • "Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)"
  • Crawl-delay: 10
  • IP 217.73.208.103
  • "Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)"
  • IP 75.98.9.250
  • "Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)"
  • IP 91.250.15.69
  • "Mozilla/5.1 (compatible; MediaBot/1.1.6; +http://mercedes-w123.net)"
  • IP 208.115.113.92
  • "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
  • http://www.proximic.com/info/spider.php
  • IP 54.211.1.18
  • "Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)"
  • IP 54.227.12.4
  • "CCBot/2.0 (http://commoncrawl.org/faq/)"
  • IP 148.251.234.39
  • "Seobility (SEO-Check; http://bit.ly/1dJuuzs)"
  • IP 130.211.186.147, 146.148.35.52, 107.178.216.118
  • "GET / HTTP/1.0" 200 10064 "-" "NerdyBot"
  • IP 91.250.15.69
  • "Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)"
  • IP 37.59.55.128
  • "Mozilla/5.0 (compatible; fr-crawler/1.1)"
  • IP 192.95.29.116
  • "Mozilla/5.0 (compatible; ca-crawler/1.0)"
  • IP 37.16.73.17
  • "Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)"
  • IP 85.10.246.243
  • "Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)"
  • IP 86.109.249.169
  • "it2media-domain-crawler/2.0"
  • http://semalt.semalt.com/crawler.php
  • IP 187.79.214.121, 177.182.110.46, 180.87.245.236, 203.106.154.207, 186.226.182.72, 91.250.157.238, 190.213.84.119, 109.31.192.115
  • "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
  • IP 212.224.119.179, 176.9.40.107
  • "Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)"
  • IP 38.111.147.84
  • "TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)"
  • IP 69.84.207.246
  • "LSSRocketCrawler/1.0 LightspeedSystems"
  • IP 88.198.1.77
  • "Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)"
  • IP 82.208.160.181
  • "wonderbot/JS 1.0"
  • IP 188.166.1.121, 95.85.43.91
  • "SafeDNSBot (https://www.safedns.com/searchbot)"
  • IP 136.243.14.225
  • "Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)"
  • 50.17.21.141, 81.169.245.220, 35.156.50.187
  • "Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)"
  • ohne robots.txt
  • IP 85.17.73.171, 5.79.68.56
  • "Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)"
  • unerwuenscht ohne robot.txt => Abfrage "GET / HTTP/1.0"
  • a14download.com
  • IP 188.40.249.87
  • "http://a14download.com" "Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)"
  • unerwuenscht ohne robot.txt => Abfrage "GET / HTTP/1.0" ### => 403 blacklist
  • IP 144.76.178.226
  • "http://medialine-it.com" "Mozilla/5.1 (compatible; MediaLBot/1.1.5; +http://medialine-it.com)"
  • IP 5.101.100.60,
  • "Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)"
  • IP 91.250.15.69
  • "http://dl2engine.net" "Mozilla/5.1 (compatible; DL2Bot/1.0; +http://dl2engine.net)"
  • IP 212.232.24.2
  • "-"
  • IP 65.132.59.34
  • "Gigabot/1.0"
  • unerwuenscht ohne robot.txt + igonriert robot = 403 blacklist
  • IP 82.192.74.245, 82.192.70.50
  • "Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)"
  • Default: Liste mit gesperrten Verzeichnissen
Back to top

Warnings

  • 10 invalid lines.
Back to top