hardforkgaming.com
robots.txt

Robots Exclusion Standard data for hardforkgaming.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	hardforkgaming.com
Base Domain	hardforkgaming.com
Scan Status	Ok
Last Scan	2024-10-28T11:18:55+00:00
Next Scan	2024-11-27T11:18:55+00:00

Last Scan

Scanned	2024-10-28T11:18:55+00:00
URL	https://hardforkgaming.com/robots.txt
Domain IPs	104.21.80.95, 172.67.177.38, 2606:4700:3033::ac43:b126, 2606:4700:3035::6815:505f
Response IP	104.21.80.95
Found	Yes
Hash	f37d06ef1673c5049f7a03a2d1ae6e79bbbecda35e031268b363f7fed286028c
SimHash	7355343a09a6

Groups

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot-sa

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

rogerbot
exabot
mj12bot
dotbot
gigabot
ahrefsbot
blackwidow
chinaclaw
custo
disco
download\ demon
ecatch
eirgrabber
emailsiphon
emailwolf
express\ webpictures
extractorpro
eyenetie
flashget
getright
getweb!
go!zilla
go-ahead-got-it
grabnet
grafula
hmview
httrack
image\ stripper
image\ sucker
indy\ library
interget
internet\ ninja
jetcar
joc\ web\ spider
larbin
leechftp
mass\ downloader
midown\ tool
mister\ pix
navroad
nearsite
netants
netspider
net\ vampire
netzip
octopus
offline\ explorer
offline\ navigator
pagegrabber
papa\ foto
pavuk
pcbrowser
realdownload
jamesbot
nerdybot
blekkobot
majestic-seo
majestic-12
ezooms
blexbot
linkdexbot
spbot
aboundexbot
archive.org_bot
backlinkcrawler
reget
sitesnagger
smartdownload
superbot
superhttp
surfbot
takeout
teleport\ pro
voideye
web\ image\ collector
web\ sucker
spyfu
webauto
webcopier
webfetch
webgo\ is
webleacher
linkminer
backlinkwatch
kwfinder
scrapebox
openlinkprofiler
scrapebox
ia_archiver
scrapebox
linkpad
scrapebox
seobility
buzzsumo
seobility
blp_bbot
blp_bbot/0.1
deepcrawl
mj12bot
linkresearchtools
moonsearch
linkresearchtools
sheer explorer
sheerexplorer
sheer_explorer
webseoanalytics
blexbot
ahrefsbot
mj12bot
rogerbot
semrushbot
seokicks-robot
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
semrushbot-sa
ahrefsbot
semrushbot
wget
scoutjet
widow
wwwoffle
xaldon\ webspider
zeus

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix

Rule	Path
Disallow	/

Rule

Path

Disallow

seokicks-robot

Rule	Path
Disallow	/

Rule

Path

Disallow

jobs.de-robot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

unisterbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

searchmetricsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

surveybot

Rule	Path
Disallow	/

Rule

Path

Disallow

seodiver

Rule	Path
Disallow	/

Rule

Path

Disallow

spbot

Rule	Path
Disallow	/

Rule

Path

Disallow

wotbox

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

meanpathbot

Rule	Path
Disallow	/

Rule

Path

Disallow

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

magpie-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

obot

Rule	Path
Disallow	/

Rule

Path

Disallow

fr-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.com

Rule	Path
Disallow	/

Rule

Path

Disallow

cloudservermarketspider

Rule	Path
Disallow	/

Rule

Path

Disallow

trendictionbot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

careerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperhey-kaus-australis

Rule	Path
Disallow	/

Rule

Path

Disallow

seoscanners.net

Rule	Path
Disallow	/

Rule

Path

Disallow

metajobbot

Rule	Path
Disallow	/

Rule

Path

Disallow

spiderbot

Rule	Path
Disallow	/

Rule

Path

Disallow

linkstats

Rule	Path
Disallow	/

Rule

Path

Disallow

jobboersebot

Rule	Path
Disallow	/

Rule

Path

Disallow

iccrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

plista

Rule	Path
Disallow	/

Rule

Path

Disallow

domain re-animator bot

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperhey-kaus-australis

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

coccoc

Rule	Path
Disallow	/

Rule

Path

Disallow

um-ic

Rule	Path
Disallow	/

Rule

Path

Disallow

mindupbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sg-orbiter

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

kraken

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

thumbsniper

Rule

Path

Disallow

r6_commentreader

Rule

Path

Disallow

implisensebot

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

python/3.5 aiohttp

Rule

Path

Disallow

toweya.com

Rule

Path

Disallow

netestate

Rule

Path

Disallow

bubing

Rule

Path

Disallow

linguee

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

domaincrawler

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

indeedbot

Rule

Path

Disallow

garlikcrawler

Rule

Path

Disallow

gosign-security-crawler

Rule

Path

Disallow

siteliner

Rule

Path

Disallow

serpstatbot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

yandex

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

sistrix crawler

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

jobs.de-robot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

unisterbot

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

searchmetricsbot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

seodiver

Rule

Path

Disallow

spbot

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

backlinkcrawler

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

obot

Rule

Path

Disallow

fr-crawler

Rule

Path

Disallow

blexbot

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

megaindex.com

Rule

Path

Disallow

cloudservermarketspider

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

exabot

Rule

Path

Disallow

careerbot

Rule

Path

Disallow

lipperhey-kaus-australis

Rule

Path

Disallow

seoscanners.net

Rule

Path

Disallow

metajobbot

Rule

Path

Disallow

spiderbot

Rule

Path

Disallow

linkstats

Rule

Path

Disallow

jobboersebot

Rule

Path

Disallow

iccrawler

Rule

Path

Disallow

plista

Rule

Path

Disallow

domain re-animator bot

Rule

Path

Disallow

lipperhey-kaus-australis

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

coccoc

Rule

Path

Disallow

um-ic

Rule

Path

Disallow

mindupbot

Rule

Path

Disallow

sg-orbiter

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

kraken

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

thumbsniper

Rule

Path

Disallow

r6_commentreader

Rule

Path

Disallow

implisensebot

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

aboundexbot

Rule

Path

Disallow

adbeat_bot

Rule

Path

Disallow

addthis

Rule

Path

Disallow

advbot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

alphaseobot

Rule

Path

Disallow

alphaseobot-sa

Rule

Path

Disallow

audisto

Rule

Path

Disallow

audisto-essential

Rule

Path

Disallow

awariorssbot

Rule

Path

Disallow

awariosmartbot

Rule

Path

Disallow

backlinkcrawler

Rule

Path

Disallow

begunadvertising

Rule

Path

Disallow

betabot

Rule

Path

Disallow

bitlybot

Rule

Path

Disallow

blexbot

Rule

Path

Disallow

booglebot

Rule

Path

Disallow

bubing

Rule

Path

Disallow

buckyohare

Rule

Path

Disallow

careerbot

Rule

Path

Disallow

ca-crawler

Rule

Path

Disallow

calculon

Rule

Path

Disallow

catchbot

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule

Path

Disallow

cms crawler

Rule

Path

Disallow

compspybot

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

crazywebcrawler-spider

Rule

Path

Disallow

cukbot

Rule

Path

Disallow

dataprovider

Rule

Path

Disallow

deepcrawl

Rule

Path

Disallow

der-bot

Rule

Path

Disallow

diffbot

Rule

Path

Disallow

dlvr.it

Rule

Path

Disallow

domainappender

Rule

Path

Disallow

domainstatsbot

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

dubaiindex

Rule

Path

Disallow

electricmonk

Rule

Path

Disallow

envolk

Rule

Path

Disallow

evc-batch/2.0

Rule

Path

Disallow

expertsearchspider

Rule

Path

Disallow

extlinksbot

Rule

Path

Disallow

fatbot

Rule

Path

Disallow

findxbot

Rule

Path

Disallow

fr-crawler

Rule

Path

Disallow

garlikcrawler

Rule

Path

Disallow

genieo

Rule

Path

Disallow

gloomarbot

Rule

Path

Disallow

grapeshot

Rule

Path

Disallow

httrack

Rule

Path

Disallow

huaweisymantecspider

Rule

Path

Disallow

hubspot crawler 1.0 http://www.hubspot.com/

Rule

Path

Disallow

hubspot links crawler 1.0 http://www.hubspot.com/

Rule

Path

Disallow

hypercrawl

Rule

Path

Disallow

hypestat

Rule

Path

Disallow

ias_crawler

Rule

Path

Disallow

iccrawler - icjobs

Rule

Path

Disallow

idmarch

Rule

Path

Disallow

implisensebot

Rule

Path

Disallow

ips-agent

Rule

Path

Disallow

irlbot

Rule

Path

Disallow

jamesbot

Rule

Path

Disallow

jobboersebot

Rule

Path

Disallow

jobdiggerspider

Rule

Path

Disallow

kalooga

Rule

Path

Disallow

kraken

Rule

Path

Disallow

larbin

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

linkpadbot

Rule

Path

Disallow

lipperhey

Rule

Path

Disallow

lssrocketcrawler

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

mappy

Rule

Path

Disallow

mauibot

Rule

Path

Disallow

mbcrawler

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

megaindex.ru 2.0

Rule

Path

Disallow

metajobbot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

nbot

Rule

Path

Disallow

neofonie

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

netcraftsurveyagent

Rule

Path

Disallow

netpeakspiderbot

Rule

Path

Disallow

netseer

Rule

Path

Disallow

nextgensearchbot

Rule

Path

Disallow

nutch

Rule

Path

Disallow

obot

Rule

Path

Disallow

openindexspider

Rule

Path

Disallow

panscient.com

Rule

Path

Disallow

piplbot

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

pockey-gethtml

Rule

Path

Disallow

proximic

Rule

Path

Disallow

pu_in

Rule

Path

Disallow

pub-crawler

Rule

Path

Disallow

radian6

Rule

Path

Disallow

ranksonicsiteauditor

Rule

Path

Disallow

ravencrawler

Rule

Path

Disallow

red

Rule

Path

Disallow

redesscrapy

Rule

Path

Disallow

riddler

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

sbsearch

Rule

Path

Disallow

scoutjet

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

seocharger-robot

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

seolyticscrawler

Rule

Path

Disallow

seoscanners.net/1

Rule

Path

Disallow

sg-orbiter

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

smabblerbot

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

spbot

Rule

Path

Disallow

spiderling

Rule

Path

Disallow

spiderlytics

Rule

Path

Disallow

ssearch_bot

Rule

Path

Disallow

ssearch crawler

Rule

Path

Disallow

steeler

Rule

Path

Disallow

stq_bot

Rule

Path

Disallow

surdotlybot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

swiftbot

Rule

Path

Disallow

the knowledge ai

Rule

Path

Disallow

thumbshots-de-bot

Rule

Path

Disallow

thumbsniper (http://thumbsniper.com)

Rule

Path

Disallow

thunderstone

Rule

Path

Disallow

tineye

Rule

Path

Disallow

toscrawler

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

ucrawler

Rule

Path

Disallow

uptimebot

Rule

Path

Disallow

vagabondo

Rule

Path

Disallow

vebidoobot

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

voltron

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

webmeasurement-bot

Rule

Path

Disallow

wesee

Rule

Path

Disallow

wevikabot

Rule

Path

Disallow

wget

Rule

Path

Disallow

wikido

Rule

Path

Disallow

wonderbot

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

x28-job-bot

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

zoombot

Rule

Path

Disallow

abonti

Rule

Path

Disallow

aboundex

Rule

Path

Disallow

aboundexbot

Rule

Path

Disallow

acunetix

Rule

Path

Disallow

admantx

Rule

Path

Disallow

afd-verbotsverfahren

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

aibot

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

aipbot

Rule

Path

Disallow

alexibot

Rule

Path

Disallow

alligator

Rule

Path

Disallow

allsubmitter

Rule

Path

Disallow

alphabot

Rule

Path

Disallow

anarchie

Rule

Path

Disallow

apexoo

Rule

Path

Disallow

archive.org_bot

Rule

Path

Disallow

aspseek

Rule

Path

Disallow

asterias

Rule

Path

Disallow

attach

Rule

Path

Disallow

autoemailspider

Rule

Path

Disallow

backdoorbot

Rule

Path

Disallow

backlink-ceck

Rule

Path

Disallow

backlink-check

Rule

Path

Disallow

backlinkcrawler

Rule

Path

Disallow

backstreet

Rule

Path

Disallow

backweb

Rule

Path

Disallow

bandit

Rule

Path

Disallow

barkrowler

Rule

Path

Disallow

batchftp

Rule

Path

Disallow

battleztar bazinga

Rule

Path

Disallow

bbbike

Rule

Path

Disallow

bdcbot

Rule

Path

Disallow

bdfetch

Rule

Path

Disallow

betabot

Rule

Path

Disallow

bigfoot

Rule

Path

Disallow

bitacle

Rule

Path

Disallow

blackboard

Rule

Path

Disallow

black hole

Rule

Path

Disallow

blackwidow

Rule

Path

Disallow

blexbot

Rule

Path

Disallow

blow

Rule

Path

Disallow

blowfish

Rule

Path

Disallow

boardreader

Rule

Path

Disallow

bolt

Rule

Path

Disallow

botalot

Rule

Path

Disallow

brandprotect

Rule

Path

Disallow

brandwatch

Rule

Path

Disallow

buddy

Rule

Path

Disallow

builtbottough

Rule

Path

Disallow

builtwith

Rule

Path

Disallow

bullseye

Rule

Path

Disallow

bunnyslippers

Rule

Path

Disallow

buzzsumo

Rule

Path

Disallow

calculon

Rule

Path

Disallow

catexplorador

Rule

Path

Disallow

cazoodlebot

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

cegbfeieh

Rule

Path

Disallow

cheesebot

Rule

Path

Disallow

cherrypicker

Rule

Path

Disallow

cheteam

Rule

Path

Disallow

chinaclaw

Rule

Path

Disallow

chlooe

Rule

Path

Disallow

claritybot

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

cloud mapping

Rule

Path

Disallow

coccocbot-web

Rule

Path

Disallow

cogentbot

Rule

Path

Disallow

cognitiveseo

Rule

Path

Disallow

collector

Rule

Path

Disallow

com.plumanalytics

Rule

Path

Disallow

copier

Rule

Path

Disallow

copyrightcheck

Rule

Path

Disallow

copyscape

Rule

Path

Disallow

cosmos

Rule

Path

Disallow

craftbot

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

crawler.feedback

Rule

Path

Disallow

crawl.sogou.com

Rule

Path

Disallow

crazywebcrawler

Rule

Path

Disallow

crescent

Rule

Path

Disallow

crunchbot

Rule

Path

Disallow

cshttp

Rule

Path

Disallow

curious

Rule

Path

Disallow

custo

Rule

Path

Disallow

databasedrivermysqli

Rule

Path

Disallow

datacha0s

Rule

Path

Disallow

dblbot

Rule

Path

Disallow

demandbase-bot

Rule

Path

Disallow

demon

Rule

Path

Disallow

deusu

Rule

Path

Disallow

devil

Rule

Path

Disallow

digincore

Rule

Path

Disallow

digitalpebble

Rule

Path

Disallow

diibot

Rule

Path

Disallow

dirbuster

Rule

Path

Disallow

disco

Rule

Path

Disallow

discobot

Rule

Path

Disallow

discoverybot

Rule

Path

Disallow

dispatch

Rule

Path

Disallow

dittospyder

Rule

Path

Disallow

dnyzbot

Rule

Path

Disallow

domainappender

Rule

Path

Disallow

domaincrawler

Rule

Path

Disallow

domainsigmacrawler

Rule

Path

Disallow

domainstatsbot

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

download wonder

Rule

Path

Disallow

dragonfly

Rule

Path

Disallow

drip

Rule

Path

Disallow

dsearch

Rule

Path

Disallow

dts agent

Rule

Path

Disallow

easydl

Rule

Path

Disallow

ebingbong

Rule

Path

Disallow

ecatch

Rule

Path

Disallow

eccp/1.0

Rule

Path

Disallow

ecxi

Rule

Path

Disallow

eirgrabber

Rule

Path

Disallow

email siphon

Rule

Path

Disallow

email wolf

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

evc-batch

Rule

Path

Disallow

evil

Rule

Path

Disallow

exabot

Rule

Path

Disallow

express webpictures

Rule

Path

Disallow

extlinksbot

Rule

Path

Disallow

extractor

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

extreme picture finder

Rule

Path

Disallow

eyenetie

Rule

Path

Disallow

ezooms

Rule

Path

Disallow

facebookscraper

Rule

Path

Disallow

fdm

Rule

Path

Disallow

femtosearchbot

Rule

Path

Disallow

fhscan

Rule

Path

Disallow

fimap

Rule

Path

Disallow

firefox/7.0

Rule

Path

Disallow

flashget

Rule

Path

Disallow

flunky

Rule

Path

Disallow

foobot

Rule

Path

Disallow

freeuploader

Rule

Path

Disallow

frontpage

Rule

Path

Disallow

fyberspider

Rule

Path

Disallow

fyrebot

Rule

Path

Disallow

galaxybot

Rule

Path

Disallow

genieo

Rule

Path

Disallow

germcrawler

Rule

Path

Disallow

getintent

Rule

Path

Disallow

getright

Rule

Path

Disallow

getweb

Rule

Path

Disallow

gigablast

Rule

Path

Disallow

gigabot

Rule

Path

Disallow

g-i-g-a-b-o-t

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

gotit

Rule

Path

Disallow

gozilla

Rule

Path

Disallow

go!zilla

Rule

Path

Disallow

grabber

Rule

Path

Disallow

grabnet

Rule

Path

Disallow

grafula

Rule

Path

Disallow

grapefx

Rule

Path

Disallow

grapeshotcrawler

Rule

Path

Disallow

gridbot

Rule

Path

Disallow

gt::www

Rule

Path

Disallow

haansoft

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

harvest

Rule

Path

Disallow

havij

Rule

Path

Disallow

headmasterseo

Rule

Path

Disallow

heritrix

Rule

Path

Disallow

hloader

Rule

Path

Disallow

hmview

Rule

Path

Disallow

htmlparser

Rule

Path

Disallow

http::lite

Rule

Path

Disallow

httrack

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

hybridbot

Rule

Path

Disallow

iblog

Rule

Path

Disallow

idbot

Rule

Path

Disallow

id-search

Rule

Path

Disallow

ilsebot

Rule

Path

Disallow

image fetch

Rule

Path

Disallow

image sucker

Rule

Path

Disallow

indeedbot

Rule

Path

Disallow

indy library

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

infotekies

Rule

Path

Disallow

instabid

Rule

Path

Disallow

intelliseek

Rule

Path

Disallow

interget

Rule

Path

Disallow

internet ninja

Rule

Path

Disallow

internetseer

Rule

Path

Disallow

internetvista monitor

Rule

Path

Disallow

ips-agent

Rule

Path

Disallow

iria

Rule

Path

Disallow

irlbot

Rule

Path

Disallow

iskanie

Rule

Path

Disallow

istellabot

Rule

Path

Disallow

jamesbot

Rule

Path

Disallow

jbrofuzz

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

jetty

Rule

Path

Disallow

jikespider

Rule

Path

Disallow

joc web spider

Rule

Path

Disallow

joomla

Rule

Path

Disallow

jorgee

Rule

Path

Disallow

justview

Rule

Path

Disallow

jyxobot

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword density

Rule

Path

Disallow

kozmosbot

Rule

Path

Disallow

lanshanbot

Rule

Path

Disallow

larbin

Rule

Path

Disallow

leechftp

Rule

Path

Disallow

leechget

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

lftp

Rule

Path

Disallow

libweb

Rule

Path

Disallow

libwhisker

Rule

Path

Disallow

lightspeedsystems

Rule

Path

Disallow

likse

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

linkpadbot

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linksmanager

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

linqiametadatadownloaderbot

Rule

Path

Disallow

linqiarssbot

Rule

Path

Disallow

linqiascrapebot

Rule

Path

Disallow

lipperhey

Rule

Path

Disallow

lipperhey spider

Rule

Path

Disallow

litemage_walker

Rule

Path

Disallow

lmspider

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

lwp-request

Rule

Path

Disallow

lwp::simple

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

magnet

Rule

Path

Disallow

mag-net

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

mail.ru_bot

Rule

Path

Disallow

majestic12

Rule

Path

Disallow

majestic seo

Rule

Path

Disallow

majestic-seo

Rule

Path

Disallow

markmonitor

Rule

Path

Disallow

markwatch

Rule

Path

Disallow

masscan

Rule

Path

Disallow

mass downloader

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

mauibot

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

meanpath bot

Rule

Path

Disallow

mediatoolkitbot

Rule

Path

Disallow

mediawords

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

metauri

Rule

Path

Disallow

mfc_tear_sample

Rule

Path

Disallow

microsoft data access

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

midown tool

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mojeek

Rule

Path

Disallow

mr.4x3

Rule

Path

Disallow

msfrontpage

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

msrabot

Rule

Path

Disallow

muhstik-scan

Rule

Path

Disallow

musobot

Rule

Path

Disallow

name intelligence

Rule

Path

Disallow

nameprotect

Rule

Path

Disallow

navroad

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

needle

Rule

Path

Disallow

nessus

Rule

Path

Disallow

netants

Rule

Path

Disallow

netcraft

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

netlyzer

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

netspider

Rule

Path

Disallow

nettrack

Rule

Path

Disallow

net vampire

Rule

Path

Disallow

netvibes

Rule

Path

Disallow

netzip

Rule

Path

Disallow

nextgensearchbot

Rule

Path

Disallow

nibbler

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

niki-bot

Rule

Path

Disallow

nikto

Rule

Path

Disallow

nimblecrawler

Rule

Path

Disallow

nimbostratus

Rule

Path

Disallow

ninja

Rule

Path

Disallow

nmap

Rule

Path

Disallow

npbot

Rule

Path

Disallow

nutch

Rule

Path

Disallow

obot

Rule

Path

Disallow

octopus

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

offline navigator

Rule

Path

Disallow

oncrawl

Rule

Path

Disallow

openfind

Rule

Path

Disallow

openlinkprofiler

Rule

Path

Disallow

openvas

Rule

Path

Disallow

openvas

Rule

Path

Disallow

orangebot

Rule

Path

Disallow

orangespider

Rule

Path

Disallow

outclicksbot

Rule

Path

Disallow

outfoxbot

Rule

Path

Disallow

pageanalyzer

Rule

Path

Disallow

page analyzer

Rule

Path

Disallow

pagegrabber

Rule

Path

Disallow

page scorer

Rule

Path

Disallow

pagescorer

Rule

Path

Disallow

pandalytics

Rule

Path

Disallow

panscient

Rule

Path

Disallow

papa foto

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

pecl::http

Rule

Path

Disallow

peoplepal

Rule

Path

Disallow

phpcrawl

Rule

Path

Disallow

picscout

Rule

Path

Disallow

picsearch

Rule

Path

Disallow

picturefinder

Rule

Path

Disallow

pimonster

Rule

Path

Disallow

pi-monster

Rule

Path

Disallow

pixray

Rule

Path

Disallow

pleasecrawl

Rule

Path

Disallow

plumanalytics

Rule

Path

Disallow

pockey

Rule

Path

Disallow

poe-component-client-http

Rule

Path

Disallow

probethenet

Rule

Path

Disallow

propowerbot

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

psbot

Rule

Path

Disallow

pump

Rule

Path

Disallow

pxbroker

Rule

Path

Disallow

pycurl

Rule

Path

Disallow

queryn metasearch

Rule

Path

Disallow

quick-crawler

Rule

Path

Disallow

rankactive

Rule

Path

Disallow

rankactivelinkbot

Rule

Path

Disallow

rankflex

Rule

Path

Disallow

rankingbot

Rule

Path

Disallow

rankingbot2

Rule

Path

Disallow

rankivabot

Rule

Path

Disallow

rankurbot

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

reaper

Rule

Path

Disallow

rebelmouse

Rule

Path

Disallow

recorder

Rule

Path

Disallow

redesscrapy

Rule

Path

Disallow

reget

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

ripper

Rule

Path

Disallow

rocketcrawler

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

rssingbot

Rule

Path

Disallow

s1z.ru

Rule

Path

Disallow

salesintelligent

Rule

Path

Disallow

sbider

Rule

Path

Disallow

scanalert

Rule

Path

Disallow

scanbot

Rule

Path

Disallow

scan.lol

Rule

Path

Disallow

scoutjet

Rule

Path

Disallow

scrapy

Rule

Path

Disallow

screaming

Rule

Path

Disallow

screenerbot

Rule

Path

Disallow

searchestate

Rule

Path

Disallow

searchmetricsbot

Rule

Path

Disallow

semrush

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

seokicks

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

seolyticscrawler

Rule

Path

Disallow

seomoz

Rule

Path

Disallow

seoprofiler

Rule

Path

Disallow

seoscanners

Rule

Path

Disallow

seositecheckup

Rule

Path

Disallow

seostats

Rule

Path

Disallow

serpstatbot

Rule

Path

Disallow

shodan

Rule

Path

Disallow

siphon

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

sitebeam

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

siteimprove

Rule

Path

Disallow

sitelockspider

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

sitesucker

Rule

Path

Disallow

site sucker

Rule

Path

Disallow

sitevigil

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

snake

Rule

Path

Disallow

snapbot

Rule

Path

Disallow

snoopy

Rule

Path

Disallow

socialrankiobot

Rule

Path

Disallow

sociscraper

Rule

Path

Disallow

sogouspider

Rule

Path

Disallow

sogou web spider

Rule

Path

Disallow

sosospider

Rule

Path

Disallow

sottopop

Rule

Path

Disallow

spacebison

Rule

Path

Disallow

spammen

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

spanner

Rule

Path

Disallow

sp_auditbot

Rule

Path

Disallow

spbot

Rule

Path

Disallow

spinn3r

Rule

Path

Disallow

sputnikbot

Rule

Path

Disallow

spyfu

Rule

Path

Disallow

sqlmap

Rule

Path

Disallow

sqlworm

Rule

Path

Disallow

sqworm

Rule

Path

Disallow

steeler

Rule

Path

Disallow

stripper

Rule

Path

Disallow

sucker

Rule

Path

Disallow

sucuri

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

swiftbot

Rule

Path

Disallow

sysscan

Rule

Path

Disallow

szukacz

Rule

Path

Disallow

t0phackteam

Rule

Path

Disallow

t8abot

Rule

Path

Disallow

takeout

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

telesphoreo

Rule

Path

Disallow

telesphorep

Rule

Path

Disallow

the intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

thumbor

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

titan

Rule

Path

Disallow

toata

Rule

Path

Disallow

toweyabot

Rule

Path

Disallow

tracemyfile

Rule

Path

Disallow

trendiction

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

trendiction.com

Rule

Path

Disallow

trendiction.de

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

turingos

Rule

Path

Disallow

turnitin

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

twengabot

Rule

Path

Disallow

twice

Rule

Path

Disallow

typhoeus

Rule

Path

Disallow

unisterbot

Rule

Path

Disallow

upflow

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

urly.warning

Rule

Path

Disallow

vacuum

Rule

Path

Disallow

vagabondo

Rule

Path

Disallow

vb project

Rule

Path

Disallow

vci

Rule

Path

Disallow

vericitecrawler

Rule

Path

Disallow

vidiblescraper

Rule

Path

Disallow

virusdie

Rule

Path

Disallow

voideye

Rule

Path

Disallow

voil

Rule

Path

Disallow

voltron

Rule

Path

Disallow

wallpapers/3.0

Rule

Path

Disallow

wallpapershd

Rule

Path

Disallow

wasalive-bot

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

webalta

Rule

Path

Disallow

webauto

Rule

Path

Disallow

web auto

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webcollage

Rule

Path

Disallow

web collage

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webdav

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

web enhancer

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

web fetch

Rule

Path

Disallow

webgo is

Rule

Path

Disallow

webimagecollector

Rule

Path

Disallow

webleacher

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

webmeup-crawler

Rule

Path

Disallow

webpix

Rule

Path

Disallow

web pix

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

web sauger

Rule

Path

Disallow

webshag

Rule

Path

Disallow

websiteextractor

Rule

Path

Disallow

websitequester

Rule

Path

Disallow

website quester

Rule

Path

Disallow

webster

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

websucker

Rule

Path

Disallow

web sucker

Rule

Path

Disallow

webwhacker

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wesee

Rule

Path

Disallow

whack

Rule

Path

Disallow

whacker

Rule

Path

Disallow

whatweb

Rule

Path

Disallow

who.is bot

Rule

Path

Disallow

widow

Rule

Path

Disallow

winhttrack

Rule

Path

Disallow

wiseguys robot

Rule

Path

Disallow

wisenutbot

Rule

Path

Disallow

wonderbot

Rule

Path

Disallow

woobot

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

wprecon

Rule

Path

Disallow

wpscan

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

www-mechanize

Rule

Path

Disallow

www::mechanize

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

x09mozilla

Rule

Path

Disallow

x22mozilla

Rule

Path

Disallow

xaldon webspider

Rule

Path

Disallow

xaldon_webspider

Rule

Path

Disallow

xenu

Rule

Path

Disallow

xpymep1.exe

Rule

Path

Disallow

youdaobot

Rule

Path

Disallow

zade

Rule

Path

Disallow

zauba

Rule

Path

Disallow

zauba.io

Rule

Path

Disallow

zermelo

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zgrab

Rule

Path

Disallow

zitebot

Rule

Path

Disallow

zmeu

Rule

Path

Disallow

zumbot

Rule

Path

Disallow

zyborg

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

updownerbot

Rule

Path

Disallow

ahrefsbot/3.1

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

ahrefsbot/2.0

Rule

Path

Disallow

ahrefs

Rule

Path

Disallow

ahrefs.com

Rule

Path

Disallow

backlink-check.de

Rule

Path

Disallow

backlinkcrawler

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

fasterfox

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

baiduspider

Rule

Path

Disallow

baiduspider-video

Rule

Path

Disallow

baiduspider-image

Rule

Path

Disallow

iwebtool

Rule

Path

Disallow

monitorbacklinks

Rule

Path

Disallow

analyzebacklinks

Rule

Path

Disallow

ranksignals

Rule

Path

Disallow

cognitiveseo

Rule

Path

Disallow

seoprofiler

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

ezooms robot

Rule

Path

Disallow

yodaobot

Rule

Path

Disallow

youdaobot

Rule

Path

Disallow

findlinks robot

Rule

Path

Disallow

shopwiki crawler

Rule

Path

Disallow

checkbot

Rule

Path

Disallow

discobot

Rule

Path

Disallow

internetseer

Rule

Path

Disallow

botrighthere

Rule

Path

Disallow

webzip

Rule

Path

Disallow

larbin

Rule

Path

Disallow

b2w/0.1

Rule

Path

Disallow

copernic

Rule

Path

Disallow

psbot

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

url_spider_pro

Rule

Path

Disallow

cherrypicker

Rule

Path

Disallow

emailcollector

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

copyrightcheck

Rule

Path

Disallow

crescent

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

cheesebot

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

alexibot

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

website quester

Rule

Path

Disallow

moget/2.1

Rule

Path

Disallow

webzip/4.0

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

netants

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

webauto

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

rma

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

asterias

Rule

Path

Disallow

httplib

Rule

Path

Disallow

turingos

Rule

Path

Disallow

spanner

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

harvest/1.5

Rule

Path

Disallow

bullseye/1.0

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

dittospyder

Rule

Path

Disallow

foobot

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

botalot

Rule

Path

Disallow

lwp-trivial/1.34

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

bunnyslippers

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

cosmos

Rule

Path

Disallow

moget

Rule

Path

Disallow

hloader

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

web image collector

Rule

Path

Disallow

the intraformant

Rule

Path

Disallow

true_robot/1.0

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

blowfish/1.0

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

miixpc/4.2

Rule

Path

Disallow

builtbottough

Rule

Path

Disallow

propowerbot/2.14

Rule

Path

Disallow

backdoorbot/1.0

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

vci webviewer vci webviewer win32

Rule

Path

Disallow

vci

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

openfind data gatherer

Rule

Path

Disallow

openfind

Rule

Path

Disallow

xenu's link sleuth 1.1c

Rule

Path

Disallow

xenu's

Rule

Path

Disallow

zeus

Rule

Path

Disallow

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

openbot

Rule

Path

Disallow

url control

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

linkscan/8.1a unix

Rule

Path

Disallow

keyword density/0.9

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

iron33/1.0.2

Rule

Path

Disallow

bookmark search tool

Rule

Path

Disallow

getright/4.2

Rule

Path

Disallow

fairad client

Rule

Path

Disallow

gaisbot

Rule

Path

Disallow

aqua_products

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

flaming attackbot

Rule

Path

Disallow

Comments

Disallow: Sistrix
Disallow: Sistrix
Disallow: Sistrix
Disallow: SEOkicks-Robot
Disallow: jobs.de-Robot
Backlink Analysis
Bot der Leipziger Unister Holding GmbH
http://moz.com/products
http://www.searchmetrics.com
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
http://www.domaintools.com/webmasters/surveybot.php
http://www.seodiver.com/bot
http://openlinkprofiler.org/bot
http://www.wotbox.com/bot/
http://www.opensiteexplorer.org/dotbot
http://moz.com/researchtools/ose/dotbot
http://www.meanpath.com/meanpathbot.html
http://www.backlinktest.com/crawler.html
http://www.brandwatch.com/magpie-crawler/
http://filterdb.iss.net/crawler/
http://webmeup-crawler.com
https://megaindex.com/crawler
http://www.cloudservermarket.com
http://www.trendiction.de/de/publisher/bot
http://www.exalead.com
http://www.career-x.de/bot.html
https://www.lipperhey.com/en/about/
https://www.lipperhey.com/en/about/
https://turnitin.com/robot/crawlerinfo.html
http://help.coccoc.com/
ubermetrics-technologies.com
datenbutler.de
http://searchgears.de/uber-uns/crawling-faq.html
http://commoncrawl.org/faq/
https://www.qwant.com/
http://linkfluence.net/
http://www.botje.com/plukkie.htm
https://www.safedns.com/searchbot
http://www.haosou.com/help/help_3_2.html
http://www.haosou.com/help/help_3_2.html
http://www.moz.com/dp/rogerbot
http://www.openhose.org/bot.html
http://www.screamingfrog.co.uk/seo-spider/
http://thumbsniper.com
http://www.radian6.com/crawler
http://cliqz.com/company/cliqzbot
https://www.aihitdata.com/about
http://www.trendiction.com/en/publisher/bot
http://warebay.com/bot.html
http://www.website-datenbank.de/
http://law.di.unimi.it/BUbiNG.html
http://www.linguee.com/bot; bot@linguee.com
https://www.semrush.com/bot/
www.sentibot.eu
http://velen.io
https://moz.com/help/guides/moz-procedures/what-is-rogerbot
http://www.garlik.com
https://www.gosign.de/typo3-extension/typo3-sicherheitsmonitor/
http://www.siteliner.com/bot
Disallow: Sistrix
Disallow: Sistrix
Disallow: Sistrix
Disallow: SEOkicks-Robot
Disallow: jobs.de-Robot
Backlink Analysis
Bot der Leipziger Unister Holding GmbH
http://moz.com/products
http://www.searchmetrics.com
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
http://www.domaintools.com/webmasters/surveybot.php
http://www.seodiver.com/bot
http://openlinkprofiler.org/bot
http://www.wotbox.com/bot/
http://www.opensiteexplorer.org/dotbot
http://moz.com/researchtools/ose/dotbot
http://www.meanpath.com/meanpathbot.html
http://www.backlinktest.com/crawler.html
http://www.brandwatch.com/magpie-crawler/
http://filterdb.iss.net/crawler/
http://webmeup-crawler.com
https://megaindex.com/crawler
http://www.cloudservermarket.com
http://www.trendiction.de/de/publisher/bot
http://www.exalead.com
http://www.career-x.de/bot.html
https://www.lipperhey.com/en/about/
https://www.lipperhey.com/en/about/
https://turnitin.com/robot/crawlerinfo.html
http://help.coccoc.com/
ubermetrics-technologies.com
datenbutler.de
http://searchgears.de/uber-uns/crawling-faq.html
http://commoncrawl.org/faq/
https://www.qwant.com/
http://linkfluence.net/
http://www.botje.com/plukkie.htm
https://www.safedns.com/searchbot
http://www.haosou.com/help/help_3_2.html
http://www.haosou.com/help/help_3_2.html
http://www.moz.com/dp/rogerbot
http://www.openhose.org/bot.html
http://www.screamingfrog.co.uk/seo-spider/
http://thumbsniper.com
http://www.radian6.com/crawler
http://cliqz.com/company/cliqzbot
https://www.aihitdata.com/about
http://www.trendiction.com/en/publisher/bot
http://warebay.com/bot.html
http://crawler.007ac9.net/
http://www.aboundex.com/crawler/
https://www.adbeat.com/operation_policy
http://advbot.org/bot.html
https://ahrefs.com/robot/
http://alphaseobot.com/bot.html
http://alphaseobot.com/bot.html
https://audisto.com/help/crawler/bot
https://audisto.com/help/crawler/bot
https://awario.com/bots.html
https://awario.com/bots.html
http://www.backlinktest.com/crawler.html
(http://begun.ru/advertiser/technologies/indexer.php)
(https://bitly.com/)
http://webmeup-crawler.com/
http://law.di.unimi.it/BUbiNG.html
https://hypefactors.com/webcrawler
no more active!?
http://catchbot.com/ (via archive.org)
http://commoncrawl.org/big-picture/frequently-asked-questions/
http://www.cmscrawler.com/ (missachtet robots.txt)
http://www.crazywebcrawler.com/
https://www.companiesintheuk.co.uk/bot.html
https://www.dataprovider.com/spider/
https://www.deepcrawl.com/bot/
https://benbernardblog.com/der-bot/
https://support.dlvrit.com/hc/en-us/articles/200402934-How-do-I-block-dlvr-it-from-retrieving-the-feeds-on-my-site-
(http://www.profound.net/domainappender)
http://domainstats.io/our-bot
https://moz.com/researchtools/ose/dotbot (previously Ezooms bot!?)
http://adressendeutschland.de/ (obeys specific rule!?)
https://www.duedil.com/our-crawler
http://www.envolk.com/envolkspiderinfo.html
http://www.expertsearch.nl/spider (redirects to JobdiggerSpider)
https://extlinks.com/Bot.html
http://www.findxbot.com/
(http://www.garlik.com)
http://www.genieo.com/webfilter.html
https://www.gloomar.com/bot
http://www.grapeshot.com/crawler/
http://www.huaweisymantec.com/en/IRL/spider/ (via archive.org)
https://knowledge.hubspot.com/articles/kcs_article/reports/why-do-i-get-an-error-in-page-performance-that-the-crawler-is-blocked-by-robots-txt-for-my-hubspot-staging-domain
https://knowledge.hubspot.com/articles/kcs_article/reports/why-do-i-get-an-error-in-page-performance-that-the-crawler-is-blocked-by-robots-txt-for-my-hubspot-staging-domain
http://www.seograph.net/bot.html
http://www.hypestat.com/bot
https://integralads.com/site-indexing-policy/
https://www.icjobs.de/bot.htm
http://www.idmarch.org/bot.html
(http://implisense.com/)
http://irl.cs.tamu.edu/crawler/
https://cognitiveseo.com/bot.html
https://www.jobbÃ¯Â¿Â½rse.com/bot.htm
http://www.jobdigger.nl/spider/
http://static.kalooga.com/legal/crawler.html
(http://linkfluence.net/)
(http://larbin.sourceforge.net/index-eng.html)
https://www.linkdex.com/en-us/about/bots/ (doesn't obey specific rule)
http://www.linkpad.ru
https://www.lipperhey.com/en/about/website-spider/ (doesn't obey specific rule)
http://ltx71.com/
https://www.brandwatch.com/magpie-crawler/
http://mappydata.net/#eng
http://www.meanpath.com/meanpathbot.html
http://www.metajob.de/the/crawler
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
https://www.neofonie.de/spider/
http://nerdybot.com/
http://www.netseer.com/crawler/
http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx (via arcvhive.org)
http://nutch.apache.org/bot.html
http://filterdb.iss.net/crawler/
https://www.openindex.io/saas/about-our-spider/
http://www.panscient.com/faq.htm
https://pipl.com/bot/
http://www.botje.com/plukkie.htm
https://www.comscore.com/proximic-spider
https://www.semanticjuice.com/web-crawler.php
(bixocrawler)
R6_FeedFetcher + R6_CommentReader (www.radian6.com/crawler)
https://ranksonic.com/ranksonic_sab.html
https://raven.zendesk.com/hc/en-us/articles/203221440-How-do-I-slow-down-Site-Auditor-s-crawl-of-my-website-
(https://raventools.com/)
https://redbot.org/about/
(http://g2pi.tsc.uc3m.es/)
http://riddler.io/about
http://moz.com/help/pro/what-is-rogerbot- (via archive.org)
https://www.safedns.com/searchbot
http://www.secretsearchenginelabs.com/secret-web-crawler.php
http://scoutjet.com/
(Blekkobot)
https://www.screamingfrog.co.uk/seo-spider/user-guide/general/
http://de.semrush.com/bot/
http://sentibot.eu/
https://seocharger.com/robot
http://www.seokicks.de/robot.html
http://crawler.seolytics.net/
http://crawler.seolytics.net/
http://www.searchgears.com/ueber-uns/crawling-faq.html
http://crawler.sistrix.net/
User-agent: sistrix
Disallow: /
http://siteexplorer.info/about.html
https://smabbler.com/en/Home/About
https://www.similartech.com/smtbot
http://www.openlinkprofiler.org/bot
http://nlp.fi.muni.cz/projects/biwec/
http://www.tkl.iis.u-tokyo.ac.jp/~crawler/
http://sur.ly/bot.html
https://swiftype.com/swiftbot
http://www.thumbshots.de/content-39-seite_auszuschliessen.html
https://thumbsniper.com/cat/news/ (via Google cache)
http://search.thunderstone.com/texis/websearch/about.html
(https://www.tineye.com/crawler.html)
http://www.toshiba.co.jp/rdc/about/crawl_info_en.htm
http://www.trendiction.com/de/publisher/bot
https://turnitin.com/robot/crawlerinfo.html
https://blog.ucoz.ru/upolicy/
https://uptime.com/uptimebot
http://www.wise-guys.nl/webcrawler.php
(https://blog.vebidoo.de/vebidoobot/)
https://velen.io/
http://80legs.com/the-80legs-web-crawler/
http://warebay.com/bot.html (expired)
http://rvs.informatik.uni-leipzig.de/bot.php
http://www.wesee.com/bot (via archive.org)
http://www.wikido.com/wikido.php
http://www.wotbox.com/bot/
http://x28.ch/bot.html
http://www.xovibot.net/
https://suite.seozoom.it/bot.html
Majestic SEO
Open Site Explorer
domaintools.com
iwebtool.com/backlink_checker
monitorbacklinks.com/seo-tools/free-backlink-checker
analyzebacklinks.com
ranksignals.com
explorer.cognitiveseo.com
seoprofiler.com

Warnings

14 invalid lines.

hardforkgaming.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

semrushbot

semrushbot-sa

ahrefsbot

dotbot

mj12bot

sistrix

sistrix crawler

sistrix

seokicks-robot

jobs.de-robot

ahrefsbot

unisterbot

dotbot

searchmetricsbot

mj12bot

surveybot

seodiver

spbot

wotbox

dotbot

meanpathbot

backlinkcrawler

magpie-crawler

obot

fr-crawler

blexbot

megaindex.ru

megaindex.com

cloudservermarketspider

trendictionbot

exabot

careerbot

lipperhey-kaus-australis

seoscanners.net

metajobbot

spiderbot

linkstats

jobboersebot

iccrawler

plista

domain re-animator bot

lipperhey-kaus-australis

turnitinbot

coccoc

um-ic

mindupbot

sg-orbiter

ccbot

qwantify

kraken

plukkie

safednsbot

haosouspider

rogerbot

openhosebot

screaming frog seo spider

thumbsniper

r6_commentreader

implisensebot

cliqzbot

aihitbot

trendictionbot

wbsearchbot

python/3.5 aiohttp

toweya.com

netestate

bubing

linguee

semrushbot

semrushbot-sa

sentibot

sentibot

velenpublicwebcrawler

domaincrawler

rogerbot

hardforkgaming.com
robots.txt