clarivate.com
robots.txt

Groups

*

Other Records

yandex
bingbot
msnbot
weneobot
acunetix
afilias web mining tool
ahrefsbot
alexibot
aqua_products
blexbot
bpimagewalker
bpimagewalker*
bubing
backdoorbot/1.0
backlinkcrawler
baiduspider
birubot
black hole
blowfish/1.0
bookmark search tool
botalot
botonparade
botrighthere
builtbottough
bullseye
bullseye/1.0
bunnyslippers
comodo ssl checker
catchbot
cegbfeieh
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
comodo-certificates-spider
content crawler
copernic
copyrightcheck
crescent
crescent internet toolpak http ole control v.1.0
dcpbot
diibot
daumoa
dittospyder
ec2linkfinder
ezooms
edisterbot
emailcollector
emailsiphon
emailwolf
erocrawler
eurobot
exdomain
exabot
extractorpro
fairad client
fasterfox
flaming attackbot
foobot
gaisbot
getright/4.2
gigabot
httrack 3.0
harvest/1.5
httrack
huaweisymantecspider
iccrawler - icjobs
iconsurf
infonavirobot
iron33/1.0.2
jennybot
jetbot
jikespider
kaloogabot
kenjin spider
keyword density/0.9
lnspiderguy
lexibot
linkscan
linkscan/8.1a unix
linkwalker
linkextractorpro
miixpc
miixpc/4.2
mj12bot
mlbot
msiecrawler
mata hari
mediapartners-google
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
mister pix
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 2000)
mozilla/4.0 (compatible; msie 4.0; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 98)
mozilla/4.0 (compatible; msie 4.0; windows me)
mozilla/4.0 (compatible; msie 4.0; windows nt)
mozilla/4.0 (compatible; msie 4.0; windows xp)
nicerspro
nerdbynature.bot
netants
netmechanic
nutch
offline explorer
oneriot
openbot
openfind
openfind data gathere
openfind data gatherer
openindexspider
opidoobot
oracle ultra search
pagepeeker
perman
pixray-seeker
propowerbot/2.14
prowebwalker
purebot
python-urllib
qualidator*
quepasacreep
queryn metasearch
rma
radiation retriever 1.1
repomonkey
repomonkey bait & tackle/v1.01
reverseget
roverbot
seokicks
seokicks-robot
swebot
scooter
scoutjet
screaming frog seo spider
semrushbot
seznambot
sitesnagger
sitebot
slurp
slysearch
smetrics
spankbot
speedy
spinn3r
surveybot
szukacz/1.4
teleport
teleportpro
telesoft
the intraformant
the intraformantuser-agent: *thunderstone*
thenomad
tighttwatbot
tineye
titan
true_robot
true_robot/1.0
turnitinbot
turnitinbot/1.5
twiceler
url control
url_spider_pro
urly warning
unister*
unisterbot
unwindfetchor
updownerbot
vci
vci webviewer
vci webviewer vci webviewer win32
voilabot
wbsearchbot
www-collector-e
web image collector
webauto
webbandit
webbandit/3.50
webcapture 2.0
webcopier
webcopier v.2.2
webcopier v3.2a
webenhancer
webreaper
webripper
websauger
webstripper
webzip/4.21
webzip/5.0
webzip
webzip/4.0
webinator
webmastercoffee
webmasterworldforumbot
website quester
webster pro
wget
wget/1.4.0
wget/1.5.2
wget/1.5.3
wget/1.6
wget/1.7
wget/1.8
wget/1.8.1
wget/1.8.1+cvs
wget/1.8.2
wget/1.9-beta
xenu's
xenu's link sleuth 1.1c
yahoo pipes 1.0
yahoo pipes 2.0
yandexbot
yeti
yeti-mobile
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
zookabot
zyborg
aihitbot
aipbot
asterias
b2w/0.1
bdbrandprotect
becomebot
bixolabs
cosmos
discobot
ecatch
findfiles.net
findlinks
gonzo
grub
grub-client
hloader
htdig
httplib
humanlinks
icjobs
ia_archiver
ia_archiver/1.6
ichiro
ips-agent
larbin
lb-spider
lex
libweb/clshttp
linkdex.com
looksmart
lwp-trivial
lwp-trivial/1.34
magpie-crawler
moget
moget/2.1
mozilla/4
mozilla/5
msnbot-media
netestate ne crawler
obot
picmole
plukkie
psbot
schrein
search17
searchpreview
sistrix
spanner
spbot
suggybot
suzuran
teoma
tocrawl/urldispatcher
turingos
wget
yacybot
ccbot
chatgpt-user
gptbot
google-extended
omgilibot

Other Records

Other Records

Comments

• Slow down crawlers
• disallow crawler

Warnings

• 1 invalid line.