rwth-aachen.de
robots.txt
Robots Exclusion Standard data for rwth-aachen.de
Resource Scan
Scan Details
| Site Domain | rwth-aachen.de |
| Base Domain | rwth-aachen.de |
| Scan Status | Ok |
| Last Scan | 2024-06-22T02:02:44+00:00 |
| Next Scan | 2024-07-22T02:02:44+00:00 |
Last Scan
| Scanned | 2024-06-22T02:02:44+00:00 |
| URL | https://rwth-aachen.de/robots.txt |
| Domain IPs | 137.226.107.63, 2a00:8a60:450::107:63 |
| Response IP | 137.226.107.63 |
| Found | Yes |
| Hash | 86522cfec45c0317fa1b12557fa3aaacd94ae57278123a24d1cf88a6114586a3 |
| SimHash | 7b3832a86ca1 |
Groups
afilias web mining tool
afiliaswebminingtool
aihitbot
askpeterbot
bdbrandprotect
bpimagewalker*
bpimagewalker
blexbot
bubing
catchbot
comodo ssl checker
comodosslchecker
comodo-certificates-spider
content crawler
contentcrawler
curl
dcpbot
discobot
docomo/2.0
dotbot
drupact
ec2linkfinder
edisterbot
erocrawler
exb language crawler
ezooms
findlinks
gonzo
gslfbot
htdig
huaweisymantecspider
icjobs
infohelfer
ips-agent
ipsagent
it2media-domain-crawler
java
kaloogabot
larbin
lb-spider
lex
libwww-perl
linkdex.com
lssrocketcrawler
ltx71 - (http://ltx71.com/)
ltx71
magpie-crawler
majesticseo
mail.ru_bot
mia
mj12bot
mlbot
msiecrawler
msnbot
netestate ne crawler
netestatenecrawler
nutch
obot
openindexspider
opidoobot
pagepeeker
picmole
pixray-seeker
psbot
qualidator*
reverseget
schrein
scooter
searchmetricsbot
search17
semrushbot
semrushbot-sa
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm
seznambot
sistrix
sistrix crawler
sistrix
slysearch
solomonobot
solomonolinkchecker
sosospider
spbot
spiderlytics
suggybot
surveybot
swebot
thunderstone
turnitinbot
tineye
unisterbot
unister
unister*
updownerbot
wbsearchbot
web image collector
webauto
webbandit
webbandit/3.50
webcopier
webinator
webenhancer
webmastercoffee
webmasterworld extractor
webmasterworldforumbot
webofantbot
websauger
website quester
webster pro
webstripper
webvac
webzip
webzip/4.0
wotbox
www-collector-e
x28-job-bot
xovi
xovibot
yandex
yacybot
yeti
yeti-mobile
youdaobot
xenu's link sleuth
xenu's
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Warnings
- 4 invalid lines.