kommune-aktiv.de
robots.txt

Robots Exclusion Standard data for kommune-aktiv.de

Resource Scan

Scan Details

Site Domain kommune-aktiv.de
Base Domain kommune-aktiv.de
Scan Status Ok
Last Scan2024-06-10T09:29:58+00:00
Next Scan 2024-07-10T09:29:58+00:00

Last Scan

Scanned2024-06-10T09:29:58+00:00
URL https://kommune-aktiv.de/robots.txt
Domain IPs 20.52.60.184
Response IP 20.52.60.184
Found Yes
Hash 2530d2a8ecf0df52544e1935e9fa6b679600205aec3a6330b6f6b053257b5611
SimHash 64d65080c6a7

Groups

*

Rule Path
Disallow /calendar/icalendar.asp
Disallow /scripts/vcard.asp

googlebot

Rule Path
Disallow /direkt.asp

bingbot
adidxbot
bingpreview
msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

ahrefsbot
baiduspider
blexbot
blogpulselive
crawly
dotbot
download ninja
echobot
envolk
euripbot
eurobot
exabot
fetch
flatlandbot
fyberspider
gigabot
gonzo*
grub-client
heise-it-markt-crawler
httrack
iccrawler
iearthworm
infometrics-bot
jakarta commons-httpclient
jobroboter
jobs.de-robot
kalooga
larbin
laycat
libwww
linko
lcc
mail.ru
microsoft.url.control
mj12bot
msiecrawler
nebullabot
netestate ne crawler
netestate foaf crawler
netestate rss crawler
netluchs
ocelli
offline explorer
pixray-seeker
psbot
ruky-bot
scoutjet
searchlink
semager
shopwiki
sitecheck.internetseer.com
sitesnagger
snapbot
sosospider
speedy
surveybot
surveybot_ignoreip
tasapspider
teleport
teleportpro
touche
twiceler
webcopier
webmeasurement-bot
webreaper
website-datenbank.de
websitewiki
webstripper
webzip
wget
woriobot
xenu
yandex
youdaobot
zealbot
zyborg
claudebot

Rule Path
Disallow /

Comments

  • ---------------------------------------------
  • Einzelne Dateien ausschliessen...
  • ---------------------------------------------
  • ---------------------------------------------
  • Website nur alle X Sekunden besuchen...
  • ---------------------------------------------
  • ---------------------------------------------
  • Bots ausschliessen, die nur Traffic machen...
  • ---------------------------------------------