besserladen.info
robots.txt

Robots Exclusion Standard data for besserladen.info

Resource Scan

Scan Details

Site Domain besserladen.info
Base Domain besserladen.info
Scan Status Ok
Last Scan2025-11-27T11:28:01+00:00
Next Scan 2025-12-04T11:28:01+00:00

Last Scan

Scanned2025-11-27T11:28:01+00:00
URL http://besserladen.info/robots.txt
Redirect https://besserladen.de/robots.txt
Redirect Domain besserladen.de
Redirect Base besserladen.de
Domain IPs 85.13.150.179
Redirect IPs 85.13.150.179
Response IP 85.13.150.179
Found Yes
Hash 88c5d0a0cdb0f575951fcc39d471241d9790b080a9467b8f140ccef92d04044b
SimHash 524c4b607620

Groups

scrapy

Rule Path
Allow /

suggybot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

kraken

Rule Path
Disallow /

genieo

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

yeti

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

seobility

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

ssearch crawler

Rule Path
Disallow /

publiclibraryarchive

Rule Path
Disallow /

publiclibraryarchive.org

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

wesee

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

netlyzer fastprobe

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /