schoenen-dunk.de
robots.txt

Robots Exclusion Standard data for schoenen-dunk.de

Resource Scan

Scan Details

Site Domain schoenen-dunk.de
Base Domain schoenen-dunk.de
Scan Status Ok
Last Scan2026-02-10T08:20:23+00:00
Next Scan 2026-02-11T08:20:23+00:00

Last Scan

Scanned2026-02-10T08:20:23+00:00
URL https://schoenen-dunk.de/robots.txt
Redirect https://www.schoenen-dunk.de/robots.txt
Redirect Domain www.schoenen-dunk.de
Redirect Base schoenen-dunk.de
Domain IPs 46.224.71.249
Redirect IPs 46.224.71.249
Response IP 46.224.71.249
Found Yes
Hash 0294f55cd3559daa1445d33d4e624d1bc66d85938a3425e47f1d798495a3c602
SimHash 7010d9d0c0f0

Groups

*

Rule Path
Disallow /flags
Disallow /forum*
Disallow /login
Disallow /pages/update.php
Disallow /post-queue

Other Records

Field Value
crawl-delay 1

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.schoenen-dunk.de/sitemap.xml

Comments

  • block known [AI and some other] bots