Log In Sign Up

onlinebaufuchs.de
robots.txt

Robots Exclusion Standard data for onlinebaufuchs.de

Resource Scan

Scan Details

Site Domain onlinebaufuchs.de
Base Domain onlinebaufuchs.de
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-11-13T03:23:06+00:00
Next Scan 2025-11-20T03:23:06+00:00

Last Successful Scan

Scanned2025-10-12T07:51:11+00:00
URL https://onlinebaufuchs.de/robots.txt
Redirect https://www.onlinebaufuchs.de/robots.txt
Redirect Domain www.onlinebaufuchs.de
Redirect Base onlinebaufuchs.de
Domain IPs 116.202.214.230
Redirect IPs 116.202.214.230
Response IP 116.202.214.230
Found Yes
Hash ac3952c1d18a1a307426c1bd5d96f243aaaa5f6eef64c90ec2582e3e89306462
SimHash 73141951c2c5

Groups

aihitbot

Rule Path
Disallow /

bigsur.ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

mistralai-user

Rule Path
Disallow /

novaact

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

gptbot

Rule Path
Allow /

claude-searchbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

ai2bot-dolma

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

addsearchbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleother

Rule Path
Disallow /

googleother-image

Rule Path
Disallow /

googleother-video

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

mycentralaiscraperbot

Rule Path
Disallow /

facebot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

linerbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Allow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

popscreen bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

wpbot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

amazon-kendra-web-crawler-*

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-user

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

novaact

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

tiktokspider

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

dataprovider.com

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

yak

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.onlinebaufuchs.de/sitemap1.xml
sitemap https://www.onlinebaufuchs.de/sitemap2.xml
sitemap https://www.onlinebaufuchs.de/sitemap3.xml
sitemap https://www.onlinebaufuchs.de/sitemap4.xml
Back to top

Comments

  • Aktuelle Liste der robots.txt vom 08.10.2025
  • ALLE bekannten KI Bots > https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
  • User-agent: ia_archiver
  • Disallow: /
  • User-agent: ia_archiver-web.archive.org
  • Disallow: /
  • XML-Sitemap
Back to top

Warnings

  • 2 invalid lines.
Back to top