Log In Sign Up

astrowoche.wunderweib.de
robots.txt

Robots Exclusion Standard data for astrowoche.wunderweib.de

Resource Scan

Scan Details

Site Domain astrowoche.wunderweib.de
Base Domain wunderweib.de
Scan Status Ok
Last Scan2025-11-25T21:07:21+00:00
Next Scan 2025-12-02T21:07:21+00:00

Last Scan

Scanned2025-11-25T21:07:21+00:00
URL https://astrowoche.wunderweib.de/robots.txt
Domain IPs 18.159.14.252, 35.156.58.21, 52.57.13.221
Response IP 52.57.13.221
Found Yes
Hash cb8341f580e2ba700d26c14111f1329a4e31a3265d057d39eac38f6f58f03f34
SimHash f2142d11dca5

Groups

*

Rule Path
Allow /
Disallow /fonts/whitelabel/
Disallow /images/whitelabel/
Disallow /status
Disallow /suche
Disallow *?exclude*=*
Disallow /*%26exclude*%3D*
Disallow *?print=*
Disallow *%26print%3D*
Disallow /wer-streamt?query=

gptbot
chatgpt-user
oai-searchbot
perplexitybot
perplexity-user
claude-web
claudebot
claude-user
google-extended
bytespider
timpibot
petalbot
neevaai
blexbot
ccbot
yandexbot
baiduspider
magpie-crawler
dataforseobot
diffbot
meta-externalagent
applebot-extended
cloudvertexbot
deepseekbot
cohere-training-data-crawler
pangubot
ai2bot
omgili
webzio-extended
scrapy
llc
imagesiftbot
mj12bot
yeti
deepseek agent
googleagent-mariner
chatgpt-operator
python-urllib
fuzz faster u fool v2.1.0-dev
anthropic-ai

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://astrowoche.wunderweib.de/sitemap.xml
Back to top

Comments

  • Legal notice: We expressly reserves the right to use its content for commercial text and data mining
  • The use of robots or other automated means to access our site or collect or mine data without the express permission of us is strictly prohibited.
  • We may, in its discretion, permit certain automated access to certain pages,
  • If you would like to apply for permission to crawl us, collect or use data, please email info@bauerxcel.de
Back to top