bigshopper.de
robots.txt

Robots Exclusion Standard data for bigshopper.de

Resource Scan

Scan Details

Site Domain bigshopper.de
Base Domain bigshopper.de
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-17T13:00:54+00:00
Next Scan 2026-03-17T13:00:54+00:00

Last Successful Scan

Scanned2025-08-18T13:42:00+00:00
URL https://bigshopper.de/robots.txt
Domain IPs 104.26.2.36, 104.26.3.36, 172.67.72.18, 2606:4700:20::681a:224, 2606:4700:20::681a:324, 2606:4700:20::ac43:4812
Response IP 104.26.2.36
Found Yes
Hash 59bf2173ec9e5cf3d583e449edf688d0b0ddae43e6dd1887dd875c2f0b6acbb9
SimHash d05cc940e898

Groups

*

Rule Path
Disallow */shop/
Disallow */send_mail/
Disallow */scripts/send_faq_searchbardata.php

adscanner

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

timpibot/0.8

Rule Path
Disallow /

Comments

  • User-agent: anthropic-ai
  • Disallow: /
  • User-agent: ClaudeBot
  • Disallow: /
  • User-agent: Claude-Web
  • Disallow: /
  • User-agent: GPTBot
  • Disallow: /