bigshopper.pl
robots.txt

Robots Exclusion Standard data for bigshopper.pl

Resource Scan

Scan Details

Site Domain bigshopper.pl
Base Domain bigshopper.pl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-11T22:17:25+00:00
Next Scan 2025-12-10T22:17:25+00:00

Last Successful Scan

Scanned2025-08-13T14:21:50+00:00
URL https://bigshopper.pl/robots.txt
Domain IPs 104.26.10.82, 104.26.11.82, 172.67.71.17, 2606:4700:20::681a:a52, 2606:4700:20::681a:b52, 2606:4700:20::ac43:4711
Response IP 172.67.71.17
Found Yes
Hash 59bf2173ec9e5cf3d583e449edf688d0b0ddae43e6dd1887dd875c2f0b6acbb9
SimHash d05cc940e898

Groups

*

Rule Path
Disallow */shop/
Disallow */send_mail/
Disallow */scripts/send_faq_searchbardata.php

adscanner

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

timpibot/0.8

Rule Path
Disallow /

Comments

  • User-agent: anthropic-ai
  • Disallow: /
  • User-agent: ClaudeBot
  • Disallow: /
  • User-agent: Claude-Web
  • Disallow: /
  • User-agent: GPTBot
  • Disallow: /