shaunwitriol.com
robots.txt

Robots Exclusion Standard data for shaunwitriol.com

Resource Scan

Scan Details

Site Domain shaunwitriol.com
Base Domain shaunwitriol.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-03-07T16:29:16+00:00
Next Scan 2025-06-05T16:29:16+00:00

Last Successful Scan

Scanned2024-05-18T10:23:53+00:00
URL https://shaunwitriol.com/robots.txt
Domain IPs 104.21.55.49, 172.67.144.209, 2606:4700:3033::ac43:90d1, 2606:4700:3035::6815:3731
Response IP 104.21.55.49
Found Yes
Hash 894aee9ad76754025ffdb24214c53557fb6d1e877e4a7cb775bc57c25823f9d6
SimHash d05dd340e790

Groups

*

Rule Path
Disallow
Disallow /index.php?m=api*
Disallow /template/*

semrushbot

Rule Path
Disallow /

my-tiny-bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /