Log In Sign Up

sleuth411.com
robots.txt

Robots Exclusion Standard data for sleuth411.com

Resource Scan

Scan Details

Site Domain sleuth411.com
Base Domain sleuth411.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-08-30T03:23:56+00:00
Next Scan 2025-11-28T03:23:56+00:00

Last Successful Scan

Scanned2025-05-08T09:34:44+00:00
URL https://sleuth411.com/robots.txt
Domain IPs 148.251.133.229
Response IP 148.251.133.229
Found Yes
Hash 38cbc0367e7ed0061c9f64c694b3e0da228ee4cd4b4b01e52b023a9c2076f353
SimHash 641409120796

Groups

*

Rule Path Comment
Disallow /private/ -
Disallow /temp/ -
Disallow /backup/ -
Disallow /*?* Block query parameters
Allow /amp/ Allow AMP pages
Allow /tag/ Allow Tag pages

badbot

Rule Path
Disallow /
Allow /public/file.html
Back to top

Other Records

Field Value
sitemap https://sleuth411.com/sitemap.xml
Back to top

Comments

  • Allow all user agents access to AMP and Tag pages
  • Block a specific user agent (e.g., BadBot) from accessing the entire site
  • Allow all user agents to access a specific file
  • Sitemap location
Back to top