Log In Sign Up

classbug.com
robots.txt

Robots Exclusion Standard data for classbug.com

Resource Scan

Scan Details

Site Domain classbug.com
Base Domain classbug.com
Scan Status Ok
Last Scan2025-03-07T20:06:34+00:00
Next Scan 2025-04-06T20:06:34+00:00

Last Scan

Scanned2025-03-07T20:06:34+00:00
URL https://classbug.com/robots.txt
Domain IPs 104.21.92.251, 172.67.201.182, 2606:4700:3030::6815:5cfb, 2606:4700:3033::ac43:c9b6
Response IP 172.67.201.182
Found Yes
Hash 2e28cc6af71779bdf63a8743cbb802aab0016cbe5ebe589821b85c6604b9a75c
SimHash 1df6cb2bd5b7

Groups

baiduspider

Rule Path
Disallow /

*

Rule Path
Disallow /users/*
Disallow /signin_client$
Disallow /hay

blexbot

Rule Path
Disallow /
Back to top

Comments

  • Baiduspider
  • Google bot gets stuck trying to view peoples user profiles.
  • No reason for any bots to go there.
  • Attempts to crawl pages that don't exist, not friendly
Back to top