Log In Sign Up

ourobets.com
robots.txt

Robots Exclusion Standard data for ourobets.com

Resource Scan

Scan Details

Site Domain ourobets.com
Base Domain ourobets.com
Scan Status Ok
Last Scan2025-08-26T12:36:31+00:00
Next Scan 2025-09-25T12:36:31+00:00

Last Scan

Scanned2025-08-26T12:36:31+00:00
URL https://www.ourobets.com/robots.txt
Domain IPs 104.21.53.103, 172.67.212.1, 2606:4700:3034::ac43:d401, 2606:4700:3035::6815:3567
Response IP 104.21.53.103
Found Yes
Hash 43a22e60a9cad5af85868e421549c1770fbb17c50f867daf0d2394956a993179
SimHash 9c7d3d310ce6

Groups

claudebot

Rule Path
Disallow /
Back to top

Comments

  • This robots.txt file controls crawling of URLs under https://example.com.
  • All crawlers are disallowed to crawl files in the "includes" directory, such
  • as .css, .js, but Google needs them for rendering, so Googlebot is allowed
  • to crawl them.
Back to top