Log In Sign Up

antirugagent.com
robots.txt

Robots Exclusion Standard data for antirugagent.com

Resource Scan

Scan Details

Site Domain antirugagent.com
Base Domain antirugagent.com
Scan Status Ok
Last Scan2025-10-12T20:04:15+00:00
Next Scan 2025-10-19T20:04:15+00:00

Last Scan

Scanned2025-10-12T20:04:15+00:00
URL https://antirugagent.com/robots.txt
Redirect https://www.antirug.ai/robots.txt
Redirect Domain www.antirug.ai
Redirect Base antirug.ai
Domain IPs 76.76.21.21
Redirect IPs 66.33.60.66, 66.33.60.67
Response IP 76.76.21.142
Found Yes
Hash c0a54ff18d98ede1a386bf0866c0dfb3622f45256cdb2e7c954b775d2c693736
SimHash 85243387a102

Groups

*

Rule Path
Allow /

googlebot
googlebot-image

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/

mediapartners-google

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/
Allow /
Allow /ca/
Allow /glossary/
Allow /blog/
Allow /tokenomics/
Allow /roadmap/
Back to top

Other Records

Field Value
sitemap https://antirugagent.com/sitemap.xml
Back to top

Comments

  • Prevent indexing of error pages and loading states
  • Prevent AdSense bot from crawling pages with minimal content
  • Allow AdSense bot to crawl content-rich pages
  • Sitemap
Back to top