Log In Sign Up

antirug.ai
robots.txt

Robots Exclusion Standard data for antirug.ai

Resource Scan

Scan Details

Site Domain antirug.ai
Base Domain antirug.ai
Scan Status Ok
Last Scan2025-10-12T02:20:51+00:00
Next Scan 2025-10-19T02:20:51+00:00

Last Scan

Scanned2025-10-12T02:20:51+00:00
URL https://antirug.ai/robots.txt
Redirect https://www.antirug.ai/robots.txt
Redirect Domain www.antirug.ai
Redirect Base antirug.ai
Domain IPs 76.76.21.21
Redirect IPs 66.33.60.130, 66.33.60.35
Response IP 76.76.21.123
Found Yes
Hash c0a54ff18d98ede1a386bf0866c0dfb3622f45256cdb2e7c954b775d2c693736
SimHash 85243387a102

Groups

*

Rule Path
Allow /

googlebot
googlebot-image

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/

mediapartners-google

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/
Allow /
Allow /ca/
Allow /glossary/
Allow /blog/
Allow /tokenomics/
Allow /roadmap/
Back to top

Other Records

Field Value
sitemap https://antirugagent.com/sitemap.xml
Back to top

Comments

  • Prevent indexing of error pages and loading states
  • Prevent AdSense bot from crawling pages with minimal content
  • Allow AdSense bot to crawl content-rich pages
  • Sitemap
Back to top