Log In Sign Up

antirug.tools
robots.txt

Robots Exclusion Standard data for antirug.tools

Resource Scan

Scan Details

Site Domain antirug.tools
Base Domain antirug.tools
Scan Status Ok
Last Scan2025-10-06T05:53:00+00:00
Next Scan 2025-10-13T05:53:00+00:00

Last Scan

Scanned2025-10-06T05:53:00+00:00
URL http://antirug.tools/robots.txt
Redirect https://www.antirug.ai/robots.txt
Redirect Domain www.antirug.ai
Redirect Base antirug.ai
Domain IPs 192.64.119.74
Redirect IPs 66.33.60.129, 66.33.60.194
Response IP 66.33.60.130
Found Yes
Hash c0a54ff18d98ede1a386bf0866c0dfb3622f45256cdb2e7c954b775d2c693736
SimHash 85243387a102

Groups

*

Rule Path
Allow /

googlebot
googlebot-image

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/

mediapartners-google

Rule Path
Disallow /api/
Disallow /auth/error/
Disallow /auth/signin/
Disallow /auth/signout/
Allow /
Allow /ca/
Allow /glossary/
Allow /blog/
Allow /tokenomics/
Allow /roadmap/
Back to top

Other Records

Field Value
sitemap https://antirugagent.com/sitemap.xml
Back to top

Comments

  • Prevent indexing of error pages and loading states
  • Prevent AdSense bot from crawling pages with minimal content
  • Allow AdSense bot to crawl content-rich pages
  • Sitemap
Back to top