Log In Sign Up

ast.org
robots.txt

Robots Exclusion Standard data for ast.org

Resource Scan

Scan Details

Site Domain ast.org
Base Domain ast.org
Scan Status Ok
Last Scan2025-10-21T02:39:43+00:00
Next Scan 2025-11-20T02:39:43+00:00

Last Scan

Scanned2025-10-21T02:39:43+00:00
URL https://www.ast.org/robots.txt
Domain IPs 104.21.42.47, 172.67.156.54, 2606:4700:3033::6815:2a2f, 2606:4700:3033::ac43:9c36
Response IP 172.67.156.54
Found Yes
Hash b2e71c85411bac28caaff9622a0cd25b69987365562b7aa8da6ee419a1e5ac45
SimHash 21019ed2c5b2

Groups

*

Rule Path
Disallow /workarea/
Disallow /widgets/
Allow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ast.org/sitemap.xml
Back to top

Comments

  • robots.txt for www.ast.org
  • Block OpenAI GPTBot and Google-Extended (optional, for AI crawler control)
  • Sitemap for search engines
Back to top