acsi.org
robots.txt

Robots Exclusion Standard data for acsi.org

Resource Scan

Scan Details

Site Domain acsi.org
Base Domain acsi.org
Scan Status Ok
Last Scan2025-10-14T19:22:50+00:00
Next Scan 2025-11-13T19:22:50+00:00

Last Scan

Scanned2025-10-14T19:22:50+00:00
URL https://acsi.org/robots.txt
Redirect https://www.acsi.org:443/robots.txt
Redirect Domain www.acsi.org
Redirect Base acsi.org
Domain IPs 3.220.41.178
Redirect IPs 3.224.187.217, 44.217.76.202
Response IP 44.217.76.202
Found Yes
Hash 686293994e26d63001c27798c0a9d83f99138050fd4e7973a603cd85dbcbd204
SimHash 6d5f193767f1

Groups

*

Rule Path
Disallow /Sitefinity
Disallow /sandbox
Disallow /textbooks
Disallow /search-results

Other Records

Field Value
crawl-delay 20

Comments

  • Do not delete /Sitefinity. Never any reason to allow indexing here
  • The same goes for sandbox
  • Also disallow search. We already have it set to "noindex", but keep getting googlebot hits
  • At launch, remove this Disallow. Add any other folders that should not be indexed