Log In Sign Up

cache.boston.com
robots.txt

Robots Exclusion Standard data for cache.boston.com

Resource Scan

Scan Details

Site Domain cache.boston.com
Base Domain boston.com
Scan Status Ok
Last Scan2025-08-22T20:54:57+00:00
Next Scan 2025-09-21T20:54:57+00:00

Last Scan

Scanned2025-08-22T20:54:57+00:00
URL https://cache.boston.com/robots.txt
Domain IPs 104.18.18.63, 104.18.19.63, 2606:4700::6812:123f, 2606:4700::6812:133f
Response IP 104.18.18.63
Found Yes
Hash 6d852a4564ec408150650025e9c2a4417c30a10c4d9c0101dc862008492ce092
SimHash ae5a41c0cfe4

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /help/contact_manager.shtml
Back to top

Comments

  • this is a "configuration" file for web robots, so that we can
  • make sure that these crawlers/robots/indexers do not follow links
  • on our site to things like cgi scripts, which could cause some
  • undesirable effects if they stumbled onto our voting scripts!
Back to top