Log In Sign Up

nsf.org
robots.txt

Robots Exclusion Standard data for nsf.org

Resource Scan

Scan Details

Site Domain nsf.org
Base Domain nsf.org
Scan Status Ok
Last Scan2024-09-22T21:48:58+00:00
Next Scan 2024-10-06T21:48:58+00:00

Last Scan

Scanned2024-09-22T21:48:58+00:00
URL https://nsf.org/robots.txt
Redirect https://www.nsf.org/robots.txt
Redirect Domain www.nsf.org
Redirect Base nsf.org
Domain IPs 52.191.90.193
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash b8acc8b872065881b06ae912611fad4b08918dd6bef1e92a1b2e56a2241f505c
SimHash 83701d023fb0

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /*?serviceType=
Back to top

Other Records

Field Value
sitemap https://www.nsf.org/sitemaps-2-sitemap.xml
Back to top

Comments

  • robots.txt for https://www.nsf.org/
  • live - don't allow web crawlers to index cpresources/ or vendor/
Back to top