nln.org
robots.txt

Robots Exclusion Standard data for nln.org

Resource Scan

Scan Details

Site Domain nln.org
Base Domain nln.org
Scan Status Ok
Last Scan2025-10-18T05:53:11+00:00
Next Scan 2025-11-17T05:53:11+00:00

Last Scan

Scanned2025-10-18T05:53:11+00:00
URL https://nln.org/robots.txt
Redirect https://www.nln.org:443/robots.txt
Redirect Domain www.nln.org
Redirect Base nln.org
Domain IPs 52.54.112.189
Redirect IPs 3.223.58.140, 34.202.196.194
Response IP 34.202.196.194
Found Yes
Hash 5d3369b2bfc8402d9c02527f51a70f38902356445ac72e9fd2afec53a5304a76
SimHash 689e993367f0

Groups

*

Rule Path
Disallow /Sitefinity
Disallow /sandbox
Disallow /search-results
Disallow /search-results

Other Records

Field Value
crawl-delay 120

Comments

  • Do not delete /Sitefinity. Never any reason to allow indexing here
  • The same goes for sandbox
  • Also disallow search. We already have it set to "noindex", but keep getting googlebot hits
  • At launch, remove this Disallow. Add any other folders that should not be indexed