Log In Sign Up

search.nyu.edu
robots.txt

Robots Exclusion Standard data for search.nyu.edu

Resource Scan

Scan Details

Site Domain search.nyu.edu
Base Domain nyu.edu
Scan Status Ok
Last Scan2025-07-01T20:01:59+00:00
Next Scan 2025-07-31T20:01:59+00:00

Last Scan

Scanned2025-07-01T20:01:59+00:00
URL https://search.nyu.edu/robots.txt
Domain IPs 2.58.104.10, 2.58.104.11
Response IP 2.58.104.11
Found Yes
Hash b8e59558ebcdd3ec8cbbdf32dc5560ec7f0d400daaa68a43b4672e08c84928ab
SimHash b339fd594312

Groups

*

Rule Path
Disallow /search/
Disallow /s/
Back to top

Comments

  • $Id$
  • Standard robots.txt for search system distribution. Must be located in
  • the top-level directory of the web server document hierarchy.
  • Ref: http://www.robotstxt.org/wc/exclusion.html
Back to top