nyu.org
robots.txt
Robots Exclusion Standard data for nyu.org
Resource Scan
Scan Details
| Site Domain | nyu.org |
| Base Domain | nyu.org |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-03-23T05:10:06+00:00 |
| Next Scan | 2024-06-21T05:10:06+00:00 |
Last Successful Scan
| Scanned | 2023-11-02T05:08:29+00:00 |
| URL | https://nyu.org/robots.txt |
| Domain IPs | 18.238.49.105, 18.238.49.121, 18.238.49.34, 18.238.49.7, 2600:9000:261f:1400:1:f7e2:cb00:93a1, 2600:9000:261f:3600:1:f7e2:cb00:93a1, 2600:9000:261f:3800:1:f7e2:cb00:93a1, 2600:9000:261f:6800:1:f7e2:cb00:93a1, 2600:9000:261f:7800:1:f7e2:cb00:93a1, 2600:9000:261f:8a00:1:f7e2:cb00:93a1, 2600:9000:261f:ea00:1:f7e2:cb00:93a1, 2600:9000:261f:ec00:1:f7e2:cb00:93a1 |
| Response IP | 18.238.49.121 |
| Found | Yes |
| Hash | 9d977f80b595cd6fb35af84f580ee009a2c931aabe85bf997c9c2565369c6c25 |
| SimHash | 66044b92c20b |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /about/leadership-university-administration/board-of-trustees/newsletter/ |
| Disallow | /alumni/the-hub/ |
| Disallow | /bursar/ |
| Disallow | /dental/dh/ |
| Disallow | /dental/endo/ |
| Disallow | /dental/perio/ |
| Disallow | /dental/histo/ |
| Disallow | /dental/students/ |
| Disallow | /dental/intranet/ |
| Disallow | /fast-facts/ |
| Disallow | /maps/ |
| Disallow | /nyucard/campuscash/admin/ |
| Disallow | /public.affairs/gata/ |
| Disallow | /residential.education/archive/ |
| Disallow | /residential.education/templates/ |
| Disallow | /sustainability/ |
| Disallow | /wagner/newsite/ |
| Disallow | /wagner/newsiteTEST/ |
| Disallow | /*? |
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /about/leadership-university-administration/board-of-trustees/newsletter/ |
| Disallow | /alumni/the-hub/ |
| Disallow | /fast-facts/ |
| Disallow | /*? |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.nyu.edu/sitemap.xml |