idnc.library.illinois.edu
robots.txt
Robots Exclusion Standard data for idnc.library.illinois.edu
Resource Scan
Scan Details
| Site Domain | idnc.library.illinois.edu |
| Base Domain | illinois.edu |
| Scan Status | Ok |
| Last Scan | 2025-02-17T00:26:38+00:00 |
| Next Scan | 2025-03-19T00:26:38+00:00 |
Last Scan
| Scanned | 2025-02-17T00:26:38+00:00 |
| URL | https://idnc.library.illinois.edu/robots.txt |
| Domain IPs | 130.126.162.124 |
| Response IP | 130.126.162.124 |
| Found | Yes |
| Hash | acb0c4f935611e1b1b1ad15f3e6b6fd11a8abf9afb57ef8ef2e6fdc30c6953fe |
| SimHash | 3e7cdd35f775 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | / |
archive.org_bot
| Rule | Path | Comment |
|---|---|---|
| Disallow | /cgi-bin/imageserver.pl | Block imageserver requests |
| Disallow | /?a=a | Block authentication |
| Disallow | /?a=cb | Block content blocking |
| Disallow | /?a=gs | Block graphserver requests |
| Disallow | /?a=is | Block imageserver requests |
| Disallow | /?a=pme | Block publication metadata editor |
| Disallow | /?a=q | Block queries |
| Disallow | /?a=tc | Block text correction |
| Disallow | /?a=ua | Block user account |
| Disallow | /?a=ur | Block user registration |
| Disallow | /*iiif* | Block all IIIF requests |
| Disallow | /?*&e=* | Block all internal links |
| Disallow | /?a=d&d=ITH | Block ITH documents |
| Disallow | /?a=d&d=DIL19690521.2.66 | - |
| Disallow | /?a=d&d=AUE19770128.2.4 | - |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 1 | Wait one second between requests |
bingbot
| Rule | Path | Comment |
|---|---|---|
| Disallow | /cgi-bin/imageserver.pl | Block imageserver requests |
| Disallow | /?a=a | Block authentication |
| Disallow | /?a=cb | Block content blocking |
| Disallow | /?a=gs | Block graphserver requests |
| Disallow | /?a=is | Block imageserver requests |
| Disallow | /?a=pme | Block publication metadata editor |
| Disallow | /?a=q | Block queries |
| Disallow | /?a=tc | Block text correction |
| Disallow | /?a=ua | Block user account |
| Disallow | /?a=ur | Block user registration |
| Disallow | /*iiif* | Block all IIIF requests |
| Disallow | /?*&e=* | Block all internal links |
| Disallow | /?a=d&d=ITH | Block ITH documents |
| Disallow | /?a=d&d=DIL19690521.2.66 | - |
| Disallow | /?a=d&d=AUE19770128.2.4 | - |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 1 | Wait one second between requests |
googlebot
| Rule | Path | Comment |
|---|---|---|
| Disallow | /cgi-bin/imageserver.pl | Block imageserver requests |
| Disallow | /?a=a | Block authentication |
| Disallow | /?a=cb | Block content blocking |
| Disallow | /?a=gs | Block graphserver requests |
| Disallow | /?a=is | Block imageserver requests |
| Disallow | /?a=pme | Block publication metadata editor |
| Disallow | /?a=q | Block queries |
| Disallow | /?a=tc | Block text correction |
| Disallow | /?a=ua | Block user account |
| Disallow | /?a=ur | Block user registration |
| Disallow | /*iiif* | Block all IIIF requests |
| Disallow | /?*&e=* | Block all internal links |
| Disallow | /?a=d&d=ITH | Block ITH documents |
| Disallow | /?a=d&d=DIL19690521.2.66 | - |
| Disallow | /?a=d&d=AUE19770128.2.4 | - |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 1 | Wait one second between requests |
yahoo
| Rule | Path | Comment |
|---|---|---|
| Disallow | /cgi-bin/imageserver.pl | Block imageserver requests |
| Disallow | /?a=a | Block authentication |
| Disallow | /?a=cb | Block content blocking |
| Disallow | /?a=gs | Block graphserver requests |
| Disallow | /?a=is | Block imageserver requests |
| Disallow | /?a=pme | Block publication metadata editor |
| Disallow | /?a=q | Block queries |
| Disallow | /?a=tc | Block text correction |
| Disallow | /?a=ua | Block user account |
| Disallow | /?a=ur | Block user registration |
| Disallow | /*iiif* | Block all IIIF requests |
| Disallow | /?*&e=* | Block all internal links |
| Disallow | /?a=d&d=ITH | Block ITH documents |
| Disallow | /?a=d&d=DIL19690521.2.66 | - |
| Disallow | /?a=d&d=AUE19770128.2.4 | - |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 1 | Wait one second between requests |
Other Records
| Field | Value |
|---|---|
| sitemap | https://idnc.library.illinois.edu/sitemap-illinois-index.xml |
| sitemap | https://idnc.library.illinois.edu/sitemap-illinois-index.xml |
| sitemap | https://idnc.library.illinois.edu/sitemap-illinois-index.xml |
| sitemap | https://idnc.library.illinois.edu/sitemap-illinois-index.xml |
Comments