library.wur.nl
robots.txt
Robots Exclusion Standard data for library.wur.nl
Resource Scan
Scan Details
| Site Domain | library.wur.nl |
| Base Domain | wur.nl |
| Scan Status | Ok |
| Last Scan | 2024-10-30T01:54:54+00:00 |
| Next Scan | 2024-11-29T01:54:54+00:00 |
Last Scan
| Scanned | 2024-10-30T01:54:54+00:00 |
| URL | https://library.wur.nl/robots.txt |
| Domain IPs | 137.224.11.134 |
| Response IP | 137.224.11.134 |
| Found | Yes |
| Hash | 7e6cfeb22916793ded0a9c499509bc7734c8bb7870afa31a18087a6eaed5bc05 |
| SimHash | 630f110905a4 |
Groups
googlebot
| Rule | Path | Comment |
|---|---|---|
| Disallow | /test | please let us test them ourselves... |
| Disallow | /cgi-bin | we do database access there... |
| Disallow | /wiki | - |
| Disallow | /metiswiki | - |
| Disallow | /local | that means: local to our site! |
| Disallow | /usage | our usage statistics |
| Disallow | /atc | agro telematica centrum test |
| Disallow | /nlinet | internal information |
| Disallow | /ctb/files | ctb decisions on the admission of pesticides |
| Disallow | /vogin | temporary |
| Disallow | /foodbib | temporary |
| Disallow | /OpenURL | Please stay away from our Open URL resolver |
| Disallow | /sfxmenu | Please stay away from our Open URL resolver |
| Disallow | /sfx | Please stay away from our Open URL resolver |
| Disallow | /sfx_local | Please stay away from our Open URL resolver |
| Disallow | /link | disallow link resolver |
| Disallow | /WebQuery/cab | Don't crawl the thesaurus |
| Disallow | /xsl | - |
| Disallow | /ns | - |
| Disallow | /java/ | - |
gsa-crawler
| Rule | Path | Comment |
|---|---|---|
| Disallow | /test | please let us test them ourselves... |
| Disallow | /cgi-bin | we do database access there... |
| Disallow | /wiki | - |
| Disallow | /metiswiki | - |
| Disallow | /local | that means: local to our site! |
| Disallow | /usage | our usage statistics |
| Disallow | /atc | agro telematica centrum test |
| Disallow | /nlinet | internal information |
| Disallow | /ctb/files | ctb decisions on the admission of pesticides |
| Disallow | /vogin | temporary |
| Disallow | /foodbib | temporary |
| Disallow | /OpenURL | Please stay away from our Open URL resolver |
| Disallow | /sfxmenu | Please stay away from our Open URL resolver |
| Disallow | /sfx | Please stay away from our Open URL resolver |
| Disallow | /sfx_local | Please stay away from our Open URL resolver |
| Disallow | /link | disallow link resolver |
| Disallow | /WebQuery/cab | Don't crawl the thesaurus |
| Disallow | /xsl | - |
| Disallow | /ns | - |
| Disallow | /java | - |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /test | please let us test them ourselves... |
| Disallow | /cgi-bin | we do database access there... |
| Disallow | /wiki | - |
| Disallow | /metiswiki | - |
| Disallow | /local | that means: local to our site! |
| Disallow | /usage | our usage statistics |
| Disallow | /atc | agro telematica centrum test |
| Disallow | /nlinet | internal information |
| Disallow | /ctb/files | ctb decisions on the admission of pesticides |
| Disallow | /vogin | temporary |
| Disallow | /foodbib | temporary |
| Disallow | /OpenURL | Please stay away from our Open URL resolver |
| Disallow | /sfxmenu | Please stay away from our Open URL resolver |
| Disallow | /sfx | Please stay away from our Open URL resolver |
| Disallow | /sfx_local | Please stay away from our Open URL resolver |
| Disallow | /link | disallow link resolver |
| Disallow | /WebQuery | 210223 BJH: weer enabled, zie nog steeds crawlers die /WebQuery crawlen |
| Disallow | /xsl | - |
| Disallow | /ns | - |
| Disallow | /java | - |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://library.wur.nl/sitemap.xml |
Comments