library.brown.edu
robots.txt
Robots Exclusion Standard data for library.brown.edu
Resource Scan
Scan Details
| Site Domain | library.brown.edu |
| Base Domain | brown.edu |
| Scan Status | Ok |
| Last Scan | 2025-08-30T20:24:43+00:00 |
| Next Scan | 2025-09-29T20:24:43+00:00 |
Last Scan
| Scanned | 2025-08-30T20:24:43+00:00 |
| URL | https://library.brown.edu/robots.txt |
| Domain IPs | 128.148.254.67 |
| Response IP | 128.148.254.67 |
| Found | Yes |
| Hash | 57aafbff02872619383c273242a92c4f181ab8f775d7befb685d40edaa1bebe4 |
| SimHash | 2acc6942c874 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /etd/pdfs/ |
| Disallow | /etd/pdfs |
| Disallow | /reserves/pdffiles/ |
| Disallow | /reserves/pdffiles |
| Disallow | /testzone/ |
| Disallow | /testzone |
| Disallow | /lincoln2/ |
| Disallow | /lincoln2 |
| Disallow | /koetting2/ |
| Disallow | /koetting2 |
| Disallow | /imago_brunoniana/ |
| Disallow | /imago_brunoniana |
| Disallow | /imago_brunoniana2/ |
| Disallow | /imago_brunoniana2 |
| Disallow | /find/Summon/ |
| Disallow | /find/Summon |
| Disallow | /find/Discover/ |
| Disallow | /easyarticle/ |
| Disallow | /easyarticle |
| Disallow | /ezborrow/admin/ |
| Disallow | /services/book_locator/ |
| Disallow | /titles/ |
| Disallow | /pdfs/ |
| Disallow | /pdfs |
| Disallow | /jpegs/ |
| Disallow | /jpegs |
| Disallow | /projects/rome/books |
| Disallow | /projects/rome/pages |
| Disallow | /projects/usep/inscription/Pictures/Thumbnails |
| Disallow | /dashboard |
| Disallow | /find |
| Disallow | /search/usep |
| Disallow | /usep/results |
| Disallow | /intranet |
| Disallow | /publications/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
googlebot
googlebot-image
msnbot-media
bingbot
| Rule | Path |
|---|---|
| Disallow | /etd/pdfs/ |
| Disallow | /etd/pdfs |
| Disallow | /reserves/pdffiles/ |
| Disallow | /reserves/pdffiles |
| Disallow | /testzone/ |
| Disallow | /testzone |
| Disallow | /lincoln2/ |
| Disallow | /lincoln2 |
| Disallow | /koetting2/ |
| Disallow | /koetting2 |
| Disallow | /imago_brunoniana/ |
| Disallow | /imago_brunoniana |
| Disallow | /imago_brunoniana2/ |
| Disallow | /imago_brunoniana2 |
| Disallow | /find/Summon/ |
| Disallow | /find/Summon |
| Disallow | /easyarticle/ |
| Disallow | /easyarticle |
| Disallow | /ezborrow/admin/ |
| Disallow | /services/book_locator/ |
| Disallow | /titles/ |
| Disallow | /pdfs/ |
| Disallow | /pdfs |
| Disallow | /jpegs/ |
| Disallow | /jpegs |
| Disallow | /projects/rome/books |
| Disallow | /projects/rome/pages |
| Disallow | /projects/usep/inscription/Pictures/Thumbnails |
| Disallow | /dashboard |
| Disallow | /find |
| Disallow | /intranet |
| Disallow | /publications/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Warnings
- 1 invalid line.