dl.begellhouse.com
robots.txt
Robots Exclusion Standard data for dl.begellhouse.com
Resource Scan
Scan Details
| Site Domain | dl.begellhouse.com |
| Base Domain | begellhouse.com |
| Scan Status | Ok |
| Last Scan | 2024-09-27T06:29:35+00:00 |
| Next Scan | 2024-10-27T06:29:35+00:00 |
Last Scan
| Scanned | 2024-09-27T06:29:35+00:00 |
| URL | https://dl.begellhouse.com/robots.txt |
| Domain IPs | 169.59.241.40, 2607:f0d0:1f02:45::1 |
| Response IP | 169.59.241.40 |
| Found | Yes |
| Hash | 07a0f76da5076cdd2f6f12ab2b17afa6ed1390e4efac023212c5f6477fc50b00 |
| SimHash | 88467b4544d3 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /flash/ |
| Disallow | /i/ |
| Disallow | /js/ |
| Disallow | /images/ |
| Disallow | /menu/ |
| Disallow | /st/ |
| Disallow | /pdf/ |
| Disallow | /files/ |
| Disallow | /lib/ |
| Disallow | /whosite_authors/ |
| Disallow | /badmin/ |
| Disallow | /user/ |
| Disallow | /order/ |
| Disallow | /search/ |
| Disallow | /temp/ |
| Disallow | /Shibboleth.sso/ |
| Disallow | /*?sgstd=* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
googlebot
| Rule | Path |
|---|---|
| Disallow | /flash/ |
| Disallow | /i/ |
| Disallow | /js/ |
| Disallow | /images/ |
| Disallow | /menu/ |
| Disallow | /st/ |
| Disallow | /pdf/ |
| Disallow | /lib/ |
| Disallow | /whosite_authors/ |
| Disallow | /badmin/ |
| Disallow | /search/ |
| Disallow | /temp/ |
| Disallow | /Shibboleth.sso/ |
| Disallow | /*?sgstd=* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
librabot
| Rule | Path |
|---|---|
| Disallow | /flash/ |
| Disallow | /i/ |
| Disallow | /js/ |
| Disallow | /images/ |
| Disallow | /menu/ |
| Disallow | /st/ |
| Disallow | /pdf/ |
| Disallow | /lib/ |
| Disallow | /whosite_authors/ |
| Disallow | /badmin/ |
| Disallow | /search/ |
| Disallow | /temp/ |
| Disallow | /Shibboleth.sso/ |
| Disallow | /*?sgstd=* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
turnitinbot
| Rule | Path |
|---|---|
| Disallow | /flash/ |
| Disallow | /i/ |
| Disallow | /js/ |
| Disallow | /images/ |
| Disallow | /menu/ |
| Disallow | /st/ |
| Disallow | /pdf/ |
| Disallow | /lib/ |
| Disallow | /whosite_authors/ |
| Disallow | /badmin/ |
| Disallow | /search/ |
| Disallow | /temp/ |
| Disallow | /Shibboleth.sso/ |
| Disallow | /*?sgstd=* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
Comments