dukeupress.edu
robots.txt
Robots Exclusion Standard data for dukeupress.edu
Resource Scan
Scan Details
Site Domain | dukeupress.edu |
Base Domain | dukeupress.edu |
Scan Status | Ok |
Last Scan | 2024-05-03T05:54:27+00:00 |
Next Scan | 2024-06-02T05:54:27+00:00 |
Last Scan
Scanned | 2024-05-03T05:54:27+00:00 |
URL | https://dukeupress.edu/robots.txt |
Domain IPs | 40.70.147.15 |
Response IP | 40.70.147.15 |
Found | Yes |
Hash | 383cebd9c1f800c64038bd99b68a65e537b74ec320f6b4b126897af9a91b71cb |
SimHash | 7345677109a7 |
Groups
*
Rule | Path |
---|---|
Allow | *.js |
Allow | *.css |
Disallow | /Admin/ |
Disallow | /App_Browsers/ |
Disallow | /App_Code/ |
Disallow | /App_Data/ |
Disallow | /App_Start/ |
Disallow | /App_WebReferences/ |
Disallow | /bin/ |
Disallow | /ClientBin/ |
Disallow | /CMSAdminControls/ |
Disallow | /CMSAPIExamples/ |
Disallow | /CMSDesk/ |
Disallow | /CMSEdit/ |
Disallow | /CMSFormControls/ |
Disallow | /CMSGlobalFiles/ |
Disallow | /CMSHelp/ |
Disallow | /CMSImportFiles/ |
Disallow | /CMSInlineControls/ |
Disallow | /CMSInstall/ |
Disallow | /CMSMasterPages/ |
Disallow | /CMSMessages/ |
Disallow | /CMSModules/ |
Disallow | /CMSResources/ |
Disallow | /CMSSiteManager/ |
Disallow | /CMSSiteUtils/ |
Disallow | /CMSWebParts/ |
Disallow | /cms/getdoc/ |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
Other Records
Field | Value |
---|---|
sitemap | https://dukeupress.edu/crawlersitemap |
Warnings
- 10 invalid lines.
Comments