duluthbudgeteer.com
robots.txt

Robots Exclusion Standard data for duluthbudgeteer.com

Resource Scan

Scan Details

Site Domain duluthbudgeteer.com
Base Domain duluthbudgeteer.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-03-11T16:03:54+00:00
Next Scan 2024-06-09T16:03:54+00:00

Last Successful Scan

Scanned2022-10-23T00:04:30+00:00
URL http://duluthbudgeteer.com/robots.txt
Redirect https://www.duluthnewstribune.com/robots.txt
Redirect Domain www.duluthnewstribune.com
Redirect Base duluthnewstribune.com
Response IP 13.224.250.124, 13.224.250.122, 13.224.250.91, 13.224.250.10
Found Yes
Hash 469faaa716536fe47459db1abba0fb8b8924660ab87c6c012cf0f88bd4f7a161
SimHash 6c30dc606f92

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Warnings

  • 2 invalid lines.