Log In Sign Up

state.mn.us
robots.txt

Robots Exclusion Standard data for state.mn.us

Resource Scan

Scan Details

Site Domain state.mn.us
Base Domain state.mn.us
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-08-23T19:02:58+00:00
Next Scan 2024-11-21T19:02:58+00:00

Last Successful Scan

Scanned2024-04-03T18:52:12+00:00
URL https://state.mn.us/robots.txt
Redirect https://mn.gov/robots.txt
Redirect Domain mn.gov
Redirect Base mn.gov
Domain IPs 66.225.237.206
Redirect IPs 66.225.237.206
Response IP 66.225.237.206
Found Yes
Hash 0a12b43868e195b24cd1bf6a6e9c8e1bb79ecad87d6e55f5ca9708099a25b1ed
SimHash f8e44a58cfe7

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Disallow /cd_upload/Search
Disallow /lawlib/archive/
Disallow /lawlib/briefs/

ultraseek

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Disallow /cd_upload/Search

vse/1.0

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Back to top

Comments

  • Disallow everything until we want to expose the site to external search
  • engines.
  • 0000-1200 GMT is 6PM to 6AM here
  • 4/14/14 Updated for DataExplorer to crawl all state sites.
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top