Log In Sign Up

mn.gov
robots.txt

Robots Exclusion Standard data for mn.gov

Resource Scan

Scan Details

Site Domain mn.gov
Base Domain mn.gov
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-08-30T02:55:58+00:00
Next Scan 2024-11-28T02:55:58+00:00

Last Successful Scan

Scanned2024-04-10T02:53:16+00:00
URL https://mn.gov/robots.txt
Domain IPs 66.225.237.206
Response IP 66.225.237.206
Found Yes
Hash 0a12b43868e195b24cd1bf6a6e9c8e1bb79ecad87d6e55f5ca9708099a25b1ed
SimHash f8e44a58cfe7

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Disallow /cd_upload/Search
Disallow /lawlib/archive/
Disallow /lawlib/briefs/

ultraseek

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Disallow /cd_upload/Search

vse/1.0

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys
Back to top

Comments

  • Disallow everything until we want to expose the site to external search
  • engines.
  • 0000-1200 GMT is 6PM to 6AM here
  • 4/14/14 Updated for DataExplorer to crawl all state sites.
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top