Log In Sign Up

borderstates.com
robots.txt

Robots Exclusion Standard data for borderstates.com

Resource Scan

Scan Details

Site Domain borderstates.com
Base Domain borderstates.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-13T06:16:03+00:00
Next Scan 2026-01-11T06:16:03+00:00

Last Successful Scan

Scanned2023-12-01T04:18:22+00:00
URL https://borderstates.com/robots.txt
Redirect https://www.borderstates.com/robots.txt
Redirect Domain www.borderstates.com
Redirect Base borderstates.com
Domain IPs 66.22.25.65
Redirect IPs 66.22.25.65
Response IP 66.22.25.65
Found Yes
Hash 2eb59f53a49cc348f09a184ca153b75d1fdde48ff47d4d08d778976c36b62523
SimHash 6c745f1ccde0

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top