everything5pounds.com
robots.txt

Robots Exclusion Standard data for everything5pounds.com

Resource Scan

Scan Details

Site Domain everything5pounds.com
Base Domain everything5pounds.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-13T12:03:21+00:00
Next Scan 2024-06-27T12:03:21+00:00

Last Successful Scan

Scanned2024-05-29T12:02:40+00:00
URL https://everything5pounds.com/robots.txt
Redirect https://www.everything5pounds.com/robots.txt
Redirect Domain www.everything5pounds.com
Redirect Base everything5pounds.com
Domain IPs 104.26.12.114, 104.26.13.114, 172.67.68.168
Redirect IPs 104.45.8.69
Response IP 104.45.8.69
Found Yes
Hash 7bb4aab91f6bb2527972e6074c8e0dc6dae067497ef8eb6f66285ff5e851740e
SimHash ec55f7b6eff8

Groups

*

Rule Path
Disallow /en/cart
Disallow /en/checkout
Disallow /en/my-account
Disallow /en/*quickView
Disallow /en/search
Disallow /en/login/pw/request
Disallow */reviewhtml/*
Disallow *//*

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.everything5pounds.com/sitemap.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot