shoppersstop.com
robots.txt

Robots Exclusion Standard data for shoppersstop.com

Resource Scan

Scan Details

Site Domain shoppersstop.com
Base Domain shoppersstop.com
Scan Status Ok
Last Scan2024-05-24T10:00:55+00:00
Next Scan 2024-06-07T10:00:55+00:00

Last Scan

Scanned2024-05-24T10:00:55+00:00
URL https://shoppersstop.com/robots.txt
Redirect https://www.shoppersstop.com/robots.txt
Redirect Domain www.shoppersstop.com
Redirect Base shoppersstop.com
Domain IPs 151.101.130.137, 151.101.194.137, 151.101.2.137, 151.101.66.137
Redirect IPs 151.101.130.137, 151.101.194.137, 151.101.2.137, 151.101.66.137, 2a04:4e42:200::649, 2a04:4e42:400::649, 2a04:4e42:600::649, 2a04:4e42::649
Response IP 199.232.46.137
Found Yes
Hash 27e47e1db52356aefed8ac343355c63d410137966f9c7e94f2bed7e21bad6817
SimHash 08d32f544fb2

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account
Allow /cartier*
Disallow /search
Disallow /en/search
Disallow /*?showType
Disallow /*?categoryCode
Disallow /*?sort
Disallow /*?pagetype
Disallow /*?q
Allow /*?PageSpeed
Disallow /*?currentPosition
Disallow /*?text&showType&sort
Disallow /ssladmin/
Disallow /sslomm/
Disallow /%26s2%3Dym
Disallow /catalogservlet?categoryid=
Disallow /checkout/multi/
Disallow /crm.
Disallow /r.srvtrck.com/v1/
Disallow /wishlist/addorremovewishlist/

cazoodlebot

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Allow /

gigabot

Rule Path
Disallow /

etaospider

Rule Path
Allow /

copyrightcheck

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

true_robot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

yandex

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.shoppersstop.com/sitemap.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block EtaoSpider as ecommerence engine
  • Unsafe robots to disallow