business.landsend.com
robots.txt

Robots Exclusion Standard data for business.landsend.com

Resource Scan

Scan Details

Site Domain business.landsend.com
Base Domain landsend.com
Scan Status Ok
Last Scan2024-05-29T04:35:46+00:00
Next Scan 2024-06-28T04:35:46+00:00

Last Scan

Scanned2024-05-29T04:35:46+00:00
URL https://business.landsend.com/robots.txt
Domain IPs 23.203.76.28
Response IP 59.151.131.201
Found Yes
Hash 69ff7f2e45d416e19ef01c22be6182d1b974ff05de8039b7c8143409c1689be5
SimHash 180fe636ef70

Groups

*

Rule Path
Disallow /cart/
Disallow /login/
Disallow /my-account/
Disallow /orderTrack/
Disallow /store/
Allow /shop-custom-logo-uniforms

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /cart/
Disallow /login/
Disallow /my-account/
Disallow /orderTrack/
Disallow /store/

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /shop-custom-logo-uniforms
Disallow /cart/
Disallow /login/
Disallow /my-account/
Disallow /orderTrack/
Disallow /store/

Other Records

Field Value
crawl-delay 1

bingbot/2.0

Rule Path
Disallow /cart/
Disallow /login/
Disallow /my-account/
Disallow /orderTrack/
Disallow /store/

Other Records

Field Value
crawl-delay 1

yandexbot/3.0

Rule Path
Disallow /

yandex

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://business.landsend.com/sitemap.xml
sitemap https://business.landsend.com/shop-custom-logo-uniforms/sitemap.xml

Comments

  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • throttle MS bots
  • blocking Yandex as they repeatedly hit dead URLs
  • blocking Rogerbot
  • blocking SemrushBot
  • blocking Commoncrawl