esi.com
robots.txt

Robots Exclusion Standard data for esi.com

Resource Scan

Scan Details

Site Domain esi.com
Base Domain esi.com
Scan Status Ok
Last Scan2024-10-31T14:53:17+00:00
Next Scan 2024-11-30T14:53:17+00:00

Last Scan

Scanned2024-10-31T14:53:17+00:00
URL https://www.esi.com/robots.txt
Domain IPs 132.147.114.72
Response IP 138.113.53.41
Found Yes
Hash 27d12de0d12d3568f5ed11a4d8d91561e84ae3f54082dc7ed2c1b6f1f969f2a4
SimHash 2c705f16cff0

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.esi.com/medias/sys_master/root/h4e/ha6/9868467732510/Homepage-en-USD-4702171402046485663.xml
sitemap https://www.esi.com/medias/sys_master/root/hf9/h71/9868467798046/Product-en-USD-6014734303396113031.xml
sitemap https://www.esi.com/medias/sys_master/root/h5a/hf0/9868467863582/CategoryLanding-en-USD-4603789194401423349.xml
sitemap https://www.esi.com/medias/sys_master/root/h14/ha9/9868467929118/Category-en-USD-1772030223392852857.xml
sitemap https://www.esi.com/medias/sys_master/root/h45/hc2/9868468060190/Content-en-USD-4744792666891170824.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot