mycnhstore.com
robots.txt

Robots Exclusion Standard data for mycnhstore.com

Resource Scan

Scan Details

Site Domain mycnhstore.com
Base Domain mycnhstore.com
Scan Status Ok
Last Scan2024-09-11T00:40:02+00:00
Next Scan 2024-10-11T00:40:02+00:00

Last Scan

Scanned2024-09-11T00:40:02+00:00
URL https://mycnhstore.com/robots.txt
Redirect https://www.mycnhstore.com/robots.txt
Redirect Domain www.mycnhstore.com
Redirect Base mycnhstore.com
Domain IPs 159.61.80.175
Redirect IPs 2600:1413:b000:1b::17d7:718, 2600:1413:b000:1b::17d7:71d, 96.17.180.16
Response IP 96.17.180.16
Found Yes
Hash d7c85df7d66fed68bfbae2f8499a6aa9d5a4d10c97fb6c6edb9417f940505e7a
SimHash 18537f77adb3

Groups

*

Rule Path
Disallow */cart$
Disallow */checkout/
Disallow /punchout/
Disallow /default
Disallow */personalarea/
Disallow */saml/login*
Disallow /login/
Disallow /DSP
Disallow /CASA
Disallow /CASB
Disallow /CASF
Disallow /CB
Disallow /CE
Disallow /CF
Disallow /kobelco
Disallow /kongskilde
Disallow /overum
Disallow /anz_row*
Disallow /amea-row*
Disallow /eu_row*
Disallow /sa_row*
Disallow */search/
Disallow */search$
Disallow /*assemblyFileReport?assemblyPath=
Disallow */default/
Disallow /*?brandCode=
Disallow /*?dealerCode=
Disallow /*?ownershipCode=
Disallow /bg/ru/
Disallow /bg/bg/
Disallow /dk/da/
Disallow /dk/ru/
Disallow /dk/it/
Disallow /ro/ro/
Disallow /ro/da/
Disallow /se/da/
Disallow /se/sv/
Disallow /us/da/
Disallow /us/de/
Disallow /us/it/
Disallow /us/nl/
Disallow /us/pl/
Disallow /us/pt/
Disallow /us/ru/
Disallow /us/tr/

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.mycnhistore.com/sitemap.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Disallow: /*?page=
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot