Log In Sign Up

mwcomponents.com
robots.txt

Robots Exclusion Standard data for mwcomponents.com

Resource Scan

Scan Details

Site Domain mwcomponents.com
Base Domain mwcomponents.com
Scan Status Ok
Last Scan2024-11-09T12:40:33+00:00
Next Scan 2024-11-16T12:40:33+00:00

Last Scan

Scanned2024-11-09T12:40:33+00:00
URL https://mwcomponents.com/robots.txt
Redirect https://www.mwcomponents.com/robots.txt
Redirect Domain www.mwcomponents.com
Redirect Base mwcomponents.com
Domain IPs 70.39.232.80
Redirect IPs 18.155.68.117, 18.155.68.124, 18.155.68.30, 18.155.68.94, 2600:9000:23d2:1000:3:6b59:e80:93a1, 2600:9000:23d2:2e00:3:6b59:e80:93a1, 2600:9000:23d2:3c00:3:6b59:e80:93a1, 2600:9000:23d2:600:3:6b59:e80:93a1, 2600:9000:23d2:8e00:3:6b59:e80:93a1, 2600:9000:23d2:a800:3:6b59:e80:93a1, 2600:9000:23d2:b600:3:6b59:e80:93a1, 2600:9000:23d2:ca00:3:6b59:e80:93a1
Response IP 18.155.68.94
Found Yes
Hash 5e7f944892920e374d1c0245c22957d77517f588b1e5eca15be8b80875212de2
SimHash 41681a122532

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /parts-kit/
Disallow /quick-view*
Disallow */autocomplete?*
Disallow /*microsite%3D*
Disallow /*location%3D*
Disallow /*.php*
Disallow /*referrer%3D*
Disallow /*render-template?template=*
Disallow /site.webmanifest
Disallow /*wlwmanifest.xml*
Back to top

Other Records

Field Value
sitemap https://www.mwcomponents.com/sitemaps-1-sitemap.xml
Back to top

Comments

  • robots.txt for https://www.mwcomponents.com/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow: /shop/*/specs*
Back to top