Log In Sign Up

mwcomponents.com
robots.txt

Robots Exclusion Standard data for mwcomponents.com

Resource Scan

Scan Details

Site Domain mwcomponents.com
Base Domain mwcomponents.com
Scan Status Ok
Last Scan2024-09-21T10:22:51+00:00
Next Scan 2024-09-28T10:22:51+00:00

Last Scan

Scanned2024-09-21T10:22:51+00:00
URL https://mwcomponents.com/robots.txt
Redirect https://www.mwcomponents.com/robots.txt
Redirect Domain www.mwcomponents.com
Redirect Base mwcomponents.com
Domain IPs 70.39.232.80
Redirect IPs 18.155.68.117, 18.155.68.124, 18.155.68.30, 18.155.68.94, 2600:9000:23d2:2c00:3:6b59:e80:93a1, 2600:9000:23d2:8c00:3:6b59:e80:93a1, 2600:9000:23d2:9600:3:6b59:e80:93a1, 2600:9000:23d2:a800:3:6b59:e80:93a1, 2600:9000:23d2:b400:3:6b59:e80:93a1, 2600:9000:23d2:bc00:3:6b59:e80:93a1, 2600:9000:23d2:be00:3:6b59:e80:93a1, 2600:9000:23d2:e800:3:6b59:e80:93a1
Response IP 18.155.68.117
Found Yes
Hash 2728730ea0236380e89d3ba77260481e3ca898f97884fda0c2960d02202ee970
SimHash 41681a122532

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /parts-kit/
Disallow */autocomplete?*
Disallow /*microsite%3D*
Disallow /*location%3D*
Disallow /*.php*
Disallow /*referrer%3D*
Disallow /*render-template?template=*
Disallow /site.webmanifest
Disallow /*wlwmanifest.xml*
Back to top

Other Records

Field Value
sitemap https://www.mwcomponents.com/sitemaps-1-sitemap.xml
Back to top

Comments

  • robots.txt for https://www.mwcomponents.com/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow: /shop/*/specs*
Back to top