Log In Sign Up

lsengineers.co.uk
robots.txt

Robots Exclusion Standard data for lsengineers.co.uk

Resource Scan

Scan Details

Site Domain lsengineers.co.uk
Base Domain lsengineers.co.uk
Scan Status Ok
Last Scan2025-03-20T04:11:19+00:00
Next Scan 2025-04-19T04:11:19+00:00

Last Scan

Scanned2025-03-20T04:11:19+00:00
URL https://lsengineers.co.uk/robots.txt
Redirect https://www.lsengineers.co.uk/robots.txt
Redirect Domain www.lsengineers.co.uk
Redirect Base lsengineers.co.uk
Domain IPs 104.26.14.63, 104.26.15.63, 172.67.70.133, 2606:4700:20::681a:e3f, 2606:4700:20::681a:f3f, 2606:4700:20::ac43:4685
Redirect IPs 104.26.14.63, 104.26.15.63, 172.67.70.133, 2606:4700:20::681a:e3f, 2606:4700:20::681a:f3f, 2606:4700:20::ac43:4685
Response IP 104.26.14.63
Found Yes
Hash 70f52dff952686240ee0c1653be2714b6f55bf8dc2ff7ea178f292ea5aac1969
SimHash 2724f5460d22

Groups

*

Rule Path
Disallow /index.php/
Disallow /wishlist/
Disallow /admin/
Disallow /review/product/
Disallow /sendfriend/
Disallow /sendfriend/product/
Disallow /enable-cookies/
Disallow /directory/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /customer/account/create/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?p=*
Disallow /*?design=*
Disallow /*?mode*
Disallow /*brand%3D*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /*?SID=
Back to top

Other Records

Field Value
sitemap https://www.lsengineers.co.uk/pub/media/sitemap.xml
Back to top

Comments

  • Enable robots.txt rules for all crawlers
  • Do not crawl common Magento folders
  • Do not crawl common Magento files
  • Do not crawl checkout and user account pages
  • Do not crawl search pages and non SEO optimized catalog links
  • Blocking filter options, parameters and pagination
  • Disallow: /*?limit=all
  • Blocking CMS directories
  • Do not crawl common server technical folders and files
  • Do not crawl links with session IDs
Back to top