Log In Sign Up

logiscenter.co.uk
robots.txt

Robots Exclusion Standard data for logiscenter.co.uk

Resource Scan

Scan Details

Site Domain logiscenter.co.uk
Base Domain logiscenter.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-11T10:37:40+00:00
Next Scan 2024-06-10T10:37:40+00:00

Last Successful Scan

Scanned2024-03-20T10:02:04+00:00
URL https://logiscenter.co.uk/robots.txt
Redirect https://www.logiscenter.co.uk/robots.txt
Redirect Domain www.logiscenter.co.uk
Redirect Base logiscenter.co.uk
Domain IPs 104.26.12.164, 104.26.13.164, 172.67.71.101, 2606:4700:20::681a:ca4, 2606:4700:20::681a:da4, 2606:4700:20::ac43:4765
Redirect IPs 104.26.12.164, 104.26.13.164, 172.67.71.101, 2606:4700:20::681a:ca4, 2606:4700:20::681a:da4, 2606:4700:20::ac43:4765
Response IP 104.26.12.164
Found Yes
Hash fe9035689f7e6e631db6a0f0a66437da2a002d2f3b0ab31cb4f00e1d0408b0c8
SimHash 2be4ef1345f1

Groups

*

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /*.php$
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /fpc/
Disallow /report/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?manufacturer*
Disallow /*?categoria*
Disallow /*?subcategoria*
Disallow /*?product*
Disallow /*?sku*
Disallow /index.php/
Disallow /*?SID=
Disallow /*SID%3D
Disallow /checkout/*
Disallow /customer/*
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalogsearch/*
Disallow /catalog/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/category/view/*
Disallow /catalog/product/view/
Disallow /catalog/product/view/*
Disallow /catalog/product_compare/
Disallow /catalogsearch/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /add/
Disallow /ajax/
Disallow /form_key/
Disallow /ajaxcart/index/options/product_id
Disallow /ajaxcart/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /checkout/*
Disallow /customer/*
Disallow /catalogsearch/*

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

arachni/v1.1

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

mozilla/5.0 (compatible; fatbot 2.0; http://www.thefind.com/crawler)

Rule Path
Disallow /

ahrefsbot/5.0

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

redtestbot/1.0

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Allow /*.js$
Allow /*.css$
Back to top

Other Records

Field Value
sitemap https://www.logiscenter.co.uk/media/sitemap_uk.xml
Back to top

Comments

  • This file contains optimizations for Magento 1
  • This also attempts to slow Google and other legitimate bots from being too aggressive.
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • Paths (clean URLs)
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot
  • Disallow: /
  • User-agent: Googlebot-image
  • Disallow: /
  • User-agent: Googlebot-mobile
  • Disallow: /
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Disallow other bots
  • Allowing assets
Back to top