dlx.co.uk
robots.txt

Robots Exclusion Standard data for dlx.co.uk

Resource Scan

Scan Details

Site Domain dlx.co.uk
Base Domain dlx.co.uk
Scan Status Ok
Last Scan2025-11-11T03:49:17+00:00
Next Scan 2025-12-11T03:49:17+00:00

Last Scan

Scanned2025-11-11T03:49:17+00:00
URL https://dlx.co.uk/robots.txt
Redirect https://www.dlx.co.uk/robots.txt
Redirect Domain www.dlx.co.uk
Redirect Base dlx.co.uk
Domain IPs 45.60.124.183, 45.60.130.183
Redirect IPs 45.60.126.183
Response IP 45.60.126.183
Found Yes
Hash afcc0a6a9e6e8fb7991fb8eca9eb86bf17751d9af453742e6bfecdccb6d303fa
SimHash 01c84852c8b5

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /directory/
Disallow /downloader/
Disallow /install/
Disallow /js/
Disallow /lib/
Disallow /phpserver/
Disallow /pkginfo/
Disallow /private/
Disallow /setup/
Disallow /skin/
Disallow /update/
Disallow /var/
Disallow /vendor/
Disallow /admin/
Disallow */api/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */catalogsearch
Disallow */checkout
Disallow */customer/
Disallow */newsletter
Disallow */order/
Disallow */poll/
Disallow */report/
Disallow */review/
Disallow */rma/
Disallow */sendfriend
Disallow */wishlist
Disallow /cron
Disallow /cron.php
Disallow */cat/*
Disallow /*/pricematch/*
Disallow /*/colour/*
Disallow /*/show/*
Disallow /*/size/*
Disallow /*/price/*
Disallow */sort-by/*
Disallow /*?*product_list_dir=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*.php
Disallow /*.sh$
Disallow /*.CSV$
Disallow /*.csv$
Disallow /*.gitignore$
Disallow /*.sample$
Disallow /*.sql$
Disallow /*.zip$
Disallow /*%26carrier%3D*
Disallow /*%26cat%3D*
Disallow /*%26color_filter%3D*
Disallow /*%26display_size%3D*
Disallow /*%26name%3D*
Disallow /*%26network%3D*
Disallow /*%26os%3D*
Disallow /*%26price%3D*
Disallow /*%26timestamp%3D*
Disallow /*%26top_seller%3D*
Disallow /*?SID=*
Disallow /*?___SID=*
Disallow /*?___store=*
Disallow /*?carrier=*
Disallow /*?cat=*
Disallow /*?color_filter=*
Disallow /*?display_size=*
Disallow /*?name=*
Disallow /*?network=*
Disallow /*?os=*
Disallow /*?PageSpeed=noscript
Disallow /*?price=*
Disallow /*?timestamp=*
Disallow /*?top_seller=*
Disallow /*?size_legacy=*
Disallow /*?base_color=*
Disallow /*?manufacturer=*
Disallow /*?insulation_type=*
Disallow /*?gender=*
Disallow /*?p=2&size_legacy=*
Disallow *?size_legacy=*
Disallow *?base_color=*
Disallow *?manufacturer=*
Disallow *?insulation_type=*
Disallow *?gender=*
Disallow *?p=2&size_legacy=*
Disallow /*/new_product_type/*
Disallow /*/product_subtype/*
Disallow /*?category_id=*
Disallow /*?adnetwork=*
Disallow /%26adnetwork%3D
Disallow /?affc=
Disallow /%26affc%3D
Disallow /*?cto_pld=*
Disallow /%26cto_pld%3D
Disallow /*?ved=*
Disallow /*?fbclid=*
Disallow /*?search=sf-*
Allow /*?p=

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

nutch

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-mobile

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

botify

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

idealo-bot

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.dlx.co.uk/sitemap.xml

Comments

  • Crawlers Setup
  • Directories (platform/system)
  • Paths (clean URLs)
  • Files
  • Do not index pages that are sorted or filtered.
  • Clean URLs - Suffixes
  • Disallow: /*?
  • CVS, SVN directory and dump files
  • Parameters (layered nav facets & misc)
  • Additional DLX layered-nav params commonly used
  • Block filters for product categories
  • Tracking/affiliate & misc
  • Allow pagination
  • Crawl-Delay (not honoured by Google; fine for others)
  • Bad bots

Warnings

  • 2 invalid lines.