leaderdoors.co.uk
robots.txt

Robots Exclusion Standard data for leaderdoors.co.uk

Resource Scan

Scan Details

Site Domain leaderdoors.co.uk
Base Domain leaderdoors.co.uk
Scan Status Ok
Last Scan2024-11-04T16:53:27+00:00
Next Scan 2024-12-04T16:53:27+00:00

Last Scan

Scanned2024-11-04T16:53:27+00:00
URL https://leaderdoors.co.uk/robots.txt
Redirect https://www.leaderdoors.co.uk/robots.txt
Redirect Domain www.leaderdoors.co.uk
Redirect Base leaderdoors.co.uk
Domain IPs 109.108.148.102
Redirect IPs 80.66.203.177
Response IP 80.66.203.177
Found Yes
Hash 603398fd98cc29434feb473709556ab8aa98fc8a78a1cc1c4bc1f9243935a26f
SimHash 580bd651ca3f

Groups

etaospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

brandverity

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pricebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

charlotte

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

*

Rule Path
Allow /
Disallow /ajax/removefrombasket/
Disallow /basket$
Disallow /checkout/
Disallow /errors/
Disallow /quickbuy/
Disallow /product_personalisation/
Disallow /register/
Disallow /register
Disallow /vs/
Disallow /images/products/yofla/
Disallow /customer_groups/choose_group/
Disallow /formbuilder/
Disallow /callbacks/
Disallow /products/
Disallow /rss/
Disallow *?gclid=*
Disallow *//*
Disallow /search/
Disallow *?cid=*
Disallow *-t1*
Disallow *-t3*
Disallow *-t4*
Disallow *-t9*
Allow /doors-c14/internal-doors-c111/internal-oak-doors-c119/unfinished-t12685
Allow /doors-c14/internal-doors-c111/unfinished-t12685
Allow /weather-bars-t30061
Allow /doors-c14/internal-doors-c111/obscure-glass-t49
Allow /doors-c14/internal-doors-c111/clear-glass-t41
Allow /doors-c14/external-doors-c18/clear-glass-t41
Allow /doors-c14/internal-doors-c111/fully-finished-t9297
Allow /doors-c14/internal-doors-c111/internal-oak-doors-c119/fully-finished-t9297
Allow /doors-c14/door-hardware-c96/black-nickel-t9318

Other Records

Field Value
sitemap https://www.leaderdoors.co.uk/sitemap-index.xml
sitemap https://www.leaderdoors.co.uk/blog/sitemap.xml

Warnings

  • 2 invalid lines.
  • `request-rate` is not a known field.