charlesclinkard.co.uk
robots.txt

Robots Exclusion Standard data for charlesclinkard.co.uk

Resource Scan

Scan Details

Site Domain charlesclinkard.co.uk
Base Domain charlesclinkard.co.uk
Scan Status Ok
Last Scan2024-11-05T08:53:45+00:00
Next Scan 2024-12-05T08:53:45+00:00

Last Scan

Scanned2024-11-05T08:53:45+00:00
URL https://charlesclinkard.co.uk/robots.txt
Redirect https://www.charlesclinkard.co.uk/robots.txt
Redirect Domain www.charlesclinkard.co.uk
Redirect Base charlesclinkard.co.uk
Domain IPs 109.108.148.102
Redirect IPs 80.66.203.178
Response IP 80.66.203.178
Found Yes
Hash 11d83bfc77c133d856b9fb3b26ddfd5a1ed3aabcad6837af0ebc535246ee62da
SimHash 720fd770ce1a

Groups

etaospider
baiduspider
brandverity
seekport crawler
seekportbot
blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
crawl-delay 60
crawl-delay 30

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pricebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

charlotte

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
crawl-delay 1

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seznambot

Rule Path
Allow /
Disallow /ajax/removefrombasket/
Disallow /basket$
Disallow /checkout/
Disallow /errors/
Disallow /quickbuy/
Disallow /product_personalisation/
Disallow /register/
Disallow /vs/
Disallow /images/products/yofla/
Disallow /customer_groups/choose_group/
Disallow /formbuilder/
Disallow /callbacks/
Disallow /ajax/formatprice/
Disallow /products/
Disallow /productimages/gallery/
Disallow /search
Disallow /wishlist/
Disallow /basket
Disallow /checkout
Disallow /compare
Disallow *?project=*
Disallow /map/?branch_select
Disallow /*?sort*
Disallow *?view_all*

Other Records

Field Value
crawl-delay 30
crawl-delay 2

amazonbot

Rule Path
Disallow /
Disallow /currency/

Other Records

Field Value
sitemap https://www.charlesclinkard.co.uk/sitemap-index.xml

Warnings

  • `request-rate` is not a known field.