Log In Sign Up

kingstonmemoryshop.co.uk
robots.txt

Robots Exclusion Standard data for kingstonmemoryshop.co.uk

Resource Scan

Scan Details

Site Domain kingstonmemoryshop.co.uk
Base Domain kingstonmemoryshop.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-24T23:30:41+00:00
Next Scan 2026-04-24T23:30:41+00:00

Last Successful Scan

Scanned2023-09-14T00:27:11+00:00
URL https://www.kingstonmemoryshop.co.uk/robots.txt
Domain IPs 104.26.10.91, 104.26.11.91, 172.67.74.112, 2606:4700:20::681a:a5b, 2606:4700:20::681a:b5b, 2606:4700:20::ac43:4a70
Response IP 104.26.10.91
Found Yes
Hash d3f5e76fb00c2d8406cf039972051185929414709b060f6bdfe3518e0e97b417
SimHash 5254d442c22a

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mojeekbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

baiduspider
baiduspider-video
baiduspider-image
baiduspider+

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

linespider

Rule Path
Disallow /

cincraw

Rule Path
Disallow /
Back to top

Warnings

  • 2 invalid lines.
Back to top