find-dvd.co.uk
robots.txt

Robots Exclusion Standard data for find-dvd.co.uk

Resource Scan

Scan Details

Site Domain find-dvd.co.uk
Base Domain find-dvd.co.uk
Scan Status Ok
Last Scan2024-10-03T20:45:45+00:00
Next Scan 2024-10-10T20:45:45+00:00

Last Scan

Scanned2024-10-03T20:45:45+00:00
URL https://find-dvd.co.uk/robots.txt
Domain IPs 213.165.92.246
Response IP 213.165.92.246
Found Yes
Hash c4671d0b38ae35d435a6ad77859fb17957d67f8678c1a85e480b6a34c440d9a7
SimHash aa0c42c2deb9

Groups

*

Rule Path
Disallow /rt.asp
Disallow /price.aspx
Disallow /price/
Disallow /r1/
Disallow /iphone/
Disallow /fp2.asp
Disallow /vouchers.aspx
Disallow /trailer.asp
Disallow /noscript/
Disallow /addreview.aspx
Disallow /ChartImg.axd
Disallow /priceHistory.aspx

semrushbot-sa

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bubing

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

woorankreview

Rule Path
Disallow /

clarabot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

bleriot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

teoma

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

exabot-images

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

linguee

Rule Path
Disallow /