dkeng.co.uk
robots.txt

Robots Exclusion Standard data for dkeng.co.uk

Resource Scan

Scan Details

Site Domain dkeng.co.uk
Base Domain dkeng.co.uk
Scan Status Ok
Last Scan2024-09-25T15:07:03+00:00
Next Scan 2024-10-25T15:07:03+00:00

Last Scan

Scanned2024-09-25T15:07:03+00:00
URL https://dkeng.co.uk/robots.txt
Redirect https://www.dkeng.co.uk/robots.txt
Redirect Domain www.dkeng.co.uk
Redirect Base dkeng.co.uk
Domain IPs 91.90.121.245
Redirect IPs 91.90.121.245
Response IP 91.90.121.245
Found Yes
Hash 928443dae67bd03cc7bc1b269ef98fad824d943576bccda37fcc7d12c361e9e7
SimHash d030a02a0fc2

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /s/
Disallow /admin/
Disallow /admin
Disallow /cpanel
Disallow /cpanel/
Disallow /cms
Disallow /cms/

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

indy library

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

softsearch.ru robot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-httplib2

Rule Path
Disallow /

expo9

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

eventmachine httpclient

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

nutch

Rule Path
Disallow /

maxpointcrawler/nutch-1.10

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

go 1.1 package http

Rule Path
Disallow /

wget

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

uipbot/1.0 (uipbot@semasio.net)

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

free website submitter/1.0

Rule Path
Disallow /

free website submitter

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /