Log In Sign Up

dpnow.co.uk
robots.txt

Robots Exclusion Standard data for dpnow.co.uk

Resource Scan

Scan Details

Site Domain dpnow.co.uk
Base Domain dpnow.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-08T15:40:21+00:00
Next Scan 2024-07-07T15:40:21+00:00

Last Successful Scan

Scanned2021-12-16T23:55:05+00:00
URL https://dpnow.co.uk/robots.txt
Response IP 72.52.133.18
Found Yes
Hash 0b5529f19ecf24539c27f98bfd3b9cec93a3e8b3743c11ce4410c17acaf4b05d
SimHash 2858669afce1

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /junk/
Disallow /galleries/

Other Records

Field Value
crawl-delay 60

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

a6-indexer

Rule Path
Disallow /

archive-it

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

betabot

Rule Path
Disallow /

blablubb

Rule Path
Disallow /

blogpulselive

Rule Path
Disallow /

bpimagewalker

Rule Path
Disallow /

brands-bot

Rule Path
Disallow /

brands-bot-logo

Rule Path
Disallow /

btcrawler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buckyohare

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cms crawler

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

copyright sheriff

Rule Path
Disallow /

cowbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

cyotekwebcopy

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

deusu

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

doctorhtml

Rule Path
Disallow /

doodlebot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

eventguru

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

finbot

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

findfiles.net

Rule Path
Disallow /

findshare

Rule Path
Disallow /

followthatpage

Rule Path
Disallow /

fujilabolx2 spider

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

gipo-crawler

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

golabbot

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hexometer

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

iconsurf

Rule Path
Disallow /

idmarch

Rule Path
Disallow /

izabee

Rule Path
Disallow /

kikbot

Rule Path
Disallow /

laserlikebot

Rule Path
Disallow /

linkaider

Rule Path
Disallow /

linkfeaturebot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lipperhey spider

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

madbbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

miggibot

Rule Path
Disallow /

mixnodecache

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

moreover

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netpeakcheckerbot

Rule Path
Disallow /

newscuratorbot

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

openstat

Rule Path
Disallow /

ospscraper

Rule Path
Disallow /

p40bot

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

pdfbot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

prospectb2b

Rule Path
Disallow /

postpost

Rule Path
Disallow /

promptcloud

Rule Path
Disallow /

psbot

Rule Path
Disallow /

putin

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

repa

Rule Path
Disallow /

rsiteauditor

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

seeker

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

searchie

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

steeler

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

thumbshots-de-bot

Rule Path
Disallow /

thumbtack-thunderdome

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

twmbot

Rule Path
Disallow /

upictobot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wangid

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

web sheriff

Rule Path
Disallow /

webbot

Rule Path
Disallow /

webindex

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

wikido

Rule Path
Disallow /

willybot

Rule Path
Disallow /

wonderbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yak

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

ahrefs

Rule Path
Disallow /
Back to top

Warnings

  • 4 invalid lines.
Back to top