Log In Sign Up

dpnow.com
robots.txt

Robots Exclusion Standard data for dpnow.com

Resource Scan

Scan Details

Site Domain dpnow.com
Base Domain dpnow.com
Scan Status Ok
Last Scan2024-11-14T02:53:06+00:00
Next Scan 2024-11-21T02:53:06+00:00

Last Scan

Scanned2024-11-14T02:53:06+00:00
URL https://dpnow.com/robots.txt
Redirect https://www.dpnow.com/robots.txt
Redirect Domain www.dpnow.com
Redirect Base dpnow.com
Domain IPs 72.52.133.18
Redirect IPs 104.21.90.78, 172.67.153.253, 2606:4700:3035::6815:5a4e, 2606:4700:3037::ac43:99fd
Response IP 104.21.90.78
Found Yes
Hash 0b5529f19ecf24539c27f98bfd3b9cec93a3e8b3743c11ce4410c17acaf4b05d
SimHash 2858669afce1

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /junk/
Disallow /galleries/

Other Records

Field Value
crawl-delay 60

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

a6-indexer

Rule Path
Disallow /

archive-it

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

betabot

Rule Path
Disallow /

blablubb

Rule Path
Disallow /

blogpulselive

Rule Path
Disallow /

bpimagewalker

Rule Path
Disallow /

brands-bot

Rule Path
Disallow /

brands-bot-logo

Rule Path
Disallow /

btcrawler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buckyohare

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cms crawler

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

copyright sheriff

Rule Path
Disallow /

cowbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

cyotekwebcopy

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

deusu

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

doctorhtml

Rule Path
Disallow /

doodlebot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

eventguru

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

finbot

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

findfiles.net

Rule Path
Disallow /

findshare

Rule Path
Disallow /

followthatpage

Rule Path
Disallow /

fujilabolx2 spider

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

gipo-crawler

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

golabbot

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hexometer

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

iconsurf

Rule Path
Disallow /

idmarch

Rule Path
Disallow /

izabee

Rule Path
Disallow /

kikbot

Rule Path
Disallow /

laserlikebot

Rule Path
Disallow /

linkaider

Rule Path
Disallow /

linkfeaturebot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lipperhey spider

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

madbbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

miggibot

Rule Path
Disallow /

mixnodecache

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

moreover

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netpeakcheckerbot

Rule Path
Disallow /

newscuratorbot

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

openstat

Rule Path
Disallow /

ospscraper

Rule Path
Disallow /

p40bot

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

pdfbot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

prospectb2b

Rule Path
Disallow /

postpost

Rule Path
Disallow /

promptcloud

Rule Path
Disallow /

psbot

Rule Path
Disallow /

putin

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

repa

Rule Path
Disallow /

rsiteauditor

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

seeker

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

searchie

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

steeler

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

thumbshots-de-bot

Rule Path
Disallow /

thumbtack-thunderdome

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

twmbot

Rule Path
Disallow /

upictobot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wangid

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

web sheriff

Rule Path
Disallow /

webbot

Rule Path
Disallow /

webindex

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

wikido

Rule Path
Disallow /

willybot

Rule Path
Disallow /

wonderbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yak

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

ahrefs

Rule Path
Disallow /
Back to top

Warnings

  • 4 invalid lines.
Back to top