Log In Sign Up

dreamsleep.net
robots.txt

Robots Exclusion Standard data for dreamsleep.net

Resource Scan

Scan Details

Site Domain dreamsleep.net
Base Domain dreamsleep.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-06-05T22:55:23+00:00
Next Scan 2024-07-05T22:55:23+00:00

Last Successful Scan

Scanned2024-04-14T20:48:15+00:00
URL http://dreamsleep.net/robots.txt
Redirect https://psychics.co.uk/robots.txt
Redirect Domain psychics.co.uk
Redirect Base psychics.co.uk
Domain IPs 217.199.187.63
Redirect IPs 54.216.252.255
Response IP 54.216.252.255
Found Yes
Hash bd3f7eb0c3a8abbebc93527c30f7821b040996b7ac5a42f80b540eaacf66ef70
SimHash 67416f726a8b

Groups

*

Rule Path
Disallow /private/
Disallow /junk/
Disallow 67ea8885ca9449eaa8d469c4597ca77c.txt
Allow /static/

a6-indexer

Rule Path
Disallow /

admantx

Rule Path
Disallow /

aibot

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

abonti

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

acunetix

Rule Path
Disallow /

afd-verbotsverfahren

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

allsubmitter

Rule Path
Disallow /

alligator

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

anarchie

Rule Path
Disallow /

apexoo

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

asterias

Rule Path
Disallow /

attach

Rule Path
Disallow /

bbbike

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backstreet

Rule Path
Disallow /

backweb

Rule Path
Disallow /

backlink-ceck

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

badass

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

bandit

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

batchftp

Rule Path
Disallow /

battleztar bazinga

Rule Path
Disallow /

battleztarbazinga

Rule Path
Disallow /

betabot

Rule Path
Disallow /

bigfoot

Rule Path
Disallow /

bitacle

Rule Path
Disallow /

blackhole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blackboard

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blow

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

bolt

Rule Path
Disallow /

botalot

Rule Path
Disallow /

brandprotect

Rule Path
Disallow /

brandwatch

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buddy

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

buzzsumo

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

catexplorador

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cshttp

Rule Path
Disallow /

calculon

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

chlooe

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cloudmapping

Rule Path
Disallow /

cogentbot

Rule Path
Disallow /

collector

Rule Path
Disallow /

copier

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

copyscape

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

craftbot

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

crescent

Rule Path
Disallow /

curious

Rule Path
Disallow /

custo

Rule Path
Disallow /

dblbot

Rule Path
Disallow /

diibot

Rule Path
Disallow /

dtsagent

Rule Path
Disallow /

datacha0s

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

databasedrivermysqli

Rule Path
Disallow /

demon

Rule Path
Disallow /

deusu

Rule Path
Disallow /

devil

Rule Path
Disallow /

digincore

Rule Path
Disallow /

digitalpebble

Rule Path
Disallow /

dirbuster

Rule Path
Disallow /

disco

Rule Path
Disallow /

discobot

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dnyzbot

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

domainsigmacrawler

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot/1.2

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

downloadwonder

Rule Path
Disallow /

dragonfly

Rule Path
Disallow /

drip

Rule Path
Disallow /

eccp/1.0

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

easydl

Rule Path
Disallow /

ebingbong

Rule Path
Disallow /

ecxi

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

evil

Rule Path
Disallow /

exabot

Rule Path
Disallow /

expresswebpictures

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

extractor

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

extremepicturefinder

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fdm

Rule Path
Disallow /

fhscan

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

fimap

Rule Path
Disallow /

firefox/7.0

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flunky

Rule Path
Disallow /

foobot

Rule Path
Disallow /

freeuploader

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

fyrebot

Rule Path
Disallow /

g-i-g-a-b-o-t

Rule Path
Disallow /

gt

Rule Path
Disallow /

galaxybot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

germcrawler

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb

Rule Path
Disallow /

getintent

Rule Path
Disallow /

gigablast

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

gozilla

Rule Path
Disallow /

gotit

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grafula

Rule Path
Disallow /

grapefx

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

gridbot

Rule Path
Disallow /

headmasterseo

Rule Path
Disallow /

hmview

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

http

Rule Path
Disallow /

httrack

Rule Path
Disallow /

haansoft

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

harvest

Rule Path
Disallow /

havij

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hloader

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

idbot

Rule Path
Disallow /

idg/it (http://spaziodati.eu/)

Rule Path
Disallow /

idg/it(http

Rule Path
Disallow /

idg/uk(http

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

iblog

Rule Path
Disallow /

id-search

Rule Path
Disallow /

ilsebot

Rule Path
Disallow /

imagefetch

Rule Path
Disallow /

imagesucker

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

indylibrary

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

infotekies

Rule Path
Disallow /

intelliseek

Rule Path
Disallow /

interget

Rule Path
Disallow /

internetninja

Rule Path
Disallow /

internetseer

Rule Path
Disallow /

iria

Rule Path
Disallow /

iskanie

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

jocwebspider

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

jbrofuzz

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

joomla

Rule Path
Disallow /

jorgee

Rule Path
Disallow /

justview

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

kenjinspider

Rule Path
Disallow /

keyworddensity

Rule Path
Disallow /

kozmosbot

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

lwp

Rule Path
Disallow /

lanshanbot

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

leechget

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lftp

Rule Path
Disallow /

libweb

Rule Path
Disallow /

libwhisker

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

lightspeedsystems

Rule Path
Disallow /

likse

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

linksmanager

Rule Path
Disallow /

linqiametadatadownloaderbot

Rule Path
Disallow /

linqiarssbot

Rule Path
Disallow /

linqiascrapebot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

litemage_walker

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

midowntool

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msfrontpage

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

mswebservicesclientprotocol

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

magnet

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

markmonitor

Rule Path
Disallow /

markwatch

Rule Path
Disallow /

massdownloader

Rule Path
Disallow /

masscan

Rule Path
Disallow /

matahari

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

metauri

Rule Path
Disallow /

microsoftdataaccess

Rule Path
Disallow /

microsofturlcontrol

Rule Path
Disallow /

misterpix

Rule Path
Disallow /

mojeek

Rule Path
Disallow /

morfeusfuckingscanner

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

mr.4x3

Rule Path
Disallow /

msrabot

Rule Path
Disallow /

musobot

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nameintelligence

Rule Path
Disallow /

nameprotect

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

needle

Rule Path
Disallow /

nessus

Rule Path
Disallow /

netants

Rule Path
Disallow /

netlyzer

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netvampire

Rule Path
Disallow /

netzip

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

nettrack

Rule Path
Disallow /

netvibes

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

nibbler

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

nikto

Rule Path
Disallow /

nimblecrawler

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

ninja

Rule Path
Disallow /

nmap

Rule Path
Disallow /

nutch

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offlineexplorer

Rule Path
Disallow /

offlinenavigator

Rule Path
Disallow /

openlinkprofiler

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openvas

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

orangespider

Rule Path
Disallow /

outclicksbot

Rule Path
Disallow /

outfoxbot

Rule Path
Disallow /

pecl

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

poe-component-client-http

Rule Path
Disallow /

pageanalyzer

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

pagescorer

Rule Path
Disallow /

panscient

Rule Path
Disallow /

papafoto

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

peoplepal

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

pi-monster

Rule Path
Disallow /

picscout

Rule Path
Disallow /

picsearch

Rule Path
Disallow /

picturefinder

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

pixray

Rule Path
Disallow /

pleasecrawl

Rule Path
Disallow /

pockey

Rule Path
Disallow /

propowerbot

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

probethenet

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pump

Rule Path
Disallow /

pxbroker

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

querynmetasearch

Rule Path
Disallow /

quick-crawler

Rule Path
Disallow /

qwant-news

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

rankactive

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

rankflex

Rule Path
Disallow /

rankingbot

Rule Path
Disallow /

rankingbot2

Rule Path
Disallow /

rankivabot

Rule Path
Disallow /

rankurbot

Rule Path
Disallow /

reget

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reaper

Rule Path
Disallow /

rebelmouse

Rule Path
Disallow /

recorder

Rule Path
Disallow /

redesscrapy

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

ripper

Rule Path
Disallow /

rocketcrawler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sbider

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seolyticscrawler

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

seostats

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

scanalert

Rule Path
Disallow /

scanbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

searchestate

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrush

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seomoz

Rule Path
Disallow /

shodan

Rule Path
Disallow /

siphon

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

sitebeam

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

sitevigil

Rule Path
Disallow /

slackbot-linkexpanding

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

snake

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

snoopy

Rule Path
Disallow /

socialrankiobot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogouwebspider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sottopop

Rule Path
Disallow /

spacebison

Rule Path
Disallow /

spammen

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

sqlworm

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

steeler

Rule Path
Disallow /

stripper

Rule Path
Disallow /

sucker

Rule Path
Disallow /

sucuri

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

t0phackteam

Rule Path
Disallow /

t8abot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

telesphoreo

Rule Path
Disallow /

telesphorep

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

theintraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

titan

Rule Path
Disallow /

toata

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

trendiction

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twice

Rule Path
Disallow /

typhoeus

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

urly.warning

Rule Path
Disallow /

urlywarning

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

vbproject

Rule Path
Disallow /

vci

Rule Path
Disallow /

vacuum

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

vericitecrawler

Rule Path
Disallow /

vidiblescraper

Rule Path
Disallow /

virusdie

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voil

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wasalive-bot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

webdav

Rule Path
Disallow /

wisenutbot

Rule Path
Disallow /

wpscan

Rule Path
Disallow /

www

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

wallpapers/3.0

Rule Path
Disallow /

wallpapershd

Rule Path
Disallow /

wesee

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webfuck

Rule Path
Disallow /

webgois

Rule Path
Disallow /

webimagecollector

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webpix

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

websucker

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webalta

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webshag

Rule Path
Disallow /

websiteextractor

Rule Path
Disallow /

websitequester

Rule Path
Disallow /

webster

Rule Path
Disallow /

whack

Rule Path
Disallow /

whacker

Rule Path
Disallow /

whatweb

Rule Path
Disallow /

who.isbot

Rule Path
Disallow /

widow

Rule Path
Disallow /

winhttrack

Rule Path
Disallow /

wiseguysrobot

Rule Path
Disallow /

wonderbot

Rule Path
Disallow /

woobot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wprecon

Rule Path
Disallow /

xaldonwebspider

Rule Path
Disallow /

xaldon_webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

zade

Rule Path
Disallow /

zauba

Rule Path
Disallow /

zermelo

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zitebot

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

backlink-check

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

com.plumanalytics

Rule Path
Disallow /

crawler.feedback

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crawler_eb_germany_2.0

Rule Path
Disallow /

demandbase-bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

evc-batch

Rule Path
Disallow /

instabid

Rule Path
Disallow /

internetvistamonitor

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

lwp-request

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

muhstik-scan

Rule Path
Disallow /

netestatenecrawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

pagescorer

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

plumanalytics

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scan.lol

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

sexsearcher

Rule Path
Disallow /

sogouspider

Rule Path
Disallow /

spider@seoscanners.net

Rule Path
Disallow /

sysscan

Rule Path
Disallow /

takeout

Rule Path
Disallow /

trendiction.com

Rule Path
Disallow /

trendiction.de

Rule Path
Disallow /

webmeup-crawler

Rule Path
Disallow /

x09mozilla

Rule Path
Disallow /

x22mozilla

Rule Path
Disallow /

xpymep1.exe

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

zauba.io

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seznam

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

turnitinbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://psychics.co.uk/sitemap.xml
Back to top

Comments

  • Blocking bots
  • Slowing crawl for good bots
Back to top

Warnings

  • 8 invalid lines.
Back to top