Log In Sign Up

superstitio.com
robots.txt

Robots Exclusion Standard data for superstitio.com

Resource Scan

Scan Details

Site Domain superstitio.com
Base Domain superstitio.com
Scan Status Ok
Last Scan2024-11-15T13:52:15+00:00
Next Scan 2024-11-22T13:52:15+00:00

Last Scan

Scanned2024-11-15T13:52:15+00:00
URL https://superstitio.com/robots.txt
Domain IPs 68.168.113.186
Response IP 68.168.113.186
Found Yes
Hash f14b6a701878ea7582f5ba1b329bd7e14563157b5ad7c6759205d4ad130b09cd
SimHash 3b964195aee0

Groups

mediapartners-google

Rule Path
Disallow

qwarrybot

Rule Path
Disallow /

docoloc

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

newspaper

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

seekport

Rule Path
Disallow /

cipacrawler

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

rankingbot

Rule Path
Disallow /

rankingbot2

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

cocolyzebot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

ucrawler

Rule Path
Disallow /

jersey

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

hypestat

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

bubing

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

laserlikebot

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

linqiametadatadownloaderbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

spiderbot/nutch-1.7

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

dazoobot

Rule Path
Disallow /

advbot

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

feedlybot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

semalt

Rule Path
Disallow /

betabot

Rule Path
Disallow /

gimme60bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

moreover

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

maxpointcrawler/nutch-1.1

Rule Path
Disallow /

nutch-1.1

Rule Path
Disallow /

maxpoint.crawler

Rule Path
Disallow /

xyzbot

Rule Path
Disallow /

nbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

facebot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

voltron

Rule Path
Disallow /

umbot-ln

Rule Path
Disallow /

umbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

easou

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ncbot

Rule Path
Disallow /

expo9

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

genieo

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

affectv

Rule Path
Disallow /

affectv robot v1.0

Rule Path
Disallow /

affectv robot v1.0/nutch-1.6

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrixcrawler

Rule Path
Disallow /

openwebindex/nutch-1.6

Rule Path
Disallow /

openwebindex

Rule Path
Disallow /

nutch-1.6

Rule Path
Disallow /

wikiwix-bot-3.0

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

kraken

Rule Path
Disallow /

kraken/0.1

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yahoo pipes 2.0

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

httrack

Rule Path
Disallow /

acoon

Rule Path
Disallow /

siteintel

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

skimwordsbot

Rule Path
Disallow /

skimbot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

steeler

Rule Path
Disallow /

gosospider

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

radian6

Rule Path
Disallow /

radian6_default

Rule Path
Disallow /

ptd-crawler

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

tineye

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

webmastercoffee

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

seodat

Rule Path
Disallow /

gimmie60

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

magpie-crawler/1.1

Rule Path
Disallow /

sbider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

my-robot

Rule Path
Disallow /

lynnbot

Rule Path
Disallow /

holmes

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

apptusbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

flatlandbot

Rule Path
Disallow /

great-plains-web-spider

Rule Path
Disallow /

verticalman

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

webalta

Rule Path
Disallow /

kaloogabot

Rule Path
Disallow /

sproose

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

onetszukaj

Rule Path
Disallow /

grub

Rule Path
Disallow /

psbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yanga

Rule Path
Disallow /

kalooga

Rule Path
Disallow /

teoma

Rule Path
Disallow /

itsapic.com_crawler

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

envolk

Rule Path
Disallow /

*

Rule Path
Disallow /turing_images/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap http://superstitio.com/sitemap.xml.gz
Back to top

Warnings

  • 6 invalid lines.
Back to top