Log In Sign Up

horsetype.com
robots.txt

Robots Exclusion Standard data for horsetype.com

Resource Scan

Scan Details

Site Domain horsetype.com
Base Domain horsetype.com
Scan Status Ok
Last Scan2024-11-09T08:46:10+00:00
Next Scan 2024-11-16T08:46:10+00:00

Last Scan

Scanned2024-11-09T08:46:10+00:00
URL https://horsetype.com/robots.txt
Domain IPs 104.21.15.108, 172.67.162.48, 2606:4700:3036::6815:f6c, 2606:4700:3037::ac43:a230
Response IP 104.21.15.108
Found Yes
Hash f99b7a4bbed9b36a0b191a1d97187ed300544caa1f08176e4e60d844a1ddb943
SimHash 3a945b37aef2

Groups

mediapartners-google

Rule Path
Disallow

crazywebcrawler

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

linqiametadatadownloaderbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

addthis.com

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

spiderbot/nutch-1.7

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

dazoobot

Rule Path
Disallow /

advbot

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

feedlybot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

semalt

Rule Path
Disallow /

betabot

Rule Path
Disallow /

gimme60bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

moreover

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

maxpointcrawler/nutch-1.1

Rule Path
Disallow /

nutch-1.1

Rule Path
Disallow /

maxpoint.crawler

Rule Path
Disallow /

xyzbot

Rule Path
Disallow /

nbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

facebot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

voltron

Rule Path
Disallow /

umbot-ln

Rule Path
Disallow /

umbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

easou

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ncbot

Rule Path
Disallow /

expo9

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

genieo

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

affectv

Rule Path
Disallow /

affectv robot v1.0

Rule Path
Disallow /

affectv robot v1.0/nutch-1.6

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrixcrawler

Rule Path
Disallow /

openwebindex/nutch-1.6

Rule Path
Disallow /

openwebindex

Rule Path
Disallow /

nutch-1.6

Rule Path
Disallow /

wikiwix-bot-3.0

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

kraken

Rule Path
Disallow /

kraken/0.1

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yahoo pipes 2.0

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

httrack

Rule Path
Disallow /

acoon

Rule Path
Disallow /

siteintel

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

skimwordsbot

Rule Path
Disallow /

skimbot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

steeler

Rule Path
Disallow /

gosospider

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

radian6

Rule Path
Disallow /

radian6_default

Rule Path
Disallow /

ptd-crawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

tineye

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

webmastercoffee

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

seodat

Rule Path
Disallow /

gimmie60

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

magpie-crawler/1.1

Rule Path
Disallow /

sbider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

my-robot

Rule Path
Disallow /

lynnbot

Rule Path
Disallow /

holmes

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

apptusbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

flatlandbot

Rule Path
Disallow /

great-plains-web-spider

Rule Path
Disallow /

verticalman

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

webalta

Rule Path
Disallow /

kaloogabot

Rule Path
Disallow /

sproose

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

onetszukaj

Rule Path
Disallow /

grub

Rule Path
Disallow /

psbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yanga

Rule Path
Disallow /

kalooga

Rule Path
Disallow /

teoma

Rule Path
Disallow /

itsapic.com_crawler

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

envolk

Rule Path
Disallow /

*

Rule Path
Disallow /turing_images/
Disallow /sendmail.php

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap http://horsetype.com/sitemap.xml.gz
Back to top

Warnings

  • 6 invalid lines.
Back to top