Log In Sign Up

joshwhoradio.net
robots.txt

Robots Exclusion Standard data for joshwhoradio.net

Resource Scan

Scan Details

Site Domain joshwhoradio.net
Base Domain joshwhoradio.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-22T00:37:46+00:00
Next Scan 2024-07-21T00:37:46+00:00

Last Successful Scan

Scanned2023-03-29T19:05:43+00:00
URL https://joshwhoradio.net/robots.txt
Redirect https://www.joshwhoradio.net/robots.txt
Redirect Domain www.joshwhoradio.net
Redirect Base joshwhoradio.net
Domain IPs 104.21.42.200, 172.67.210.12, 2606:4700:3031::6815:2ac8, 2606:4700:3032::ac43:d20c
Redirect IPs 104.21.42.200, 172.67.210.12, 2606:4700:3031::6815:2ac8, 2606:4700:3032::ac43:d20c
Response IP 172.67.210.12
Found Yes
Hash 67d153bc94ab1080e8ca52eb4aa578757974ea659284779cf532fb0376d97754
SimHash 683a45d8e688

Groups

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

owler

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

theoldreader.com

Rule Path
Disallow /

semantic-visions.com

Rule Path
Disallow /

proximic

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

viglink

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

sputnik

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

curious george

Rule Path
Disallow /

curious george - www.analyticsseo.com

Rule Path
Disallow /

curious george - www.analyticsseo.com/crawler

Rule Path
Disallow /

http://site.ru

Rule Path
Disallow /

site.ru

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

genieo

Rule Path
Disallow /

genieo/1.0

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seobility

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

istellabot/1.01.18

Rule Path
Disallow /

istellabot/1.01.18 +http://www.tiscali.it/

Rule Path
Disallow /

istellabot/1.10.2 +http://www.tiscali.it/

Rule Path
Disallow /

mozilla/5.0 (compatible; istellabot/1.01.18 +http://www.tiscali.it/)

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

grouphigh

Rule Path
Disallow /

grouphigh/1.0

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

james bot

Rule Path
Disallow /

leikibot

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

livelap

Rule Path
Disallow /

lssrocket

Rule Path
Disallow /

magpie

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

gluten free crawler/1.0

Rule Path
Disallow /

serpstatbot/1.0

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

steeler

Rule Path
Disallow /

steeler/3.5

Rule Path
Disallow /

daum

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yeti-mobile

Rule Path
Disallow /

mr.4x3 powered

Rule Path
Disallow /

sjuupbot

Rule Path
Disallow /

viglink

Rule Path
Disallow /

pi-monster

Rule Path
Disallow /

tracemyfile/1.0

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

cowbot/1.0

Rule Path
Disallow /

deskyobot

Rule Path
Disallow /

deskyobot/1.0

Rule Path
Disallow /

ltx71+-+(http://ltx71.com/)

Rule Path
Disallow /

pandalytics/1.0
ccbot/2.0
surdotlybot
cincraw/1.0
twingly recon-klondike/1.0
yak/1.0
df bot 1.0
crawlson/1.0
ioncrawl
ltx71
woorankreview/2.0
dataforseobot/1.0

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

filibot/1.0

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

*

Rule Path
Disallow /*blackhole
Disallow /?blackhole
Back to top

Comments

  • http://www.trendiction.com/en/publisher/bot
  • http://www.apple.com/go/applebot
  • http://www.grapeshot.co.uk/crawler.php
  • http://www.semrush.com/bot.html
  • http://www.semrush.com/bot.html
  • http://mj12bot.com
  • http://www.opensiteexplorer.org/dotbot
  • http://www.netseer.com/crawler.html
  • http://www.pinterest.com/bot.html
  • http://getintent.com/bot.html
  • User-Agent: aranhabot
  • Crawl-delay: 10
  • Blekkobot
  • Block BlexBot
  • Baiduspider
  • "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)"
  • "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible;Yeti-Mobile/0.1; +http://help.naver.com/robots/)"
Back to top