rsf.is
robots.txt

Robots Exclusion Standard data for rsf.is

Archived Snapshots

Resource Scan

Scan Details

Site Domain	rsf.is
Base Domain	rsf.is
Scan Status	Ok
Last Scan	2024-10-31T05:37:45+00:00
Next Scan	2024-11-30T05:37:45+00:00

Last Scan

Scanned	2024-10-31T05:37:45+00:00
URL	https://rsf.is/robots.txt
Domain IPs	3.164.182.111, 3.164.182.18, 3.164.182.41, 3.164.182.61
Response IP	18.165.122.15
Found	Yes
Hash	28b9510b91805802d87cc47e9a769fb9ac3e4e21b1c33a4cf577432abeacfdfe
SimHash	b01d95886448

Groups

mediapartners-google

Rule	Path
Disallow

Rule

Path

Disallow

adbeat_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

admantx platform semantic analyzer - admantx inc. - www.admantx.com - support@admantx.com

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aolbot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

attribot/1.1 (compatible; attribot-site; http://static.attribyte.com/robotreadme.txt)

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

bitlybot

Rule	Path
Disallow	/

Rule

Path

Disallow

bot.araturka.com

Rule	Path
Disallow	/

Rule

Path

Disallow

butterfly/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

claritybot

Rule	Path
Disallow	/

Rule

Path

Disallow

claritydailybot

Rule	Path
Disallow	/

Rule

Path

Disallow

cms crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler4j (http://code.google.com/p/crawler4j/)

Rule	Path
Disallow	/

Rule

Path

Disallow

crazywebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

dialogsearch.com

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

domainappender

Rule	Path
Disallow	/

Rule

Path

Disallow

domainsigmacrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

domain re-animator bot

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms robot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

favorg

Rule	Path
Disallow	/

Rule

Path

Disallow

findxbot

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboardproxy

Rule	Path
Disallow	/

Rule

Path

Disallow

friendica

Rule	Path
Disallow	/

Rule

Path

Disallow

gigablastopensource/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

google-http-java-client

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

hatena star

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix/2.0.2 +http://www.adsafemedia.com

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack 3

Rule	Path
Disallow	/

Rule

Path

Disallow

inagist url resolver

Rule	Path
Disallow	/

Rule

Path

Disallow

insitesbot

Rule	Path
Disallow	/

Rule

Path

Disallow

jack

Rule	Path
Disallow	/

Rule

Path

Disallow

james bot

Rule	Path
Disallow	/

Rule

Path

Disallow

java

Rule	Path
Disallow	/

Rule

Path

Disallow

js-kit url resolver, http://js-kit.com/

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

livelapbot/0.2 (http://site.livelap.com/crawler)

Rule	Path
Disallow	/

Rule

Path

Disallow

ls session

Rule	Path
Disallow	/

Rule

Path

Disallow

lssrocketcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

ltx71

Rule	Path
Disallow	/

Rule

Path

Disallow

madaali.de

Rule	Path
Disallow	/

Rule

Path

Disallow

maxpointcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

meanpathbot

Rule	Path
Disallow	/

Rule

Path

Disallow

metauri api/2.0 +metauri.com

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mozilla/4.0 (cms crawler: http://www.cmscrawler.com)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 8.0; windows nt 6.0 ; claritydailybot)

Rule

Path

Disallow

mozilla/5.0 (compatible; 200pleasebot/1.0; +http://www.200please.com/bot)

Rule

Path

Disallow

mozilla/5.0 (compatible;acapbot/0.1;treat like googlebot)

Rule

Path

Disallow

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

Rule

Path

Disallow

mozilla/5.0 (compatible; aolbot/4.0; +http://www.aol-soft.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; domainappender /1.0; +http://www.profound.net/domainappender)

Rule

Path

Disallow

mozilla/5.0 (compatible; domainsigmacrawler/0.1; +http://domainsigma.com/robot)

Rule

Path

Disallow

mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)

Rule

Path

Disallow

mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; genieo/1.0 http://www.genieo.com/webfilter.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Rule

Path

Disallow

mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

Rule

Path

Disallow

mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Rule

Path

Disallow

mozilla/5.0 (compatible; openhosebot/2.1; +http://www.openhose.org/bot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; pad-bot/9.0; +http://www.descargarprogramagratis.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule

Path

Disallow

mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

Rule

Path

Disallow

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; smtbot/1.0; +http://www.similartech.com/smtbot)

Rule

Path

Disallow

mozilla/5.0 (compatible; softlistbot/2.2; +http://www.softlist.us/)

Rule

Path

Disallow

mozilla/5.0 (compatible; tweetedtimes bot/1.0; +http://tweetedtimes.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; tweetmemebot/3.0; +http://tweetmeme.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; umbot-ln/1.0; mailto: crawling@ubermetrics-technologies.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; xovibot/2.0; +http://www.xovibot.net/)

Rule

Path

Disallow

mozilla/5.0 (macintosh; intel mac os x 10.9; rv:28.0) gecko/20100101 firefox/28.0 (flipboardproxy/1.1; +http://flipboard.com/browserproxy)

Rule

Path

Disallow

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.1.6) gecko/20070725 firefox/2.0.0.6 - james bot - webcrawler http://cognitiveseo.com/bot.html

Rule

Path

Disallow

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.1.2) gecko/20090729 firefox/3.5.2 (.net clr 3.5.30729; diffbot/0.1; +http://www.diffbot.com)

Rule

Path

Disallow

mozilla/5.0 (windows nt 6.2) insitesbot/1.0

Rule

Path

Disallow

ms search 4.0 robot

Rule

Path

Disallow

ms search 5.0 robot

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

nativehost

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

netshelter contentscan

Rule

Path

Disallow

newsme/1.0; feedback@news.me

Rule

Path

Disallow

niki-bot

Rule

Path

Disallow

ning/1.0

Rule

Path

Disallow

node/simplecrawler 0.3.9 (http://github.com/cgiffard/node-simplecrawler.git)

Rule

Path

Disallow

nutch

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

pad-bot

Rule

Path

Disallow

paperlibot

Rule

Path

Disallow

peerindex

Rule

Path

Disallow

perl lwp

Rule

Path

Disallow

photon

Rule

Path

Disallow

postano

Rule

Path

Disallow

proximic

Rule

Path

Disallow

pulsecrawler/1.1

Rule

Path

Disallow

queryseekerspider

Rule

Path

Disallow

quipu

Rule

Path

Disallow

raven

Rule

Path

Disallow

ravencrawler

Rule

Path

Disallow

recorded future

Rule

Path

Disallow

ruby

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

seolyticscrawler

Rule

Path

Disallow

showyoubot (http://showyou.com/crawler)

Rule

Path

Disallow

simplecrawler

Rule

Path

Disallow

sistrix crawler

Rule

Path

Disallow

smeshbot

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

srmse/nutch

Rule

Path

Disallow

softlistbot

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

stratagems kumo

Rule

Path

Disallow

tbot-nutch

Rule

Path

Disallow

turnitin robot

Rule

Path

Disallow

tweetmemebot

Rule

Path

Disallow

typhoeus - https://github.com/typhoeus/typhoeus

Rule

Path

Disallow

umbot-ln

Rule

Path

Disallow

uptimerobot/2.0

Rule

Path

Disallow

webindex

Rule

Path

Disallow

wesee

Rule

Path

Disallow

wesee:ads/pagebot (http://www.wesee.com/bot/)

Rule

Path

Disallow

wesee:ads/picturebot (http://www.wesee.com/bot/)

Rule

Path

Disallow

wiseguys robot

Rule

Path

Disallow

voltron

Rule

Path

Disallow

woobot

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

yahoo pipes 1.0

Rule

Path

Disallow

yioopbot

Rule

Path

Disallow

Comments

See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
To ban all spiders from the entire site uncomment the next two lines:
User-Agent: *
Disallow: /
Googlebot and AdSense

Warnings

6 invalid lines.

rsf.isrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

mediapartners-google

adbeat_bot

admantx platform semantic analyzer - admantx inc. - www.admantx.com - support@admantx.com

ahrefsbot

aolbot/4.0

attribot/1.1 (compatible; attribot-site; http://static.attribyte.com/robotreadme.txt)

baiduspider-image

bitlybot

bot.araturka.com

butterfly/1.0

claritybot

claritydailybot

cms crawler

crawler4j (http://code.google.com/p/crawler4j/)

crazywebcrawler

dialogsearch.com

diffbot

domainappender

domainsigmacrawler

domain re-animator bot

ezooms robot

exabot

favorg

findxbot

flipboardproxy

friendica

gigablastopensource/1.0

google-http-java-client

grapeshot

hatena star

heritrix

heritrix/2.0.2 +http://www.adsafemedia.com

httrack 3

inagist url resolver

insitesbot

jack

james bot

java

js-kit url resolver, http://js-kit.com/

linkdexbot

livelapbot/0.2 (http://site.livelap.com/crawler)

ls session

lssrocketcrawler

ltx71

madaali.de

maxpointcrawler

meanpathbot

metauri api/2.0 +metauri.com

mj12bot

mozilla/4.0 (cms crawler: http://www.cmscrawler.com)

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot)

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 4.0 robot)

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 5.0 robot)

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 5.0 robot)

mozilla/4.0 (compatible; msie 8.0; windows nt 6.0 ; claritydailybot)

mozilla/5.0 (compatible; 200pleasebot/1.0; +http://www.200please.com/bot)

mozilla/5.0 (compatible;acapbot/0.1;treat like googlebot)

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

mozilla/5.0 (compatible; aolbot/4.0; +http://www.aol-soft.com/)

mozilla/5.0 (compatible; domainappender /1.0; +http://www.profound.net/domainappender)

mozilla/5.0 (compatible; domainsigmacrawler/0.1; +http://domainsigma.com/robot)

mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)

mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)

mozilla/5.0 (compatible; genieo/1.0 http://www.genieo.com/webfilter.html)

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

mozilla/5.0 (compatible; openhosebot/2.1; +http://www.openhose.org/bot.html)

mozilla/5.0 (compatible; pad-bot/9.0; +http://www.descargarprogramagratis.com/)

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

mozilla/5.0 (compatible; smtbot/1.0; +http://www.similartech.com/smtbot)

mozilla/5.0 (compatible; softlistbot/2.2; +http://www.softlist.us/)

rsf.is
robots.txt