ffxivaddicts.com
robots.txt

Robots Exclusion Standard data for ffxivaddicts.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	ffxivaddicts.com
Base Domain	ffxivaddicts.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-05-16T08:39:06+00:00
Next Scan	2024-08-14T08:39:06+00:00

Last Successful Scan

Scanned	2024-01-18T07:48:26+00:00
URL	https://ffxivaddicts.com/robots.txt
Domain IPs	104.21.12.186, 172.67.195.64, 2606:4700:3033::6815:cba, 2606:4700:3035::ac43:c340
Response IP	172.67.195.64
Found	Yes
Hash	2fa0ab077d6f5675567563ec3390ea9b4795f81a0c656d86c0b8f1835ac5e306
SimHash	305953d86408

Groups

aboundexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

acapbot

Rule	Path
Disallow	/

Rule

Path

Disallow

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

adnormcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aolbot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

attribot

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

bot.araturka.com

Rule	Path
Disallow	/

Rule

Path

Disallow

butterfly/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

buzzsumo

Rule	Path
Disallow	/

Rule

Path

Disallow

cis455crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

claritybot

Rule	Path
Disallow	/

Rule

Path

Disallow

claritydailybot

Rule	Path
Disallow	/

Rule

Path

Disallow

clickagy intelligence bot

Rule	Path
Disallow	/

Rule

Path

Disallow

cms crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

commoncrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler4j (http://code.google.com/p/crawler4j/)

Rule	Path
Disallow	/

Rule

Path

Disallow

crazywebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

crazywebcrawler-spider

Rule	Path
Disallow	/

Rule

Path

Disallow

dataparksearch

Rule	Path
Disallow	/

Rule

Path

Disallow

deepcrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

deusu

Rule	Path
Disallow	/

Rule

Path

Disallow

dialogsearch.com

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dj-research

Rule	Path
Disallow	/

Rule

Path

Disallow

domainappender

Rule	Path
Disallow	/

Rule

Path

Disallow

domainmacrocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

domainsigmacrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

domain re-animator bot

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms robot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

experibot

Rule	Path
Disallow	/

Rule

Path

Disallow

favorg

Rule	Path
Disallow	/

Rule

Path

Disallow

findxbot

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboardproxy

Rule	Path
Disallow	/

Rule

Path

Disallow

friendica

Rule	Path
Disallow	/

Rule

Path

Disallow

getproxi.es-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

gigablastopensource/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

gluten free crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

google-http-java-client

Rule	Path
Disallow	/

Rule

Path

Disallow

gocrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshotcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

guzzlehttp

Rule	Path
Disallow	/

Rule

Path

Disallow

hatena star

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack 3

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

inagist url resolver

Rule

Path

Disallow

insitesbot

Rule

Path

Disallow

iskanie

Rule

Path

Disallow

jack

Rule

Path

Disallow

james bot

Rule

Path

Disallow

java

Rule

Path

Disallow

js-kit url resolver

Rule

Path

Disallow

komodiabot

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

livelapbot

Rule

Path

Disallow

ls session

Rule

Path

Disallow

lssrocketcrawler

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

madaali.de

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

memorybot

Rule

Path

Disallow

metauri api/2.0 +metauri.com

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mozilla/4.0 (cms crawler: http://www.cmscrawler.com)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 8.0; windows nt 6.0 ; claritydailybot)

Rule

Path

Disallow

mozilla/5.0 (compatible; 200pleasebot/1.0; +http://www.200please.com/bot)

Rule

Path

Disallow

mozilla/5.0 (compatible;acapbot/0.1;treat like googlebot)

Rule

Path

Disallow

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

Rule

Path

Disallow

mozilla/5.0 (compatible; aolbot/4.0; +http://www.aol-soft.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; domainappender /1.0; +http://www.profound.net/domainappender)

Rule

Path

Disallow

mozilla/5.0 (compatible; domainsigmacrawler/0.1; +http://domainsigma.com/robot)

Rule

Path

Disallow

mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)

Rule

Path

Disallow

mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; genieo/1.0 http://www.genieo.com/webfilter.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

Rule

Path

Disallow

mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Rule

Path

Disallow

mozilla/5.0 (compatible; openhosebot/2.1; +http://www.openhose.org/bot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; pad-bot/9.0; +http://www.descargarprogramagratis.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule

Path

Disallow

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule

Path

Disallow

mozilla/5.0 (compatible; softlistbot/2.2; +http://www.softlist.us/)

Rule

Path

Disallow

mozilla/5.0 (compatible; tweetedtimes bot/1.0; +http://tweetedtimes.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; tweetmemebot/3.0; +http://tweetmeme.com/)

Rule

Path

Disallow

mozilla/5.0 (compatible; umbot-ln/1.0; mailto: crawling@ubermetrics-technologies.com)

Rule

Path

Disallow

mozilla/5.0 (compatible; xovibot/2.0; +http://www.xovibot.net/)

Rule

Path

Disallow

mozilla/5.0 (macintosh; intel mac os x 10.9; rv:28.0) gecko/20100101 firefox/28.0 (flipboardproxy/1.1; +http://flipboard.com/browserproxy)

Rule

Path

Disallow

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.1.6) gecko/20070725 firefox/2.0.0.6 - james bot - webcrawler http://cognitiveseo.com/bot.html

Rule

Path

Disallow

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.1.2) gecko/20090729 firefox/3.5.2 (.net clr 3.5.30729; diffbot/0.1; +http://www.diffbot.com)

Rule

Path

Disallow

mozilla/5.0 (windows nt 6.2) insitesbot/1.0

Rule

Path

Disallow

ms search 4.0 robot

Rule

Path

Disallow

ms search 5.0 robot

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

nativehost

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

netseer crawler

Rule

Path

Disallow

netshelter contentscan

Rule

Path

Disallow

newsme/1.0; feedback@news.me

Rule

Path

Disallow

niki-bot

Rule

Path

Disallow

ning/1.0

Rule

Path

Disallow

node/simplecrawler 0.3.9 (http://github.com/cgiffard/node-simplecrawler.git)

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

pad-bot

Rule

Path

Disallow

panscient.com

Rule

Path

Disallow

paperlibot

Rule

Path

Disallow

peerindex

Rule

Path

Disallow

perl lwp

Rule

Path

Disallow

photon

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

postano

Rule

Path

Disallow

proximic

Rule

Path

Disallow

pulsecrawler/1.1

Rule

Path

Disallow

pulsepoint xt3 web scraper

Rule

Path

Disallow

queryseekerspider

Rule

Path

Disallow

quipu

Rule

Path

Disallow

raven

Rule

Path

Disallow

ravencrawler

Rule

Path

Disallow

rbot

Rule

Path

Disallow

recorded future

Rule

Path

Disallow

riddler

Rule

Path

Disallow

ruby

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

scrapy

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

searchie

Rule

Path

Disallow

semantic-visions.com crawler

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

seolyticscrawler

Rule

Path

Disallow

showyoubot (http://showyou.com/crawler)

Rule

Path

Disallow

simplecrawler

Rule

Path

Disallow

sistrix crawler

Rule

Path

Disallow

siteluxbot

Rule

Path

Disallow

smeshbot

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

srmse/nutch

Rule

Path

Disallow

softlistbot

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

spbot

Rule

Path

Disallow

squirrly

Rule

Path

Disallow

ssearch_bot

Rule

Path

Disallow

stratagems kumo

Rule

Path

Disallow

tbot-nutch

Rule

Path

Disallow

telegrambot

Rule

Path

Disallow

tweetmemebot

Rule

Path

Disallow

twmbot

Rule

Path

Disallow

typhoeus - https://github.com/typhoeus/typhoeus

Rule

Path

Disallow

umbot-ln

Rule

Path

Disallow

uptimerobot/2.0

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

webindex

Rule

Path

Disallow

vebot

Rule

Path

Disallow

wesee_bot

Rule

Path

Disallow

wiseguys robot

Rule

Path

Disallow

voltron

Rule

Path

Disallow

woobot

Rule

Path

Disallow

woriobot

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

x-cad-se

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

yahoo pipes 1.0

Rule

Path

Disallow

yioopbot

Rule

Path

Disallow

Comments

Blocklist

Warnings

6 invalid lines.

ffxivaddicts.comrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

aboundexbot

acapbot

admantx

adnormcrawler

ahrefsbot

aolbot/4.0

attribot

baiduspider-image

bot.araturka.com

butterfly/1.0

buzzsumo

cis455crawler

claritybot

claritydailybot

clickagy intelligence bot

cms crawler

commoncrawler

crawler4j (http://code.google.com/p/crawler4j/)

crazywebcrawler

crazywebcrawler-spider

dataparksearch

deepcrawl

deusu

dialogsearch.com

diffbot

dj-research

domainappender

domainmacrocrawler

domainsigmacrawler

domain re-animator bot

ezooms robot

exabot

experibot

favorg

findxbot

flipboardproxy

friendica

getproxi.es-bot

gigablastopensource/1.0

gluten free crawler

google-http-java-client

gocrawl

grapeshot

grapeshotcrawler

guzzlehttp

hatena star

heritrix

heritrix

httrack 3

ia_archiver

inagist url resolver

insitesbot

iskanie

jack

james bot

java

js-kit url resolver

komodiabot

linkdexbot

livelapbot

ls session

lssrocketcrawler

ltx71

madaali.de

meanpathbot

megaindex.ru

memorybot

metauri api/2.0 +metauri.com

mj12bot

mozilla/4.0 (cms crawler: http://www.cmscrawler.com)

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot)

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 4.0 robot)

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 5.0 robot)

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 5.0 robot)

mozilla/4.0 (compatible; msie 8.0; windows nt 6.0 ; claritydailybot)

ffxivaddicts.com
robots.txt