arcunia.com
robots.txt

Robots Exclusion Standard data for arcunia.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	arcunia.com
Base Domain	arcunia.com
Scan Status	Ok
Last Scan	2024-10-25T05:59:43+00:00
Next Scan	2024-11-24T05:59:43+00:00

Last Scan

Scanned	2024-10-25T05:59:43+00:00
URL	https://arcunia.com/robots.txt
Domain IPs	37.72.98.167
Response IP	37.72.98.167
Found	Yes
Hash	685c8ab78dd1aafed4bb62110adc373cfbfa7ed5d7fe7656968871f70ed93b87
SimHash	a0150214074e

Groups

admantx-euaspb\2.5

Rule	Path
Disallow	/

Rule

Path

Disallow

adreview\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

adstxt.com\1.2

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot\7.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot\5.2

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot\6.1

Rule	Path
Disallow	/

Rule

Path

Disallow

alphabot\3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot\0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

apache-httpclient\4.5.3

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot\0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

archive.org

Rule	Path
Disallow	/

Rule

Path

Disallow

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

arkerarequester

Rule	Path
Disallow	/

Rule

Path

Disallow

axios\0.21.1

Rule	Path
Disallow	/

Rule

Path

Disallow

b2b bot

Rule	Path
Disallow	/

Rule

Path

Disallow

baidu.com

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider\2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler\0.7

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler\0.9

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler\0.5.1

Rule	Path
Disallow	/

Rule

Path

Disallow

bingbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bingbot\2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bingpreview\1.0b

Rule	Path
Disallow	/

Rule

Path

Disallow

bitsightbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bitsightbot\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

boardreader favicon fetcher \1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

boardreader

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

cbsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot\2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

censysinspect\1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

cincraw

Rule	Path
Disallow	/

Rule

Path

Disallow

cincraw\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cipacrawler\3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

checkmarknetwork\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

clarabot\1.4

Rule	Path
Disallow	/

Rule

Path

Disallow

cliqzbot\3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-image

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-image\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-web\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-web\4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cortex\1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

contacts-crawler\0.2

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

crazywebcrawler 0.9.10

Rule

Path

Disallow

crios\16.15

Rule

Path

Disallow

dataforseobot\1.0

Rule

Path

Disallow

dataprovider

Rule

Path

Disallow

daum

Rule

Path

Disallow

daum\4.1

Rule

Path

Disallow

diffbot

Rule

Path

Disallow

discordbot\2.0

Rule

Path

Disallow

dispatch\0.11.3

Rule

Path

Disallow

dnyzbot\1.0

Rule

Path

Disallow

domaincheck.io crawler

Rule

Path

Disallow

domaincrawler

Rule

Path

Disallow

domaincrawler\3.0

Rule

Path

Disallow

domainsigmacrawler\0.1

Rule

Path

Disallow

domainstatsbot\1.0

Rule

Path

Disallow

dotbot\1.1

Rule

Path

Disallow

duckduckbot-https

Rule

Path

Disallow

duckduckbot

Rule

Path

Disallow

duckduckbot-https\1.1

Rule

Path

Disallow

duckduckgo-favicons-bot\1.0

Rule

Path

Disallow

exabot\3.0

Rule

Path

Disallow

extlinksbot

Rule

Path

Disallow

extlinksbot\1.5

Rule

Path

Disallow

evc-batch\2.0

Rule

Path

Disallow

ev-crawler\1.0

Rule

Path

Disallow

facebookexternalhit\1.1

Rule

Path

Disallow

feedlybot

Rule

Path

Disallow

findxbot

Rule

Path

Disallow

seekport crawler

Rule

Path

Disallow

seekport

Rule

Path

Disallow

garlikcrawler\1.2

Rule

Path

Disallow

geedobot

Rule

Path

Disallow

go-resty\2.0.0

Rule

Path

Disallow

g-i-g-a-b-o-t

Rule

Path

Disallow

grequests\0.10

Rule

Path

Disallow

go-http-client

Rule

Path

Disallow

go-http-client\1.1

Rule

Path

Disallow

gptbot\1.2

Rule

Path

Disallow

gptbot

Rule

Path

Disallow

guzzlehttp

Rule

Path

Disallow

hubspot webcrawler

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

iceweasel\30.0

Rule

Path

Disallow

idbot\1.1

Rule

Path

Disallow

imagefetcher\8.0

Rule

Path

Disallow

imagesiftbot

Rule

Path

Disallow

infopath.2

Rule

Path

Disallow

ioncrawl

Rule

Path

Disallow

ips-agent

Rule

Path

Disallow

istellabot\t.1.13

Rule

Path

Disallow

jooblebot\2.0

Rule

Path

Disallow

komodiabot\1.0

Rule

Path

Disallow

konqueror\3

Rule

Path

Disallow

libcurl

Rule

Path

Disallow

line-poker\1.0

Rule

Path

Disallow

linkdexbot\2.2

Rule

Path

Disallow

linkedinbot\1.0

Rule

Path

Disallow

lunabot

Rule

Path

Disallow

magic browser

Rule

Path

Disallow

mail.ru_bot

Rule

Path

Disallow

mail.ru_bot\robots\2.0

Rule

Path

Disallow

mappy\1.0

Rule

Path

Disallow

mechanize

Rule

Path

Disallow

mechanize\2.7.5 ruby\2.2.3p173

Rule

Path

Disallow

megaindex

Rule

Path

Disallow

megaindex.ru\2.0

Rule

Path

Disallow

metauri api

Rule

Path

Disallow

mauibot

Rule

Path

Disallow

mixrankbot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mj12bot\v1.4.7

Rule

Path

Disallow

mj12bot\v1.4.8

Rule

Path

Disallow

mojeekbot\0.6

Rule

Path

Disallow

msnbot\2.0b

Rule

Path

Disallow

netcraft web server survey

Rule

Path

Disallow

netcraftsurveyagent

Rule

Path

Disallow

netcraftsurveyagent\1.0

Rule

Path

Disallow

neevabot\1.0

Rule

Path

Disallow

newspaper

Rule

Path

Disallow

newspaper\0.1.7

Rule

Path

Disallow

nimbostratus-bot\v1.3.2

Rule

Path

Disallow

ntentbot

Rule

Path

Disallow

nutch-1.4

Rule

Path

Disallow

oai-searchbot\1.0

Rule

Path

Disallow

searchbot

Rule

Path

Disallow

obot\2.3.1

Rule

Path

Disallow

obot

Rule

Path

Disallow

orbbot

Rule

Path

Disallow

orbbot\1.1

Rule

Path

Disallow

okhttp\3.8.1

Rule

Path

Disallow

openindexspider

Rule

Path

Disallow

openlinkprofiler

Rule

Path

Disallow

orbbot\1.1

Rule

Path

Disallow

netcraftsurveyagent\1.0

Rule

Path

Disallow

photon\1.0

Rule

Path

Disallow

pinterestbot

Rule

Path

Disallow

piplbot

Rule

Path

Disallow

python-requests

Rule

Path

Disallow

python-urllib\3.6

Rule

Path

Disallow

python-urllib\2.7

Rule

Path

Disallow

python-urllib\1.17

Rule

Path

Disallow

quick-crawler

Rule

Path

Disallow

qwantify\2.4w

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

qwantify\bleriot\1.1

Rule

Path

Disallow

redirector

Rule

Path

Disallow

rogerbot\1.1

Rule

Path

Disallow

rukicrawler

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

scamadviserexternalhit\1.0

Rule

Path

Disallow

scrapy\1.5.0

Rule

Path

Disallow

scrapy\2.10.0

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

seekportbot

Rule

Path

Disallow

semrushbot\1.2~bl

Rule

Path

Disallow

semrushbot\1.0

Rule

Path

Disallow

censysinspect\1.1

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

senutobot\1.0

Rule

Path

Disallow

senutobot

Rule

Path

Disallow

senuto

Rule

Path

Disallow

seokicks

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

serpstatbot\1.0

Rule

Path

Disallow

serpstatbot\2.1

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

seznambot\3.2

Rule

Path

Disallow

siteexplorer\1.1b

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

skypeuripreview preview\0.5

Rule

Path

Disallow

sogou pic spider\3.0

Rule

Path

Disallow

sogou web spider\4.0

Rule

Path

Disallow

spbot\5.0.3

Rule

Path

Disallow

spbot

Rule

Path

Disallow

special_archiver\3.1.1

Rule

Path

Disallow

spiderling

Rule

Path

Disallow

startmebot

Rule

Path

Disallow

surdotlybot\1.0

Rule

Path

Disallow

surveybot\2.3

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

tenfourfox\4.3

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

tor browser\4.1

Rule

Path

Disallow

trident\6.0

Rule

Path

Disallow

twitterbot\1.0

Rule

Path

Disallow

universalfeedparser\5.1.3

Rule

Path

Disallow

uptimebot

Rule

Path

Disallow

velenpublicwebcrawler\1.0

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

virusdie crawler

Rule

Path

Disallow

wappalyzer

Rule

Path

Disallow

webclient\1.0

Rule

Path

Disallow

webprosbot\2.0

Rule

Path

Disallow

woorankreview

Rule

Path

Disallow

woorankreview\2.0

Rule

Path

Disallow

worldbot\0.1

Rule

Path

Disallow

worldbot

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

www-mechanize

Rule

Path

Disallow

y!j-wsc\1.0

Rule

Path

Disallow

yak\1.0

Rule

Path

Disallow

yandex

Rule

Path

Disallow

yandexbot\3.0

Rule

Path

Disallow

yandexfavicons\1.0

Rule

Path

Disallow

yandexbot

Rule

Path

Disallow

yandeximages

Rule

Path

Disallow

yandexmetrika\2.0

Rule

Path

Disallow

yandexrenderresourcesbot\1.0

Rule

Path

Disallow

yandexuserproxy

Rule

Path

Disallow

yeti

Rule

Path

Disallow

yisouspider\5.0

Rule

Path

Disallow

y!j-wsc

Rule

Path

Disallow

y!j-wsc\1.0

Rule

Path

Disallow

yoozbot-2.2

Rule

Path

Disallow

zoominfobot

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.arcunia.com/1_index_sitemap.xml

Comments

robots.txt automatically generated by PrestaShop e-commerce open-source solution
http://www.prestashop.com - http://www.prestashop.com/forums
This file is to prevent the crawling and indexing of certain parts
of your site by web crawlers and spiders run by sites like Yahoo!
and Google. By telling these "robots" where not to go on your site,
you save bandwidth and server resources.
For more information about the robots.txt standard, see:
http://www.robotstxt.org/robotstxt.html

Warnings

2 invalid lines.

arcunia.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

admantx-euaspb\2.5

adreview\1.0

adstxt.com\1.2

ahrefsbot\7.0

ahrefsbot\5.2

ahrefsbot\6.1

alphabot\3.2

amazonbot\0.1

apache-httpclient\4.5.3

applebot\0.1

archive.org

archive.org_bot

arkerarequester

axios\0.21.1

b2b bot

baidu.com

baiduspider

baiduspider\2.0

barkrowler

barkrowler\0.7

barkrowler\0.9

barkrowler\0.5.1

bingbot

bingbot\2.0

bingpreview\1.0b

bitsightbot

bitsightbot\1.0

blexbot\1.0

boardreader favicon fetcher \1.0

boardreader

bubing

bytespider

cbsbot

ccbot\2.0

censysinspect\1.1

cincraw

cincraw\1.0

cipacrawler\3.0

checkmarknetwork\1.0

claudebot

claudebot\1.0

clarabot\1.4

cliqzbot\3.0

coccocbot-image

coccocbot-image\1.0

coccocbot-web\1.0

coccocbot-web\4.0

cortex\1.0

contacts-crawler\0.2

crawler4j

crazywebcrawler 0.9.10

crios\16.15

dataforseobot\1.0

dataprovider

daum

daum\4.1

diffbot

discordbot\2.0

dispatch\0.11.3

dnyzbot\1.0

domaincheck.io crawler

domaincrawler

domaincrawler\3.0

domainsigmacrawler\0.1

domainstatsbot\1.0

dotbot\1.1

duckduckbot-https

duckduckbot

duckduckbot-https\1.1

duckduckgo-favicons-bot\1.0

exabot\3.0

extlinksbot

extlinksbot\1.5

evc-batch\2.0

ev-crawler\1.0

arcunia.com
robots.txt