Log In Sign Up

artodata.net
robots.txt

Robots Exclusion Standard data for artodata.net

Resource Scan

Scan Details

Site Domain artodata.net
Base Domain artodata.net
Scan Status Ok
Last Scan2025-10-10T21:21:57+00:00
Next Scan 2025-10-17T21:21:57+00:00

Last Scan

Scanned2025-10-10T21:21:57+00:00
URL https://artodata.net/robots.txt
Domain IPs 93.191.156.251
Response IP 93.191.156.251
Found Yes
Hash e80f3bcc69f7be90d0a96e1126f305bdc0feb4ce1dd98a088250a12a9d2cc8d1
SimHash 6b54d53106b2

Groups

*

Rule Path
Disallow /site/admin/
Disallow /libs/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /
Back to top

Comments

  • ===== Sitemaps =====
  • Sitemap must be an absolute URL (protocol + domain), not a relative path
  • Sitemap: https://www.arto.dk/sitemap.xml
  • ===== General rules for all bots =====
  • ===== Slow down aggressive but legitimate crawlers =====
  • ===== Safe to Block (no major SEO impact) =====
  • SEO backlink crawler, high request rate, no direct SEO value unless you use Ahrefs.
  • SEO auditing crawler, often hits hard, only useful if you pay for SEMrush.
  • Majestic SEO crawler, mainly for their backlink index, not useful unless you use Majestic.
  • Moz crawler for link data, can be very aggressive, only useful for Moz subscribers.
  • Russian SEO bot, known for scraping, not a real search engine.
  • Hawaiian ISP-based crawler used for scraping, often ignores crawl-delay.
  • Huawei's search engine bot, no SEO loss unless targeting Huawei search.
  • Old European search engine crawler, low relevance today.
  • Aggressive link scraper used for SEO databases, no benefit unless you use BLEX.
  • Common Crawl bot, archives web pages for public datasets, not needed for SEO.
  • Obsolete search engine bot, now mostly used for scraping.
  • French search engine bot, low relevance unless targeting France.
  • ===== Maybe Block (depends on audience) =====
  • Chinese search engine, block unless you target mainland China traffic.
  • Russian search engine, block unless you target Russian-speaking audiences.
  • South Korean search engine bot, block unless targeting South Korea.
  • Chinese search engine crawler, block unless targeting Chinese market.
Back to top