published-assets.ari-build.com
robots.txt

Robots Exclusion Standard data for published-assets.ari-build.com

Resource Scan

Scan Details

Site Domain published-assets.ari-build.com
Base Domain ari-build.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-09-30T09:41:14+00:00
Next Scan 2024-12-29T09:41:14+00:00

Last Successful Scan

Scanned2024-02-11T09:39:08+00:00
URL https://published-assets.ari-build.com/robots.txt
Domain IPs 18.245.143.110, 18.245.143.123, 18.245.143.58, 18.245.143.76
Response IP 52.222.236.105
Found Yes
Hash b0907e92766640b12ccc8690cedf7384dcc35e2b144c59e16d73a316033b25d5
SimHash 434cc805e6ba

Groups

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

adsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

amazonbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 50

yandex

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

audioeye

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow

Other Records

Field Value
sitemap http://static-assets.ari-build.com/sitemap.xml

Comments

  • 08022017: (http) - ARI Network Services
  • Block MegaIndex.ru
  • SEO Star
  • Slow Amazonbot
  • Block YandexBot
  • Block Baidu
  • Block Youdao
  • Block Majestic
  • Slow Rogerbot (Moz's crawler)
  • Slow AudioEye
  • Slow BingBot