rolabs.net
robots.txt

Robots Exclusion Standard data for rolabs.net

Resource Scan

Scan Details

Site Domain rolabs.net
Base Domain rolabs.net
Scan Status Ok
Last Scan2025-09-08T23:34:04+00:00
Next Scan 2025-10-08T23:34:04+00:00

Last Scan

Scanned2025-09-08T23:34:04+00:00
URL https://rolabs.net/robots.txt
Domain IPs 104.21.38.209, 172.67.138.148, 2606:4700:3030::ac43:8a94, 2606:4700:3033::6815:26d1
Response IP 172.67.138.148
Found Yes
Hash 4a2a728c2ff5fb6fb1dc1c9384fc13d12e2b43400afd25d1c8771cb8e3823176
SimHash 4e0d10d3f595

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /_nuxt/
Disallow /admin/
Allow /products
Allow /services
Allow /contact
Allow /partners

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

Other Records

Field Value
sitemap https://rolabs.net/sitemap.xml

Comments

  • Sitemap
  • Блокируем API и внутренние папки
  • Разрешаем основные страницы
  • Allow: /api/
  • Crawl-delay для вежливого краулинга
  • Специальные правила для поисковых ботов