Log In Sign Up

uk.kayali.com
robots.txt

Robots Exclusion Standard data for uk.kayali.com

Resource Scan

Scan Details

Site Domain uk.kayali.com
Base Domain kayali.com
Scan Status Ok
Last Scan2026-03-17T10:37:43+00:00
Next Scan 2026-03-31T10:37:43+00:00

Last Scan

Scanned2026-03-17T10:37:43+00:00
URL https://uk.kayali.com/robots.txt
Domain IPs 23.227.38.74, 2620:127:f00f:e::
Response IP 23.227.38.74
Found Yes
Hash b031925d0f987c2f93fd0f8e3794e55e2cb0b57f91d08440d260c5693cc1c9ee
SimHash 2e508cd2fdf1

Groups

*

Rule Path
Disallow /admin
Disallow /account
Disallow /orders
Disallow /cart$
Disallow /carts
Disallow /checkout$
Disallow /checkouts/
Disallow /a/downloads/-/*
Disallow /sf_private_access_tokens
Disallow /.well-known/shopify/monorail
Disallow /apple-app-site-association
Disallow /search
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /*?*oseid=*
Disallow /collections/*?*sort_by=*
Disallow /*/collections/*?*sort_by=*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /recommendations/products
Disallow /*/recommendations/products

adsbot-google

Rule Path
Disallow /cart$
Disallow /carts
Disallow /checkout$
Disallow /checkouts/
Disallow /orders
Disallow /sf_private_access_tokens
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /*?*oseid=*
Disallow /cdn/wpm/*.js

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

nutch

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.example.com/sitemap.xml
Back to top

Comments

  • System & private
  • Search and previews
  • Collection/Blog noise (sorting, filters, URL-encoded “+”)
  • Product recommendations API endpoints
  • Allow policy pages to index (valuable content)
  • (Remove any Disallow for /policies/)
  • Allow: /policies/
  • AdsBot needs explicit rules (it otherwise ignores generic UA blocks)
  • Throttle certain crawlers (optional)
Back to top