Log In Sign Up

kluster.ai
robots.txt

Robots Exclusion Standard data for kluster.ai

Resource Scan

Scan Details

Site Domain kluster.ai
Base Domain kluster.ai
Scan Status Ok
Last Scan2025-08-16T08:32:30+00:00
Next Scan 2025-09-15T08:32:30+00:00

Last Scan

Scanned2025-08-16T08:32:30+00:00
URL https://kluster.ai/robots.txt
Redirect https://www.kluster.ai/robots.txt
Redirect Domain www.kluster.ai
Redirect Base kluster.ai
Domain IPs 216.198.79.1
Redirect IPs 216.198.79.1, 64.29.17.1
Response IP 64.29.17.1
Found Yes
Hash 403b579a6d5c5fed9544718f5bcdd1f1732d37e5763c59482a92c759d176b362
SimHash 619fc81267e2

Groups

*

Rule Path
Allow /
Allow /blog/
Allow /about/
Allow /privacy-policy/
Allow /terms-of-use/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://kluster.ai/sitemap.xml
Back to top

Comments

  • robots.txt for kluster.ai
  • Allow all search engines to crawl the site
  • Sitemap location
  • Block crawling of sensitive areas (if any exist in the future)
  • Disallow: /admin/
  • Disallow: /api/private/
  • Allow specific crawling of important areas
  • Crawl delay for being respectful to server resources
  • Google-specific directives
  • Bing-specific directives
  • Block known bad bots (optional - can be removed if not needed)
Back to top