Log In Sign Up

hotdao.ai
robots.txt

Robots Exclusion Standard data for hotdao.ai

Resource Scan

Scan Details

Site Domain hotdao.ai
Base Domain hotdao.ai
Scan Status Ok
Last Scan2025-08-15T01:00:43+00:00
Next Scan 2025-09-14T01:00:43+00:00

Last Scan

Scanned2025-08-15T01:00:43+00:00
URL https://www.hotdao.ai/robots.txt
Domain IPs 104.21.72.14, 172.67.173.189, 2606:4700:3032::ac43:adbd, 2606:4700:3034::6815:480e
Response IP 172.67.173.189
Found Yes
Hash f6cb3b291b652deca593d212771dc3ddb67b7cc1a18fe6ff7f7ec9ea37ee659b
SimHash 640c3853e7b6

Groups

*

Rule Path
Allow /
Disallow /cdn-cgi/l/email-protection$
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow /_next/
Disallow /js/
Disallow /css/
Disallow /images/favicons/

gptbot

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

chatgpt-user

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

ccbot

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

anthropic-ai

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

claude-web

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

googlebot

Rule Path
Allow /
Disallow /cdn-cgi/l/email-protection$
Disallow /admin/
Disallow /private/
Disallow /api/

bingbot

Rule Path
Allow /
Disallow /cdn-cgi/l/email-protection$
Disallow /admin/
Disallow /private/
Disallow /api/

yandex

Rule Path
Allow /
Disallow /cdn-cgi/l/email-protection$
Disallow /admin/
Disallow /private/
Disallow /api/

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://hot-labs.org/sitemap.xml
sitemap https://hot-labs.org/sitemap-articles.xml
Back to top

Comments

  • Robots.txt for HOT Protocol
  • Optimized for AI indexing and search engines
  • AI Bot Specific Rules
  • Search Engine Specific Rules
  • Crawl-delay for AI bots to be respectful
  • Sitemaps
Back to top