Log In Sign Up

topia.io
robots.txt

Robots Exclusion Standard data for topia.io

Resource Scan

Scan Details

Site Domain topia.io
Base Domain topia.io
Scan Status Ok
Last Scan2025-10-27T12:44:43+00:00
Next Scan 2025-11-26T12:44:43+00:00

Last Scan

Scanned2025-10-27T12:44:43+00:00
URL https://topia.io/robots.txt
Redirect https://topia-website.webflow.io/robots.txt
Redirect Domain topia-website.webflow.io
Redirect Base webflow.io
Domain IPs 104.26.4.61, 104.26.5.61, 172.67.71.171, 2606:4700:20::681a:43d, 2606:4700:20::681a:53d, 2606:4700:20::ac43:47ab
Redirect IPs 104.18.36.248, 172.64.151.8, 2606:4700:440c::ac40:9708, 2a06:98c1:3100::6812:24f8
Response IP 172.64.151.8
Found Yes
Hash 8eefc42e6fa6a1b64d2da9d96b2a90564a3b99721b7df605ac6e078856076481
SimHash 05159cd0ede5

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

googleother

Rule Path
Allow /

bingbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

gptbot

Rule Path
Allow /

amazonbot-extended

Rule Path
Allow /

bytespider

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://topia.io/sitemap.xml
sitemap https://schoolspace.io/sitemap.xml
Back to top

Comments

  • robots.txt for https://topia.io
  • Goal: maximize search and AI retrieval crawlability while disallowing AI model training
  • Default: allow everything for standard web indexing crawlers
  • Core search crawlers
  • OpenAI - search and live fetch
  • Anthropic
  • Perplexity
  • Common Crawl and others
  • Optional sensitive areas
  • Disallow: /admin/
  • Disallow: /account/
  • Disallow: /api/private/
Back to top