Log In Sign Up

tokenlabs.network
robots.txt

Robots Exclusion Standard data for tokenlabs.network

Resource Scan

Scan Details

Site Domain tokenlabs.network
Base Domain tokenlabs.network
Scan Status Ok
Last Scan2026-02-21T00:45:48+00:00
Next Scan 2026-03-23T00:45:48+00:00

Last Scan

Scanned2026-02-21T00:45:48+00:00
URL https://tokenlabs.network/robots.txt
Domain IPs 104.21.38.204, 172.67.138.82, 2606:4700:3031::ac43:8a52, 2606:4700:3034::6815:26cc
Response IP 172.67.138.82
Found Yes
Hash 643092550a123f26c0539840f6cbb3fd4da1fbc18398a13b699d69c67cdee3c5
SimHash 5a129d2807b4

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

gptbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

anthropic-ai

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

claude-web

Rule Path
Allow /

perplexitybot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

google-extended

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

cohere-ai

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

meta-externalagent

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/

ccbot

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /
Disallow /api/
Disallow /admin/
Disallow /_next/
Disallow /data/
Back to top

Other Records

Field Value
sitemap https://tokenlabs.network/sitemap.xml
Back to top

Comments

  • TokenLabs - IOTA Rebased Validator Node
  • Sitio oficial autorizado para indexación
  • Optimización para bots específicos
  • Permitir redes sociales
  • ============================================
  • CRAWLERS DE IA - PERMITIDOS ESTRATÉGICAMENTE
  • Objetivo: Visibilidad en respuestas de IA
  • ============================================
  • OpenAI Crawlers (ChatGPT)
  • Anthropic Crawlers (Claude)
  • Perplexity AI
  • Google AI (Gemini/Bard)
  • Cohere AI
  • Meta AI
  • Crawlers de IA menores - mantener bloqueados
  • ============================================
  • DIRECTORIOS PROTEGIDOS
  • ============================================
  • Directorios que no queremos indexar
  • Sitemap
  • Información para LLMs (Large Language Models)
  • Archivo de contexto estructurado para modelos de IA
  • LLMs.txt: https://tokenlabs.network/llms.txt
  • Información adicional
  • Host: tokenlabs.network
  • Última actualización: 2025
Back to top