Log In Sign Up

millbody.com
robots.txt

Robots Exclusion Standard data for millbody.com

Resource Scan

Scan Details

Site Domain millbody.com
Base Domain millbody.com
Scan Status Ok
Last Scan2025-10-26T16:11:57+00:00
Next Scan 2025-11-25T16:11:57+00:00

Last Scan

Scanned2025-10-26T16:11:57+00:00
URL https://millbody.com/robots.txt
Domain IPs 31.97.25.219
Response IP 31.97.25.219
Found Yes
Hash cc8e6689e4785fd8829a8ecf8d9865f76fc1c2a233f004ac3164fc232b334061
SimHash 2e325c102656

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-content/cache/
Disallow /wp-content/uploads/wpforms/
Disallow /wp-json/wp/v2/users
Disallow /wp-json/oembed/
Disallow /?s=
Disallow /search/
Disallow /author/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-signup.php
Disallow /wp-activate.php
Disallow /cgi-bin/
Disallow /*?attachment_id=
Disallow /*?replytocom=
Disallow /*?p=*&preview=true
Disallow /*?page_id=*&preview=true
Allow /wp-content/uploads/
Allow /wp-content/themes/*/images/
Allow /wp-content/themes/*/css/
Allow /wp-content/themes/*/js/
Allow /wp-content/themes/*/fonts/
Allow /wp-includes/css/
Allow /wp-includes/js/

googlebot

Rule Path
Allow /wp-content/uploads/
Allow /*.css$
Allow /*.js$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Disallow /wp-admin/
Disallow /?s=

bingbot

Rule Path
Allow /wp-content/uploads/
Disallow /wp-admin/
Disallow /?s=

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /wp-content/uploads/
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 2

baiduspider

Rule Path
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Disallow /wp-admin/
Allow /wp-content/uploads/

perplexitybot

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /*.css$
Allow /*.js$
Allow /*.json$
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /?s=

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /*.pdf$
Allow /*.doc$
Allow /*.docx$
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /?s=

Other Records

Field Value
crawl-delay 1

chatgpt-user

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 1

claudebot

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /*.css$
Allow /*.js$
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /?s=

Other Records

Field Value
crawl-delay 1

googleother

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /*.css$
Allow /*.js$
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 1

facebookbot

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 2

microsoft-copilotbot

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /wp-json/
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 1

coherebot

Rule Path
Allow /
Allow /wp-content/uploads/
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 2

youbot

Rule Path
Allow /
Allow /wp-content/uploads/
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

majesticseo

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

openai-*
anthropic-*
google-*

Rule Path
Allow /wp-json/ai-growth-seo/v1/mcp/
Allow /.well-known/mcp
Back to top

Other Records

Field Value
sitemap https://millbody.com/sitemap.xml
Back to top

Comments

  • Robots.txt gerado automaticamente pelo AI Growth Agent (https://aigrowthagent.co) - AI Growth SEO v1.10.10
  • Gerado em: 2025-10-26 13:12:02 -03
  • Website: https://millbody.com
  • ===========================================
  • REGRAS GERAIS - OTIMIZADAS PARA SEO
  • ===========================================
  • Permitir recursos importantes
  • ===========================================
  • REGRAS ESPECÍFICAS POR CRAWLER
  • ===========================================
  • Google - Otimizações específicas
  • Bing - Configurações otimizadas
  • Yahoo - Configurações básicas
  • Yandex - Motor de busca russo
  • Baidu - Motor de busca chinês
  • DuckDuckGo
  • Perplexity AI - Otimizado para indexação LLM
  • OpenAI GPTBot - Otimizado para treinamento GPT
  • ChatGPT User Agent
  • Anthropic Claude - Otimizado para LLM
  • Google Gemini/Bard - Google AI
  • Meta AI - Facebook LLM
  • Microsoft Copilot
  • Cohere AI
  • You.com AI Search
  • Crawlers agressivos - bloquear completamente
  • ===========================================
  • SITEMAPS - DESCOBERTA AUTOMÁTICA
  • ===========================================
  • Sitemap principal
  • ===========================================
  • REGRAS PERSONALIZADAS
  • ===========================================
  • ===========================================
  • AI Agent Discovery Rules (2024 Convention)
  • ===========================================
  • ===========================================
  • MCP (Model Context Protocol) Discovery
  • ===========================================
  • Primary MCP endpoint for AI agent discovery
  • Alternative discovery methods
  • Well-known: https://millbody.com/.well-known/mcp
  • OpenAPI: https://millbody.com/wp-json/ai-growth-seo/v1/mcp/schema
  • ===========================================
  • Agentic Discovery (agentic-robots-txt)
  • ===========================================
  • AI Training Consent
  • Agentic Resource Discovery
  • Rate Limiting Directives
  • Fim do robots.txt gerado pelo AI Growth Agent (https://aigrowthagent.co) - AI Growth SEO
Back to top

Warnings

  • 2 invalid lines.
  • `agent-guide` is not a known field.
  • `agentics-manifest` is not a known field.
  • `ai-training` is not a known field.
  • `attribution` is not a known field.
  • `burst-allowance` is not a known field.
  • `max-requests-per-minute` is not a known field.
  • `mcp-capabilities` is not a known field.
  • `mcp-discovery` is not a known field.
  • `mcp-endpoint` is not a known field.
  • `mcp-schema` is not a known field.
  • `retry-after` is not a known field.
  • `snippet-length` is not a known field.
Back to top