Log In Sign Up

goat.network
robots.txt

Robots Exclusion Standard data for goat.network

Resource Scan

Scan Details

Site Domain goat.network
Base Domain goat.network
Scan Status Ok
Last Scan2025-10-17T11:46:50+00:00
Next Scan 2025-11-16T11:46:50+00:00

Last Scan

Scanned2025-10-17T11:46:50+00:00
URL https://goat.network/robots.txt
Redirect https://www.goat.network/robots.txt
Redirect Domain www.goat.network
Redirect Base goat.network
Domain IPs 75.2.70.75, 99.83.190.102
Redirect IPs 13.203.125.58, 13.233.175.166, 3.109.243.18
Response IP 13.233.175.166
Found Yes
Hash bc30da79b3c37ccd57a7ba738219d9026a3dba588e1bc5e7444eb4b4d5d83877
SimHash f1d74250c5d3

Groups

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

chatgpt-user/2.0

Rule Path
Allow /

gptbot

Rule Path Comment
Allow / everything else

anthropic-ai

Product Comment
anthropic-ai bulk model training
Rule Path
Allow /

claudebot

Product Comment
claudebot chat citation fetch
Rule Path
Allow /

claude-web

Product Comment
claude-web web-focused crawl
Rule Path
Allow /

perplexitybot

Product Comment
perplexitybot index builder
Rule Path
Allow /

perplexity-user

Product Comment
perplexity-user human-triggered visit
Rule Path
Allow /

google-extended

Rule Path
Allow /

bingbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

facebookbot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

ai2bot

Rule Path
Allow /

ccbot

Rule Path
Allow /

diffbot

Rule Path
Allow /

omgili

Rule Path
Allow /

timpibot

Rule Path
Allow /

youbot

Rule Path
Allow /

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.goat.network/sitemap.xml
Back to top

Comments

  • robots.txt v1.2 by RankBrain (https://www.Badcryptomarketing.com)
  • ----- OPENAI ------
  • Search (shows my webpages as links inside ChatGPT search). NOT used for model training.
  • User-driven browsing from ChatGPT and Custom GPTs. Acts after a human click.
  • Model-training crawler. Opt-out here if I don’t want content in GPT-4o or GPT-5.
  • ------ ANTHROPIC (Claude) ---------
  • ------ PERPLEXITY -------
  • ------ GOOGLE (Gemini) --------
  • ------ MICROSOFT (Bing / Copilot) ------
  • ----- AMAZON --------
  • ------ APPLE -------
  • ------ META -------
  • -------- LINKEDIN --------
  • ------- BYTEDANCE --------
  • ------ DUCKDUCKGO -------
  • ------- COHERE --------
  • ------ ALLEN INSTITUTE / COMMON CRAWL / OTHER RESEARCH --------
  • ------ EMERGING SEARCH START-UPS -------
Back to top