Log In Sign Up

blog.provesrc.com
robots.txt

Robots Exclusion Standard data for blog.provesrc.com

Resource Scan

Scan Details

Site Domain blog.provesrc.com
Base Domain provesrc.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-09T00:22:39+00:00
Next Scan 2025-12-09T00:22:39+00:00

Last Successful Scan

Scanned2025-09-17T04:24:43+00:00
URL https://blog.provesrc.com/robots.txt
Domain IPs 64.176.167.197
Response IP 64.176.167.197
Found Yes
Hash 14383b5d97835569d1bad1daacae267a669e119879b73888a3282ade70320523
SimHash f1d74250c5b2

Groups

*

Rule Path
Disallow /wp-json/
Disallow /?rest_route=
Disallow /verified/
Disallow /domain/
Disallow /lp/
Disallow /wp-admin/

trendictionbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Allow /

chatgpt-user
chatgpt-user/2.0

Rule Path
Allow /

gptbot

Rule Path Comment
Allow / everything else

anthropic-ai

Product Comment
anthropic-ai bulk model training
Rule Path
Allow /

claudebot
claude-web

Product Comment
claudebot chat citation fetch
claude-web web-focused crawl
Rule Path
Allow /

perplexitybot

Product Comment
perplexitybot index builder
Rule Path
Allow /

perplexity-user

Product Comment
perplexity-user human-triggered visit
Rule Path
Allow /

google-extended

Rule Path
Allow /

bingbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot
applebot-extended

Rule Path
Allow /

facebookbot
meta-externalagent

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

ai2bot
ccbot
diffbot
omgili

Rule Path
Allow /

timpibot
youbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://provesrc.com/sitemap_index.xml
sitemap https://provesrc.com/blog/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
  • ——— OPENAI ———
  • Search (shows my webpages as links inside ChatGPT search). NOT used for model training.
  • User-driven browsing from ChatGPT and Custom GPTs. Acts after a human click.
  • Model-training crawler. Opt-out here if I don’t want content in GPT-4o or GPT-5.
  • ——— ANTHROPIC (Claude) ———
  • ——— PERPLEXITY ———
  • ——— GOOGLE (Gemini) ———
  • ——— MICROSOFT (Bing / Copilot) ———
  • ——— AMAZON ———
  • ——— APPLE ———
  • ——— META ———
  • ——— LINKEDIN ———
  • ——— BYTEDANCE ———
  • ——— DUCKDUCKGO ———
  • ——— COHERE ———
  • ——— ALLEN INSTITUTE / COMMON CRAWL / OTHER RESEARCH ———
  • ——— EMERGING SEARCH START-UPS ———
Back to top