Log In Sign Up

zkverify.io
robots.txt

Robots Exclusion Standard data for zkverify.io

Resource Scan

Scan Details

Site Domain zkverify.io
Base Domain zkverify.io
Scan Status Ok
Last Scan2025-11-27T23:41:47+00:00
Next Scan 2025-12-27T23:41:47+00:00

Last Scan

Scanned2025-11-27T23:41:47+00:00
URL https://zkverify.io/robots.txt
Domain IPs 104.18.18.184, 104.18.19.184, 2606:4700::6812:12b8, 2606:4700::6812:13b8
Response IP 104.18.18.184
Found Yes
Hash 080b63440f3f34cab2079c38ba4aadeca0c2f9ff64f577b056b1d03f62dc60fe
SimHash 2b1b5b826958

Groups

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

*

Rule Path
Disallow

gptbot
oai-searchbot
chatgpt-user

Rule Path
Disallow

claudebot
claude-web
claude-user
claude-searchbot

Rule Path
Disallow

google-extended

Rule Path
Disallow

applebot
applebot-extended

Rule Path
Disallow

perplexitybot
perplexity-user

Rule Path
Disallow

grokcrawler
xai-grok

Rule Path
Disallow

mistralai-user

Rule Path
Disallow

deepseekbot

Rule Path
Disallow

phindbot

Rule Path
Disallow

youbot

Rule Path
Disallow

meta-externalagent
meta-externalfetcher
facebookexternalhit
facebot

Rule Path
Disallow

redditbot

Rule Path
Disallow

ccbot

Rule Path
Disallow

amazonbot

Rule Path
Disallow

bingbot
slurp
duckduckbot

Rule Path
Disallow
Back to top

Comments

  • robots.txt — allow all crawlers except Baidu & Yandex
  • Sitemap will be updated with actual domain
  • --- Block Baidu & Yandex (explicitly) ---
  • --- Catch-all: allow everything else ---
  • --- Explicit OK for AI/LLM + social/link-preview crawlers ---
  • OpenAI (training + search + on-demand fetch)
  • Anthropic (Claude)
  • Google (AI data usage control)
  • Apple
  • Perplexity
  • xAI / Grok (no official UA published; kept as placeholders)
  • Mistral (on-demand fetcher)
  • DeepSeek
  • Phind
  • You.com
  • Meta (Facebook/Instagram/Messenger + Meta AI)
  • Reddit link preview
  • Common Crawl
  • Amazon
  • Microsoft/Bing, Yahoo, DuckDuckGo (still allowed)
Back to top