Log In Sign Up

grapevine.is
robots.txt

Robots Exclusion Standard data for grapevine.is

Resource Scan

Scan Details

Site Domain grapevine.is
Base Domain grapevine.is
Scan Status Ok
Last Scan2025-10-04T14:14:10+00:00
Next Scan 2025-10-11T14:14:10+00:00

Last Scan

Scanned2025-10-04T14:14:10+00:00
URL https://grapevine.is/robots.txt
Domain IPs 217.9.143.30
Response IP 217.9.143.30
Found Yes
Hash bcb78047e97deadad1c5dce33e925389a5ef507fad94b700be10c25d0c3e482c
SimHash 7010e991e590

Groups

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

bard

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

researchbot

Rule Path
Disallow /

traversabot

Rule Path
Disallow /

webcrawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

baiduspider

Rule Path
Disallow

yandexbot

Rule Path
Disallow

facebookexternalhit

Rule Path
Disallow

twitterbot

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

*

Rule Path
Disallow
Back to top

Comments

  • Block major AI crawlers and training bots
  • Block common research and scraping bots
  • Allow legitimate search engines
  • Default rule for all other bots
  • Sitemap location (optional - update with your actual sitemap URL)
  • Sitemap: https://hagi.is/sitemap.xml
Back to top