Log In Sign Up

hellion-initiative.de
robots.txt

Robots Exclusion Standard data for hellion-initiative.de

Resource Scan

Scan Details

Site Domain hellion-initiative.de
Base Domain hellion-initiative.de
Scan Status Ok
Last Scan2025-10-14T04:28:37+00:00
Next Scan 2025-11-13T04:28:37+00:00

Last Scan

Scanned2025-10-14T04:28:37+00:00
URL https://hellion-initiative.de/robots.txt
Domain IPs 104.21.76.124, 172.67.194.249, 2606:4700:3030::6815:4c7c, 2606:4700:3036::ac43:c2f9
Response IP 104.21.76.124
Found Yes
Hash 941d4fc6947e1f5c8c7ffadf8711ee30684074e968483337f97108272ede7831
SimHash 6b1fd042e366

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googleother

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

baiduspider

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

seznambot

Rule Path
Allow /

yeti

Rule Path
Allow /

coccocbot

Rule Path
Allow /

sogou

Rule Path
Allow /

yahoo! slurp

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-searchbot

Rule Path
Allow /

claude-user

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-user

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

anchor browser

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

mistralai-user

Rule Path
Disallow /

novellum ai crawl

Rule Path
Disallow /

proratainc

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /abteilungen/
Allow /api/v2/public/
Disallow /intern/
Disallow /api/auth/
Disallow /api/v2/admin/
Disallow /api/v2/department/
Disallow /api/v2/member/
Disallow /api/v2/user/
Disallow /_next/
Disallow /api/
Disallow /*.json
Disallow /*.xml

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://hellion-initiative.de/sitemap.xml
Back to top

Comments

  • Hellion Initiative - robots.txt
  • https://hellion-initiative.de
  • Erstellt am: 10. Oktober 2025
  • Basierend auf: Cloudflare AI Bot List + Custom Rules
  • ============================================
  • ERLAUBTE BOTS (Standard-Zugriff)
  • ============================================
  • ─────────────────────────────────────────────
  • SUCH MASCHINEN (High Priority)
  • ─────────────────────────────────────────────
  • Google Search
  • Bing Search
  • Yandex Search
  • Baidu Search
  • DuckDuckGo Search
  • Seznam Search (Czech Republic)
  • Naver Search (Korea)
  • Cốc Cốc Search (Vietnam)
  • Sogou Search (China)
  • Yahoo Search
  • ─────────────────────────────────────────────
  • CLAUDE AI (Anthropic) - Entwicklungs-Assistenz
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • ARCHIVIERUNG & SOCIAL MEDIA
  • ─────────────────────────────────────────────
  • Internet Archive (Wayback Machine)
  • Social Media Crawler (für Open Graph / Link Previews)
  • ─────────────────────────────────────────────
  • SEO-CRAWLER (mit Crawl-Delay für Performance)
  • ─────────────────────────────────────────────
  • ============================================
  • BLOCKIERTE AI CRAWLER & TRAINING BOTS
  • (Basierend auf Cloudflare Managed robots.txt)
  • ============================================
  • ─────────────────────────────────────────────
  • OPENAI (ChatGPT & GPT-4)
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • META / FACEBOOK AI
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • GOOGLE AI TRAINING (nicht Google Search!)
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • PERPLEXITY AI
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • APPLE AI
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • AMAZON AI
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • BYTEDANCE (TikTok AI)
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • COMMON CRAWL (AI Training Dataset)
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • ANDERE AI CRAWLER
  • ─────────────────────────────────────────────
  • ─────────────────────────────────────────────
  • AGGRESSIVE / BAD CRAWLERS
  • ─────────────────────────────────────────────
  • ============================================
  • STANDARD-REGEL für alle anderen Bots
  • ============================================
  • SEO-optimierte Bereiche (explizit erlaubt)
  • Geschützte Bereiche (via Middleware geschützt)
  • System-Dateien ausschließen
  • Crawl-Delay für bessere Performance
  • Sitemap-Verweis
Back to top