Log In Sign Up

starkinsider.com
robots.txt

Robots Exclusion Standard data for starkinsider.com

Resource Scan

Scan Details

Site Domain starkinsider.com
Base Domain starkinsider.com
Scan Status Ok
Last Scan2026-03-28T17:26:35+00:00
Next Scan 2026-04-04T17:26:35+00:00

Last Scan

Scanned2026-03-28T17:26:35+00:00
URL https://starkinsider.com/robots.txt
Domain IPs 172.66.40.205, 172.66.43.51, 2606:4700:3108::ac42:28cd, 2606:4700:3108::ac42:2b33
Response IP 172.66.40.205
Found Yes
Hash 08384e26b3564eeea4a0112d3d7e85668078b93197f9c102d9d5db44e07dae5d
SimHash 713c5171e79f

Groups

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

applebot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

ahrefssiteaudit

Rule Path
Allow /

semrushbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

dotbot

Rule Path
Allow /

ccbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

webziobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-includes/
Disallow /*/attachment/
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Back to top

Other Records

Field Value
sitemap https://www.starkinsider.com/wp-sitemap.xml
sitemap https://www.starkinsider.com/video-sitemap.xml
sitemap https://www.starkinsider.com/news-sitemap.xml
Back to top

Comments

  • /robots.txt
  • StarkInsider.com AI Bot Policy
  • Philosophy: Welcome AI bots that provide attribution & citations
  • Block aggressive training scrapers that don't give back
  • ===== AI Usage =====
  • Crawl here to learn more about Stark Insider
  • AI-Attribution: required
  • AI-Snippet-Length: 200
  • AI-Index: /ai-facts.json
  • ===== BENEFICIAL AI BOTS - ALLOWED =====
  • These bots provide citations, search results, or attribution
  • OpenAI - Search & Citation Bots (beneficial)
  • Anthropic (Claude) - Citation & Web Access
  • Perplexity - Search Engine with Citations
  • ===== TRADITIONAL SEARCH ENGINES - ALLOWED =====
  • Essential for SEO and organic traffic
  • ===== SEO & RESEARCH TOOLS - ALLOWED =====
  • Legitimate business tools
  • ===== AGGRESSIVE AI SCRAPERS - BLOCKED =====
  • Training bots that don't provide attribution or value back
  • Common Crawl - Used by many AI companies for training
  • ByteDance/TikTok AI scrapers
  • Commercial scrapers that sell data
  • Other aggressive scrapers
  • ===== WORDPRESS PROTECTION =====
  • Standard WordPress security
  • ===== SITEMAPS =====
  • Silencio
Back to top