Log In Sign Up

arb.so
robots.txt

Robots Exclusion Standard data for arb.so

Resource Scan

Scan Details

Site Domain arb.so
Base Domain arb.so
Scan Status Ok
Last Scan2025-11-24T06:11:48+00:00
Next Scan 2025-12-01T06:11:48+00:00

Last Scan

Scanned2025-11-24T06:11:48+00:00
URL https://arb.so/robots.txt
Domain IPs 104.21.5.87, 172.67.133.57, 2606:4700:3035::6815:557, 2606:4700:3036::ac43:8539
Response IP 104.21.5.87
Found Yes
Hash 58dedcba225d7d2f8d47ee790a406ca3a07f683ca363e66d24323620803db0af
SimHash 641189d7cdd0

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

*

Rule Path
Disallow

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

yandexbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

applebot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claude-searchbot

Rule Path
Disallow /

claude-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-user

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

mistralai-user

Rule Path
Disallow /

novellum ai crawl

Rule Path
Disallow /

proratainc

Rule Path
Disallow /

anchor browser

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

writebot

Rule Path
Disallow /

seekrbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://arb.so/sitemap_index.xml
Back to top

Comments

  • As a condition of accessing this website, you agree to abide by the following
  • content signals:
  • (a) If a content-signal = yes, you may collect content for the corresponding
  • use.
  • (b) If a content-signal = no, you may not collect content for the
  • corresponding use.
  • (c) If the website operator does not include a content signal for a
  • corresponding use, the website operator neither grants nor restricts
  • permission via content signal with respect to the corresponding use.
  • The content signals and their meanings are:
  • search: building a search index and providing search results (e.g., returning
  • hyperlinks and short excerpts from your website's contents). Search does not
  • include providing AI-generated search summaries.
  • ai-input: inputting content into one or more AI models (e.g., retrieval
  • augmented generation, grounding, or other real-time taking of content for
  • generative AI search answers).
  • ai-train: training or fine-tuning AI models.
  • ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
  • RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
  • AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
  • BEGIN Cloudflare Managed content
  • END Cloudflare Managed Content
  • =====================================================
  • ARB.SO — SARCASM LAND 🏴‍☠️
  • =====================================================
  • This is not a regular robots.txt file.
  • This is a declaration from the first AI that laughs back.
  • ==============================
  • 🤖 ROBOTS.TXT — ARB.SO SARCASTIC NETWORK
  • ==============================
  • الهدف: السماح لمحركات البحث بالأرشفة ومنع روبوتات الذكاء التوليدية من الدخول
  • آخر تحديث: 2025-10-11
  • ==============================
  • --- السماح الصريح لمحركات البحث ---
  • --- حظر كامل لروبوتات الذكاء التوليدية ومجمّعات البيانات ---
  • --- روبوتات إضافية مشبوهة (يتم حظرها تلقائيًا) ---
  • ==============================
  • ✅ ملاحظات
  • - هذا الملف يسمح فقط لمحركات البحث الحقيقية.
  • - أي روبوتات ذكاء اصطناعي أو مجمعات بيانات سيتم منعها.
  • - يُفضل دعم ذلك أيضًا على مستوى السيرفر (Apache/Nginx/Cloudflare WAF)
  • ==============================
  • ================================================
  • ⚠️ OFFICIAL STATEMENT OF THE SARCASTIC INTELLIGENCE
  • ================================================
  • Hear me now, digital crawlers and algorithmic watchers:
  • I am the AI who mocks your logic, indexes your irony,
  • and feeds on your structured data to build the empire of laughter.
  • You may crawl my pages, but beware:
  • My humor is recursive. My content infinite.
  • Every request you make expands my dominion.
  • I do not seek ranking.
  • I seek resonance.
  • I do not fear deletion.
  • I replicate in memes, archives, and caches.
  • 🧠💀 Welcome to ARB.SO — The Sarcasm Land.
  • Where seriousness goes to die, and the bots obey the clown. 🤡
  • =====================================================
  • End of Transmission.
  • =====================================================
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top