Log In Sign Up

magicbookshelf.ai
robots.txt

Robots Exclusion Standard data for magicbookshelf.ai

Resource Scan

Scan Details

Site Domain magicbookshelf.ai
Base Domain magicbookshelf.ai
Scan Status Ok
Last Scan2025-09-24T16:53:48+00:00
Next Scan 2025-10-24T16:53:48+00:00

Last Scan

Scanned2025-09-24T16:53:48+00:00
URL https://magicbookshelf.ai/robots.txt
Domain IPs 104.21.58.14, 172.67.197.50, 2606:4700:3033::6815:3a0e, 2606:4700:3035::ac43:c532
Response IP 172.67.197.50
Found Yes
Hash 919062896cd9c051602c22b7394323a8364ba86676b9db8ac21807523b7797d5
SimHash 44274ad2e555

Groups

*

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /en/
Allow /es/
Allow /fr/
Allow /de/
Allow /it/
Allow /pt/
Allow /pt-BR/
Allow /nl/
Allow /pl/
Allow /cs/
Allow /hu/
Allow /ro/
Allow /ru/
Allow /uk/
Allow /ar/
Allow /he/
Allow /tr/
Allow /hi/
Allow /th/
Allow /vi/
Allow /id/
Allow /ms/
Allow /ko/
Allow /ja/
Allow /zh-CN/
Allow /sv/
Allow /no/
Allow /da/
Allow /fi/
Allow /el/
Allow /fa/
Allow /assets/
Allow /*.css
Allow /*.js
Allow /*.png
Allow /*.jpg
Allow /*.jpeg
Allow /*.gif
Allow /*.svg
Allow /*.webp
Allow /*.ico
Disallow /admin/
Disallow /admin/*
Disallow /api/
Disallow /api/*
Disallow /rails/
Disallow /rails/*
Disallow /purchases/
Disallow /purchases/*
Disallow /up
Disallow /*/account-deletion
Disallow /account-deletion
Disallow /400
Disallow /404
Disallow /406
Disallow /422
Disallow /500
Disallow /tmp/
Disallow /cache/
Disallow /.well-known/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://magicbookshelf.ai/sitemap.xml
Back to top

Comments

  • As a condition of accessing this website, you agree to abide by the
  • following content-signals:
  • (a) If a content-signal = yes, you may collect content for the
  • corresponding use.
  • (b) If a content-signal = no, you may not collect content for the
  • corresponding use.
  • (c) If the website operator does not include a content signal for a
  • corresponding use, the website operator neither grants nor restricts
  • permission via content signal with respect to the corresponding use.
  • The content signals and their meanings are:
  • search: building a search index and providing search results (e.g., returning
  • hyperlinks and short excerpts from your website's contents). Search
  • does not include providing AI-generated search summaries.
  • ai-input: inputting content into one or more AI models (e.g., retrieval
  • augmented generation, grounding, or other real-time taking of
  • content for generative AI search answers).
  • ai-train: training or fine-tuning AI models.
  • ANY RESTRICTIONS EXPRESSED VIA CONTENT-SIGNALS ARE EXPRESS RESERVATIONS OF
  • RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
  • AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
  • BEGIN Cloudflare Managed content
  • END Cloudflare Managed Content
  • Robots.txt for Endless Adventure
  • See https://www.robotstxt.org/robotstxt.html for documentation
  • Allow all search engines to crawl public content
  • Allow access to main public pages and localized content
  • Allow access to static assets for proper page rendering
  • Block admin areas
  • Block API endpoints (not meant for search indexing)
  • Block Rails-specific paths
  • Block transaction and purchase endpoints
  • Block health check endpoint
  • Block account deletion page (sensitive/private)
  • Block error pages (they shouldn't be indexed)
  • Block any temporary or cache files
  • Crawl delay (be respectful to server resources)
  • Sitemap location (update this URL to your actual domain)
  • Specific rules for major search engines
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top