Log In Sign Up

manamurah.com
robots.txt

Robots Exclusion Standard data for manamurah.com

Resource Scan

Scan Details

Site Domain manamurah.com
Base Domain manamurah.com
Scan Status Ok
Last Scan2025-10-29T13:26:30+00:00
Next Scan 2025-11-05T13:26:30+00:00

Last Scan

Scanned2025-10-29T13:26:30+00:00
URL https://manamurah.com/robots.txt
Domain IPs 104.21.85.237, 172.67.212.27, 2606:4700:3035::6815:55ed, 2606:4700:3035::ac43:d41b
Response IP 172.67.212.27
Found Yes
Hash e9a3f7e744573d5fdb625ab2e10c46b1196256d4bbc90fb78877cc27751ead09
SimHash 52019d53efc0

Groups

*

Rule Path
Allow /
Disallow /api/private
Disallow /req/private
Disallow /_app

Other Records

Field Value
crawl-delay 2

googlebot

Rule Path
Allow /
Allow /search
Disallow /api/private
Disallow /req/private

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /
Allow /search
Disallow /api/private
Disallow /req/private

Other Records

Field Value
crawl-delay 0

gptbot

Rule Path
Allow /
Allow /search
Allow /api/v1/

Other Records

Field Value
crawl-delay 1

chatgpt-user

Rule Path
Allow /
Allow /search

Other Records

Field Value
crawl-delay 1

ccbot

Rule Path
Allow /
Allow /search

Other Records

Field Value
crawl-delay 2

anthropic-ai

Rule Path
Allow /
Allow /search

Other Records

Field Value
crawl-delay 1

claude-web

Rule Path
Allow /
Allow /search

Other Records

Field Value
crawl-delay 1

scrapy
screaming frog seo spider
ahrefsbot
semrushbot
mj12bot

Rule Path
Disallow /
Allow /search?*
Back to top

Other Records

Field Value
sitemap https://manamurah.com/sitemap.xml
Back to top

Comments

  • Robots.txt for ManaMurah.com
  • Allow all legitimate bots to crawl the site
  • Search engines
  • AI Bots and LLM crawlers - Allow access for AI understanding
  • Block aggressive scrapers
  • Sitemap location
  • Allow search page with parameters
Back to top