memoro.ai
robots.txt

Robots Exclusion Standard data for memoro.ai

Resource Scan

Scan Details

Site Domain memoro.ai
Base Domain memoro.ai
Scan Status Ok
Last Scan2025-11-25T02:50:10+00:00
Next Scan 2025-12-25T02:50:10+00:00

Last Scan

Scanned2025-11-25T02:50:10+00:00
URL https://memoro.ai/robots.txt
Domain IPs 104.21.0.250, 172.67.151.130, 2606:4700:3033::6815:fa, 2606:4700:3034::ac43:9782
Response IP 172.67.151.130
Found Yes
Hash b758e1a084bea55ff0108e9ce1f6a8ab97482cb90c84e243cde8a74d79770dfc
SimHash 6425de50ec82

Groups

*

Rule Path Comment
Allow / -
Allow /en/ -
Allow /features -
Allow /prices -
Allow /download -
Allow /blog -
Allow /team -
Allow /testimonials -
Disallow /api/ -
Disallow /_astro/ -
Disallow /admin/ -
Disallow /private/ -
Disallow /404 -
Disallow /*.json$ -
Disallow /*.xml$ -
Disallow /search -
Disallow /de/ -
Disallow /*?* -
Disallow /* *

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://memoro.ai/sitemap-index.xml

Comments

  • Robots.txt for Memoro.ai
  • https://www.memoro.ai
  • Last updated: 2025-01-04
  • Default rules for all bots
  • Block technical and admin paths
  • Block old German URLs (now redirected)
  • Block duplicate content patterns
  • Crawl-delay in seconds (removed for better crawling)
  • Crawl-delay: 1
  • Sitemap location
  • Common search engine bots
  • Block bad bots