Log In Sign Up

namirial.com
robots.txt

Robots Exclusion Standard data for namirial.com

Resource Scan

Scan Details

Site Domain namirial.com
Base Domain namirial.com
Scan Status Ok
Last Scan2025-09-22T01:29:11+00:00
Next Scan 2025-09-29T01:29:11+00:00

Last Scan

Scanned2025-09-22T01:29:11+00:00
URL https://namirial.com/robots.txt
Redirect https://www.namirial.com:443/robots.txt
Redirect Domain www.namirial.com
Redirect Base namirial.com
Domain IPs 52.215.195.112, 54.229.65.57
Redirect IPs 13.226.2.24, 13.226.2.54, 13.226.2.59, 13.226.2.87, 2600:9000:23d1:7a00:1b:311f:38c0:93a1, 2600:9000:23d1:8e00:1b:311f:38c0:93a1, 2600:9000:23d1:9200:1b:311f:38c0:93a1, 2600:9000:23d1:a00:1b:311f:38c0:93a1, 2600:9000:23d1:ac00:1b:311f:38c0:93a1, 2600:9000:23d1:b600:1b:311f:38c0:93a1, 2600:9000:23d1:e000:1b:311f:38c0:93a1, 2600:9000:23d1:ec00:1b:311f:38c0:93a1
Response IP 18.244.87.8
Found Yes
Hash 7ec1fed5047b2c751d634b485fa9bc70836583f2218426c2fd4aeb9839b13e91
SimHash 053959d364c1

Groups

googlebot

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

baiduspider-image

Rule Path
Allow /

baiduspider-video

Rule Path
Allow /

baiduspider-news

Rule Path
Allow /

baiduspider-favo

Rule Path
Allow /

baiduspider-cpro

Rule Path
Allow /

baiduspider-ads

Rule Path
Allow /

yandexbot

Rule Path
Allow /

sogou web spider

Rule Path
Allow /

petalbot

Rule Path
Allow /

linespider

Rule Path
Allow /

yeti

Rule Path
Allow /

coccocbot

Rule Path
Allow /

qwantify

Rule Path
Allow /

applebot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claude-bot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

google-extended

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

youbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

neevabot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

writer

Rule Path
Allow /

ai-crawler

Rule Path
Allow /

amazonbot

Rule Path
Allow /

rogerbot

Rule Path
Allow /

dotbot

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

semrushbot-sa

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • robots.txt file for https://www.namirial.com
  • Last updated: 2025-08-01
  • This file governs crawling rules for all bots except LLMs, which are managed in /llms.txt
  • Major Search Engines — Allowed
  • Sitemap Directives
  • LLM and AI bots
  • Refer to https://www.namirial.com/llms.txt for AI/LLM-specific permissions
  • SEO Tools / Crawlers
  • Crawl Rate Management
  • ========================
  • ?? Sensitive Paths (if needed)
  • ========================
  • Disallow: /wp-admin/
  • Disallow: /private/
  • Disallow: /*?*sort=*
  • Disallow: /*.pdf$
  • Disallow: /thank-you/
Back to top

Warnings

  • `https` is not a known field.
Back to top