Log In Sign Up

islamicurdubooks.com
robots.txt

Robots Exclusion Standard data for islamicurdubooks.com

Resource Scan

Scan Details

Site Domain islamicurdubooks.com
Base Domain islamicurdubooks.com
Scan Status Ok
Last Scan2026-03-30T07:25:59+00:00
Next Scan 2026-04-29T07:25:59+00:00

Last Scan

Scanned2026-03-30T07:25:59+00:00
URL https://islamicurdubooks.com/robots.txt
Domain IPs 191.101.228.16, 2a02:4780:84:ac57:8c60:7d2a:9617:3d77, 2a02:4780:84:b001:cc7b:cc30:26c8:2fcf, 93.127.196.170
Response IP 77.37.48.46
Found Yes
Hash 231e4fa64e6165f439fa4d7d23a950949f4669695b94adc38eb8552e2ab4ba07
SimHash 601c4e326781

Groups

*

Rule Path
Disallow /admin/
Disallow /testing/
Disallow /private/
Disallow /tmp/
Disallow /backup/
Disallow /api/
Allow /
Allow /quran/
Allow /hadith/

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

similarwebbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

exabot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

downthemall

Rule Path
Disallow /

downthemall!

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://islamicurdubooks.com/sitemaps/sitemap-index.xml
Back to top

Comments

  • -------------------------------
  • Global Rules
  • -------------------------------
  • Allow main content
  • -------------------------------
  • Allow Major Search Engines
  • -------------------------------
  • -------------------------------
  • Archive & Research Bots
  • -------------------------------
  • -------------------------------
  • Analytics / SimilarWeb
  • -------------------------------
  • -------------------------------
  • Block SEO Scraper Bots
  • -------------------------------
  • -------------------------------
  • Block aggressive crawlers
  • -------------------------------
  • -------------------------------
  • Download / Scraping Tools
  • -------------------------------
  • -------------------------------
  • Sitemap
  • -------------------------------
Back to top