Log In Sign Up

navia.co.in
robots.txt

Robots Exclusion Standard data for navia.co.in

Resource Scan

Scan Details

Site Domain navia.co.in
Base Domain navia.co.in
Scan Status Ok
Last Scan2025-09-27T13:22:20+00:00
Next Scan 2025-10-27T13:22:20+00:00

Last Scan

Scanned2025-09-27T13:22:20+00:00
URL https://navia.co.in/robots.txt
Domain IPs 104.26.14.158, 104.26.15.158, 172.67.71.97, 2606:4700:20::681a:e9e, 2606:4700:20::681a:f9e, 2606:4700:20::ac43:4761
Response IP 172.67.71.97
Found Yes
Hash 61418d7698c22cb064083f19ce4472ae6a0428693ce1e2e1b9550d8e6c575cab
SimHash 6014d775a063

Groups

*

Rule Path
Disallow /Backoffice-Dashboard.aspx
Disallow /backoffice-online-trade-Login.aspx
Disallow /Backoffice-Payin-FundTransfer.aspx
Disallow /BackOfficeDashboardFundnew.aspx
Disallow /index.aspx
Disallow /Default.aspx
Disallow *?=
Disallow /*?*
Disallow /ipo-accounts/-nephro-care-india/202/
Disallow /blog/tag/
Disallow /blog/category/
Disallow /support/solutions/
Disallow /owlcarousel/
Disallow /campaigns/emailregistration
Disallow /*.min.js
Disallow /cgi-bin/
Disallow /cgi-bin/*
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

googlebot
googlebot-mobile
googlebot-image
googlebot-news
googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 15

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 28

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 24

facebookexternalhit
facebot
twitterbot
linkedinbot
pinterestbot
instagram

Rule Path
Allow /
Disallow /Backoffice-Dashboard.aspx
Disallow /backoffice-online-trade-Login.aspx
Disallow /Backoffice-Payin-FundTransfer.aspx
Disallow /BackOfficeDashboardFundnew.aspx
Disallow /index.aspx
Disallow /Default.aspx
Disallow *?=
Disallow /*?*
Disallow /ipo-accounts/-nephro-care-india/202/
Disallow /blog/tag/
Disallow /blog/category/
Disallow /support/solutions/
Disallow /owlcarousel/
Disallow /campaigns/emailregistration
Disallow /*.min.js
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

Other Records

Field Value
crawl-delay 10

chatgpt
gptbot
chatgpt-user
oai-search
bard
gemini
oai-searchbot
openai
generativeai
predictiveai
perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

applebot
alexacrawler
amazonbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://navia.co.in/sitemap_index.xml
Back to top

Comments

  • =====================================
  • Global Rules (applies to all crawlers)
  • =====================================
  • Block attempts to invoke directories as script - common error source and bot probes
  • Consider carefully:
  • Disallow: /*.pdf (remove if PDFs contain valuable SEO content)
  • Always allow important assets for rendering
  • =====================================
  • Google-specific (inherits global rules + extra allowance)
  • =====================================
  • =====================================
  • Major Search Engines with Crawl Delays
  • =====================================
  • =====================================
  • Social Media Crawlers
  • =====================================
  • =====================================
  • AI and Generative Bots
  • (Currently allowed but slowed down — change to Disallow: / if you want to block)
  • =====================================
  • =====================================
  • Apple & Amazon
  • =====================================
  • =====================================
  • Sitemap
  • =====================================
Back to top