Log In Sign Up

navayan.com
robots.txt

Robots Exclusion Standard data for navayan.com

Resource Scan

Scan Details

Site Domain navayan.com
Base Domain navayan.com
Scan Status Ok
Last Scan2025-09-22T20:37:27+00:00
Next Scan 2025-10-22T20:37:27+00:00

Last Scan

Scanned2025-09-22T20:37:27+00:00
URL https://navayan.com/robots.txt
Domain IPs 162.215.240.128
Response IP 162.215.240.128
Found Yes
Hash d8a0cc218aaafac92b69faab787341d5888059dd08fc744c2532a3d897120593
SimHash 29050907b6fa

Groups

*

Rule Path
Disallow /img/nympmpf110414
Disallow /js
Disallow /css
Disallow /cgi-bin
Disallow /inc

*

Rule Path
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.rtf$
Disallow /*.wma$
Disallow /*.wmv$
Disallow /*.avi$
Disallow /*.dat$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.class$
Disallow /*.anw$
Disallow /*.css$
Allow /img
Disallow /img/nympmpf110414

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.navayan.com/sitemap.xml
Back to top

Comments

  • See https://www.robotstxt.org/wc/norobots.html
  • for documentation on how to use the robots.txt file
  • Ban all spiders for the entire site
  • User-agent: *
  • Disallow:
  • Disallow: /
  • Disallow: */*
  • Disallow all files in these directories
  • Disallow robots from parsing individual post feeds and trackbacks
  • Disallow: /feed
  • Disallow: /trackback
  • Disallow: */trackback*
  • Disallow files ending with the following extensions
  • Disallow: /*.ico$
  • Disallow: /*.gif$
  • Disallow: /*.jpg$
  • Disallow: /*.jpeg$
  • Disallow: /*.png$
  • Disallow: /*.flv$
  • Disallow: /*.rm$
  • Disallow: /*.ram$
  • Disallow: /*.htm$
  • Disallow: /*.html$
  • Disallow: /*.php$
  • Disallow: /*.php*
  • Disallow: /*.gz$
  • Disallow: /*.cgi$
  • Disallow: /*.xml$
  • Disallow: /*.htaccess
  • Disallow WayBack archiving site
  • User-agent: ia_archiver
  • Disallow: /
  • Allow google image bot to search all images
  • User-agent: Googlebot-Image
  • allow adsense bot on entire site
  • User-agent: Mediapartners-Google*
  • Disallow: /*?*
  • Allow: /*
Back to top