Log In Sign Up

krutidevtounicode.net
robots.txt

Robots Exclusion Standard data for krutidevtounicode.net

Resource Scan

Scan Details

Site Domain krutidevtounicode.net
Base Domain krutidevtounicode.net
Scan Status Ok
Last Scan2025-10-06T10:00:55+00:00
Next Scan 2025-10-13T10:00:55+00:00

Last Scan

Scanned2025-10-06T10:00:55+00:00
URL https://krutidevtounicode.net/robots.txt
Domain IPs 104.21.24.4, 172.67.215.65, 2606:4700:3031::6815:1804, 2606:4700:3032::ac43:d741
Response IP 172.67.215.65
Found Yes
Hash 9f11843874897faaeaa3f5dbaf6d79f33020fbf3d0780ad6895c0c868c3ae852
SimHash 7f9c5943e6a4

Groups

*

Rule Path
Allow /
Allow /*.css
Allow /*.js
Allow /*.png
Allow /*.jpg
Allow /*.gif
Allow /*.svg
Allow /*.webp
Allow /sitemap.xml
Disallow /admin/
Disallow /private/
Disallow /temp/
Disallow /.git/
Disallow /node_modules/

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.zip$
Back to top

Other Records

Field Value
sitemap https://krutidevtounicode.net/sitemap.xml
Back to top

Comments

  • Robots.txt for krutidevtounicode.net
  • Optimized for better Google crawling and indexing
  • Allow all important files
  • Disallow sensitive or unnecessary directories
  • Specific crawler instructions
  • Sitemap location
  • Performance optimization - prevent crawling of unnecessary files
Back to top