Log In Sign Up

khogendrarupini.com
robots.txt

Robots Exclusion Standard data for khogendrarupini.com

Resource Scan

Scan Details

Site Domain khogendrarupini.com
Base Domain khogendrarupini.com
Scan Status Ok
Last Scan2026-03-03T08:47:53+00:00
Next Scan 2026-03-10T08:47:53+00:00

Last Scan

Scanned2026-03-03T08:47:53+00:00
URL https://khogendrarupini.com/robots.txt
Domain IPs 2a02:4780:84:8fb:2ecb:a881:c16a:8ed2, 2a02:4780:85:2cc5:c0a1:3176:6151:a236, 77.37.66.18, 93.127.187.60
Response IP 77.37.75.116
Found Yes
Hash 1eb7a13542021164410f6e66c219762cf1bb88503ddc81c1e6c3140b01c3b350
SimHash 61464f936f86

Groups

*

Rule Path
Disallow
Disallow /admin/
Disallow /private/
Disallow /internal/
Disallow /temp/
Disallow /backup/
Disallow /*.sql$
Disallow /*.json$
Disallow /*.zip$
Disallow /*.log$
Disallow /*?*

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://khogendrarupini.com/sitemap.xml
Back to top

Comments

  • robots.txt for khogendrarupini.com
  • Created on 2025-01-04
  • Purpose: Define crawling rules for search engines
  • Allow all user-agents full access to the site
  • Exclude sensitive or non-public directories (if they exist)
  • Block specific file types from being crawled (if necessary)
  • Exclude query strings to avoid duplicate content
  • Crawl-delay for specific bots (optional, for reducing server load)
  • Sitemap for better indexing
Back to top