Log In Sign Up

kaaldev.com
robots.txt

Robots Exclusion Standard data for kaaldev.com

Resource Scan

Scan Details

Site Domain kaaldev.com
Base Domain kaaldev.com
Scan Status Ok
Last Scan2025-12-29T02:13:06+00:00
Next Scan 2026-01-05T02:13:06+00:00

Last Scan

Scanned2025-12-29T02:13:06+00:00
URL https://kaaldev.com/robots.txt
Domain IPs 147.93.17.87, 2a02:4780:11:1927:0:376b:3108:3
Response IP 147.93.17.87
Found Yes
Hash 2c546872b85cbe9b6b3636138540f6841ef517612f05cea11b04307bc177780f
SimHash 27d89ed365fa

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /
Disallow /private/
Disallow /*.log$
Disallow /temp/
Disallow /cache/
Allow /styles.css
Allow /script.js
Allow /public/images/
Allow /kaaldev-keywords.json
Allow /fivem-seo-implementation-strategy.md
Allow /ai-discovery.json

googlebot

Rule Path
Allow /kaaldev-keywords.json
Allow /fivem-seo-implementation-strategy.md

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /kaaldev-keywords.json
Allow /fivem-seo-implementation-strategy.md

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://kaaldev.com/sitemap.xml
sitemap https://kaaldev.com/ai-sitemap.xml
sitemap https://kaaldev.com/fivem-sitemap.xml
sitemap https://kaaldev.com/keywords-sitemap.xml
Back to top

Comments

  • Sitemap locations
  • AI Discovery file
  • AI-specific robots.txt: https://kaaldev.com/ai-robots.txt
  • AI discovery metadata: https://kaaldev.com/ai-discovery.json
  • Crawl-delay for respectful crawling
  • Allow all major search engines
  • Block access to sensitive files
  • Allow access to CSS and JS files for proper rendering
  • Specifically allow SEO and keyword files for better indexing
  • FiveM and gaming related content
Back to top