Log In Sign Up

jayleekr.github.io
robots.txt

Robots Exclusion Standard data for jayleekr.github.io

Resource Scan

Scan Details

Site Domain jayleekr.github.io
Base Domain jayleekr.github.io
Scan Status Ok
Last Scan2025-10-12T14:56:34+00:00
Next Scan 2025-10-26T14:56:34+00:00

Last Scan

Scanned2025-10-12T14:56:34+00:00
URL https://jayleekr.github.io/robots.txt
Domain IPs 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153, 2606:50c0:8000::153, 2606:50c0:8001::153, 2606:50c0:8002::153, 2606:50c0:8003::153
Response IP 185.199.109.153
Found Yes
Hash cb4a8a5bc1373790b7f41bf6ea7bc8bace6811ba8f422b5910af9bdb7aa77ebd
SimHash 2e4e1de325e6

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /
Disallow /test-results/
Disallow /playwright-report/
Disallow /_astro/
Disallow /backup/
Disallow /node_modules/
Disallow /src/
Disallow /dist/
Disallow /.git/
Disallow /.vscode/
Disallow /temp/
Disallow /tmp/
Disallow /*?search=*
Disallow /*%26search%3D*
Allow /*.css
Allow /*.js
Allow /*.png
Allow /*.jpg
Allow /*.jpeg
Allow /*.gif
Allow /*.svg
Allow /*.webp
Allow /*.ico
Allow /*.woff
Allow /*.woff2
Allow /rss-styles.xsl

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://jayleekr.github.io/sitemap-index.xml
Back to top

Comments

  • Robots.txt for jayleekr.github.io - Optimized for SEO
  • Last updated: 2025-01-22
  • Allow all major search engines with specific rules
  • Social media crawlers
  • Specific crawling rules - Disallow development and build files
  • Disallow search result pages to avoid duplicate content
  • Allow important assets for better rendering
  • Sitemaps
  • RSS Feeds for content discovery
  • Main feed: https://jayleekr.github.io/rss.xml
  • English feed: https://jayleekr.github.io/rss/en.xml
  • General crawl delay for respectful crawling
Back to top