Log In Sign Up

ledsreact.com
robots.txt

Robots Exclusion Standard data for ledsreact.com

Resource Scan

Scan Details

Site Domain ledsreact.com
Base Domain ledsreact.com
Scan Status Ok
Last Scan2025-09-24T01:31:39+00:00
Next Scan 2025-10-24T01:31:39+00:00

Last Scan

Scanned2025-09-24T01:31:39+00:00
URL https://ledsreact.com/robots.txt
Domain IPs 3.165.75.118, 3.165.75.63, 3.165.75.70, 3.165.75.89
Response IP 3.165.75.118
Found Yes
Hash 2ccd53eaa9403fa33ba1d85b01b422440d5636f610086ce951127ac9ea47806e
SimHash 29189ff3c4e8

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /_next/
Disallow /.next/
Disallow /admin/
Disallow /dashboard/
Disallow /private/
Disallow /temp/
Disallow /tmp/
Disallow /*.json$
Disallow /*.log$
Disallow /*test*
Disallow /*dev*
Disallow /config/
Disallow /.env*
Disallow /.git/
Disallow /node_modules/
Disallow /*?*utm_*
Disallow /*?*ref=*
Disallow /*?*fbclid=*
Disallow /*?*gclid=*
Allow /img/
Allow /video/
Allow /whitepapers/
Allow /clients/
Allow /product/
Allow /*.css$
Allow /*.js$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.webp$
Allow /*.svg$
Allow /*.ico$
Allow /*.pdf$

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://ledsreact.com/sitemap.xml
Back to top

Comments

  • Robots.txt for Ledsreact - Sports Performance Testing Technology
  • Website: https://ledsreact.com
  • Generated following SEO best practices
  • Default rule for all robots
  • Allow crawling of all public content
  • Disallow crawling of technical and admin areas
  • Disallow crawling of development and testing files
  • Disallow crawling of sensitive configuration files
  • Disallow crawling of duplicate content with URL parameters
  • Allow crawling of important static assets
  • Special rules for major search engines
  • Block competitive analysis and aggressive scraping bots
  • Note: These are legitimate tools but blocked to limit competitive intelligence
  • Currently allowing this one, for ourselves
  • User-agent: SemrushBot
  • Disallow: /
  • Block known aggressive scrapers
  • Sitemap references
  • Host declaration (helps with international targeting)
Back to top

Warnings

  • `host` is not a known field.
Back to top