Log In Sign Up

horseracing.com
robots.txt

Robots Exclusion Standard data for horseracing.com

Resource Scan

Scan Details

Site Domain horseracing.com
Base Domain horseracing.com
Scan Status Ok
Last Scan2026-01-29T01:18:29+00:00
Next Scan 2026-02-05T01:18:29+00:00

Last Scan

Scanned2026-01-29T01:18:29+00:00
URL https://horseracing.com/robots.txt
Redirect https://www.horseracing.com/robots.txt
Redirect Domain www.horseracing.com
Redirect Base horseracing.com
Domain IPs 104.21.86.5, 172.67.213.81, 2606:4700:3030::6815:5605, 2606:4700:3030::ac43:d551
Redirect IPs 104.21.86.5, 172.67.213.81, 2606:4700:3030::6815:5605, 2606:4700:3030::ac43:d551
Response IP 172.67.213.81
Found Yes
Hash a3288bc1f9ca8b7beacc8903260d0916e50fd65d10009f405c64c7dc7be5343a
SimHash 2900d31602a3

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Disallow /cgi-bin/
Allow /
Allow /*.css$
Allow /*.js$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.png$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Allow /*.pdf$
Back to top

Other Records

Field Value
sitemap https://www.horseracing.com/sitemap_index.xml
Back to top

Comments

  • --- Block typical admin/system directories ---
  • --- Allow everything else — public content, forums, search, feeds, pagination, query URLs, etc. ---
  • --- Allow static resources — essential for proper rendering and SEO ---
  • --- Sitemap(s) — helps crawlers discover all pages efficiently ---
Back to top