sportsreference.com
robots.txt
Robots Exclusion Standard data for sportsreference.com
Resource Scan
Scan Details
| Site Domain | sportsreference.com |
| Base Domain | sportsreference.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-12-01T13:11:28+00:00 |
| Next Scan | 2026-01-30T13:11:28+00:00 |
Last Successful Scan
| Scanned | 2025-09-25T16:05:19+00:00 |
| URL | https://sportsreference.com/robots.txt |
| Redirect | https://www.sports-reference.com/robots.txt |
| Redirect Domain | www.sports-reference.com |
| Redirect Base | sports-reference.com |
| Domain IPs | 3.214.133.4, 3.220.145.57, 54.144.168.231 |
| Redirect IPs | 104.18.18.43, 104.18.19.43, 2606:4700::6812:122b, 2606:4700::6812:132b |
| Response IP | 104.18.18.43 |
| Found | Yes |
| Hash | f281b66a4d963fac44c2f807726194b81d08ea735faa5302630da7c3e9a855c1 |
| SimHash | 783240738b13 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /bb/ |
| Disallow | /olympics/athlete_search.cgi |
| Disallow | /cfb/search.cgi |
| Disallow | /cbb/search.cgi |
| Disallow | /cbb/boxscores/index.cgi?* |
| Disallow | /cbb/players/*/splits/*/?* |
| Disallow | /cbb/players/*/splits/*?* |
| Disallow | /cbb/players/*/splits/* |
| Disallow | /cbb/players/*/splits* |
| Disallow | /cfb/boxscores/index.cgi?* |
| Disallow | /cfb/players/*/splits/*/?* |
| Disallow | /cfb/players/*/splits/*?* |
| Disallow | /cfb/players/*/splits/* |
| Disallow | /cfb/players/*/splits* |
| Disallow | /cfb/schools/*/*/splits/*/?* |
| Disallow | /cfb/schools/*/*/splits/*?* |
| Disallow | /cfb/schools/*/*/splits/* |
| Disallow | /cfb/schools/*/*/splits* |
| Disallow | /cfb/req/ |
| Disallow | /cfb/short/ |
| Disallow | /cfb/nocdn/ |
| Disallow | /cbb/req/ |
| Disallow | /cbb/short/ |
| Disallow | /cbb/nocdn/ |