basketballreference.com
robots.txt
Robots Exclusion Standard data for basketballreference.com
Resource Scan
Scan Details
| Site Domain | basketballreference.com |
| Base Domain | basketballreference.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-09-06T10:35:17+00:00 |
| Next Scan | 2025-09-20T10:35:17+00:00 |
Last Successful Scan
| Scanned | 2025-08-21T19:16:00+00:00 |
| URL | https://basketballreference.com/robots.txt |
| Redirect | https://www.basketball-reference.com/robots.txt |
| Redirect Domain | www.basketball-reference.com |
| Redirect Base | basketball-reference.com |
| Domain IPs | 107.23.40.38, 3.208.217.127, 54.235.117.106 |
| Redirect IPs | 104.18.6.178, 104.18.7.178, 2606:4700::6812:6b2, 2606:4700::6812:7b2 |
| Response IP | 104.18.6.178 |
| Found | Yes |
| Hash | c40e95b326e66e3c0249ebb08472e13da88909d9ea555bca449468bff2e17f2b |
| SimHash | e8014602eea3 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /basketball/ |
| Disallow | /blazers/ |
| Disallow | /dump/ |
| Disallow | /fc/ |
| Disallow | /my/ |
| Disallow | /7103 |
| Disallow | /play-index/*.cgi?* |
| Disallow | /play-index/plus/*.cgi?* |
| Disallow | */gamelog/ |
| Disallow | */splits/ |
| Disallow | */on-off/ |
| Disallow | */lineups/ |
| Disallow | */shooting/ |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Comments