basketball-reference.com
robots.txt
Robots Exclusion Standard data for basketball-reference.com
Resource Scan
Scan Details
| Site Domain | basketball-reference.com |
| Base Domain | basketball-reference.com |
| Scan Status | Ok |
| Last Scan | 2024-11-09T23:08:26+00:00 |
| Next Scan | 2024-11-16T23:08:26+00:00 |
Last Scan
| Scanned | 2024-11-09T23:08:26+00:00 |
| URL | https://basketball-reference.com/robots.txt |
| Redirect | https://www.basketball-reference.com/robots.txt |
| Redirect Domain | www.basketball-reference.com |
| Redirect Base | basketball-reference.com |
| Domain IPs | 104.18.6.178, 104.18.7.178, 2606:4700::6812:6b2, 2606:4700::6812:7b2 |
| Redirect IPs | 104.18.6.178, 104.18.7.178, 2606:4700::6812:6b2, 2606:4700::6812:7b2 |
| Response IP | 104.18.7.178 |
| Found | Yes |
| Hash | c40e95b326e66e3c0249ebb08472e13da88909d9ea555bca449468bff2e17f2b |
| SimHash | e8014602eea3 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /basketball/ |
| Disallow | /blazers/ |
| Disallow | /dump/ |
| Disallow | /fc/ |
| Disallow | /my/ |
| Disallow | /7103 |
| Disallow | /play-index/*.cgi?* |
| Disallow | /play-index/plus/*.cgi?* |
| Disallow | */gamelog/ |
| Disallow | */splits/ |
| Disallow | */on-off/ |
| Disallow | */lineups/ |
| Disallow | */shooting/ |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Comments