baseball-reference.com
robots.txt
Robots Exclusion Standard data for baseball-reference.com
Resource Scan
Scan Details
| Site Domain | baseball-reference.com |
| Base Domain | baseball-reference.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-09-09T04:56:20+00:00 |
| Next Scan | 2025-09-23T04:56:20+00:00 |
Last Successful Scan
| Scanned | 2025-08-25T04:54:38+00:00 |
| URL | https://baseball-reference.com/robots.txt |
| Redirect | https://www.baseball-reference.com/robots.txt |
| Redirect Domain | www.baseball-reference.com |
| Redirect Base | baseball-reference.com |
| Domain IPs | 104.18.22.245, 104.18.23.245, 2606:4700::6812:16f5, 2606:4700::6812:17f5 |
| Redirect IPs | 104.18.22.245, 104.18.23.245, 2606:4700::6812:16f5, 2606:4700::6812:17f5 |
| Response IP | 104.18.22.245 |
| Found | Yes |
| Hash | 18c82a0ba6e250390995cbd2f4a39a63db6d8cf47984267495ddef26299068c4 |
| SimHash | 6546f9e367b1 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /baseball/ |
| Disallow | /oldpi/ |
| Disallow | /travel/travelguide.cgi |
| Disallow | /player_search.cgi |
| Disallow | /friv/random.cgi |
| Disallow | /friv/trade-partners.cgi |
| Disallow | /random.cgi |
| Disallow | /friv/compare.cgi |
| Disallow | /friv/scomp_bat.cgi |
| Disallow | /friv/scomp_pitch.cgi |
| Disallow | /friv/standings.cgi |
| Disallow | /compare.cgi |
| Disallow | /friv/killframe.cgi |
| Disallow | /feedback/index.cgi |
| Disallow | /games/streaks.cgi |
| Disallow | /games/teams.cgi |
| Disallow | /games/situational.cgi |
| Disallow | /games/standings.cgi |
| Disallow | /games/head2head.cgi |
| Disallow | /games/head2head-games.cgi |
| Disallow | /sponsors/ |
| Disallow | /my/ |
| Disallow | /play-index/ |
| Disallow | /friv/sp.cgi |
| Disallow | /friv/uniform_players.cgi |
| Disallow | /friv/uniform_players2.cgi |
| Disallow | /friv/uniform_players3.cgi |
| Disallow | /bpv/ |
| Disallow | /bullpen/Special%3AWhatlinkshere/ |
| Disallow | /bullpen/Special%3ARecentchangeslinked/ |
| Disallow | /players/split.cgi |
| Disallow | /leagues/split.cgi |
| Disallow | /teams/split.cgi |
| Disallow | /pl/player_search* |
| Disallow | /bullpen/Talk%3A* |
| Disallow | /bullpen/User%3A* |
| Disallow | /ancient/ |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Comments