baseballreference.com
robots.txt
Robots Exclusion Standard data for baseballreference.com
Resource Scan
Scan Details
| Site Domain | baseballreference.com |
| Base Domain | baseballreference.com |
| Scan Status | Ok |
| Last Scan | 2024-05-26T10:36:41+00:00 |
| Next Scan | 2024-06-09T10:36:41+00:00 |
Last Scan
| Scanned | 2024-05-26T10:36:41+00:00 |
| URL | https://baseballreference.com/robots.txt |
| Redirect | https://www.baseball-reference.com/robots.txt |
| Redirect Domain | www.baseball-reference.com |
| Redirect Base | baseball-reference.com |
| Domain IPs | 34.193.127.3, 67.202.44.235 |
| Redirect IPs | 104.18.38.165, 172.64.149.91, 2606:4700:4400::6812:26a5, 2606:4700:4400::ac40:955b |
| Response IP | 104.18.38.165 |
| Found | Yes |
| Hash | 6e2ded2bcb6f1e6561999bdfee008413f9c943e4a40007443b170685046c1223 |
| SimHash | 6546fbeb67b1 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /baseball/ |
| Disallow | /oldpi/ |
| Disallow | /travel/travelguide.cgi |
| Disallow | /player_search.cgi |
| Disallow | /friv/random.cgi |
| Disallow | /friv/trade-partners.cgi |
| Disallow | /random.cgi |
| Disallow | /friv/compare.cgi |
| Disallow | /friv/scomp_bat.cgi |
| Disallow | /friv/scomp_pitch.cgi |
| Disallow | /friv/standings.cgi |
| Disallow | /compare.cgi |
| Disallow | /friv/killframe.cgi |
| Disallow | /feedback/index.cgi |
| Disallow | /games/streaks.cgi |
| Disallow | /games/teams.cgi |
| Disallow | /games/situational.cgi |
| Disallow | /games/standings.cgi |
| Disallow | /games/head2head.cgi |
| Disallow | /games/head2head-games.cgi |
| Disallow | /oracle/at-distance.cgi |
| Disallow | /oracle/link.cgi |
| Disallow | /oracle/linkability.cgi |
| Disallow | /sponsors/ |
| Disallow | /my/ |
| Disallow | /play-index/ |
| Disallow | /friv/sp.cgi |
| Disallow | /friv/uniform_players.cgi |
| Disallow | /friv/uniform_players2.cgi |
| Disallow | /friv/uniform_players3.cgi |
| Disallow | /bpv/ |
| Disallow | /bullpen/Special%3AWhatlinkshere/ |
| Disallow | /bullpen/Special%3ARecentchangeslinked/ |
| Disallow | /players/split.cgi |
| Disallow | /leagues/split.cgi |
| Disallow | /teams/split.cgi |
| Disallow | /pl/player_search* |
| Disallow | /bullpen/Talk%3A* |
| Disallow | /bullpen/User%3A* |
| Disallow | /ancient/ |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Comments