pro-football-reference.com
robots.txt
Robots Exclusion Standard data for pro-football-reference.com
Resource Scan
Scan Details
| Site Domain | pro-football-reference.com |
| Base Domain | pro-football-reference.com |
| Scan Status | Ok |
| Last Scan | 2024-05-03T14:06:23+00:00 |
| Next Scan | 2024-05-10T14:06:23+00:00 |
Last Scan
| Scanned | 2024-05-03T14:06:23+00:00 |
| URL | https://pro-football-reference.com/robots.txt |
| Redirect | https://www.pro-football-reference.com/robots.txt |
| Redirect Domain | www.pro-football-reference.com |
| Redirect Base | pro-football-reference.com |
| Domain IPs | 104.18.40.110, 172.64.147.146, 2606:4700:4400::6812:286e, 2606:4700:4400::ac40:9392 |
| Redirect IPs | 104.18.40.110, 172.64.147.146, 2606:4700:4400::6812:286e, 2606:4700:4400::ac40:9392 |
| Response IP | 172.64.147.146 |
| Found | Yes |
| Hash | ed66b0b4b4ab5a4abb4348b90c88e1899d38613d1e39d91eebf1d1cb89a6c80b |
| SimHash | 4865d726c010 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /football/ |
| Disallow | /cgi-bin/search.cgi |
| Disallow | /player_search.fcgi |
| Disallow | /player_search.fcgi* |
| Disallow | /search/player_search.fcgi |
| Disallow | /search/player_search.fcgi* |
| Disallow | /my/ |
| Disallow | /players/*/*/splits |
| Disallow | /players/*/*/gamelog |
| Disallow | /players/*/*/big-games |
| Disallow | /players/*/*/field-goals |
| Disallow | /players/*/*/super-bowl |
| Disallow | /players/*/*/touchdowns |
| Disallow | /players/*/*/pick-sixes |
| Disallow | /players/*/*/penalties |
| Disallow | /players/*/*/fantasy |
| Disallow | /players/*/*/receiving-plays |
| Disallow | /players/*/*/red-zone-receiving-plays |
| Disallow | /players/*/*/passing-plays |
| Disallow | /players/*/*/red-zone-passing-plays |
| Disallow | /players/*/*/rushing-plays |
| Disallow | /players/*/*/red-zone-rushing-plays |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |