pfref.com
robots.txt
Robots Exclusion Standard data for pfref.com
Resource Scan
Scan Details
| Site Domain | pfref.com |
| Base Domain | pfref.com |
| Scan Status | Ok |
| Last Scan | 2024-10-27T03:43:53+00:00 |
| Next Scan | 2024-11-10T03:43:53+00:00 |
Last Scan
| Scanned | 2024-10-27T03:43:53+00:00 |
| URL | https://pfref.com/robots.txt |
| Redirect | https://www.pro-football-reference.com/robots.txt |
| Redirect Domain | www.pro-football-reference.com |
| Redirect Base | pro-football-reference.com |
| Domain IPs | 3.212.236.45, 44.217.214.192, 54.225.107.61 |
| Redirect IPs | 104.18.12.41, 104.18.13.41, 2606:4700::6812:c29, 2606:4700::6812:d29 |
| Response IP | 104.18.13.41 |
| Found | Yes |
| Hash | ed66b0b4b4ab5a4abb4348b90c88e1899d38613d1e39d91eebf1d1cb89a6c80b |
| SimHash | 4865d726c010 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /football/ |
| Disallow | /cgi-bin/search.cgi |
| Disallow | /player_search.fcgi |
| Disallow | /player_search.fcgi* |
| Disallow | /search/player_search.fcgi |
| Disallow | /search/player_search.fcgi* |
| Disallow | /my/ |
| Disallow | /players/*/*/splits |
| Disallow | /players/*/*/gamelog |
| Disallow | /players/*/*/big-games |
| Disallow | /players/*/*/field-goals |
| Disallow | /players/*/*/super-bowl |
| Disallow | /players/*/*/touchdowns |
| Disallow | /players/*/*/pick-sixes |
| Disallow | /players/*/*/penalties |
| Disallow | /players/*/*/fantasy |
| Disallow | /players/*/*/receiving-plays |
| Disallow | /players/*/*/red-zone-receiving-plays |
| Disallow | /players/*/*/passing-plays |
| Disallow | /players/*/*/red-zone-passing-plays |
| Disallow | /players/*/*/rushing-plays |
| Disallow | /players/*/*/red-zone-rushing-plays |
| Disallow | /req/ |
| Disallow | /short/ |
| Disallow | /nocdn/ |