psu.edu
robots.txt
Robots Exclusion Standard data for psu.edu
Resource Scan
Scan Details
| Site Domain | psu.edu |
| Base Domain | psu.edu |
| Scan Status | Ok |
| Last Scan | 2024-05-19T11:45:03+00:00 |
| Next Scan | 2024-06-18T11:45:03+00:00 |
Last Scan
| Scanned | 2024-05-19T11:45:03+00:00 |
| URL | https://psu.edu/robots.txt |
| Redirect | https://www.psu.edu:443/robots.txt |
| Redirect Domain | www.psu.edu |
| Redirect Base | psu.edu |
| Domain IPs | 75.2.126.119, 99.83.249.53 |
| Redirect IPs | 108.157.254.113, 108.157.254.121, 108.157.254.31, 108.157.254.89, 2600:9000:2753:3600:18:7374:6240:93a1, 2600:9000:2753:3800:18:7374:6240:93a1, 2600:9000:2753:4a00:18:7374:6240:93a1, 2600:9000:2753:5800:18:7374:6240:93a1, 2600:9000:2753:8000:18:7374:6240:93a1, 2600:9000:2753:c400:18:7374:6240:93a1, 2600:9000:2753:f600:18:7374:6240:93a1, 2600:9000:2753:fc00:18:7374:6240:93a1 |
| Response IP | 108.157.254.31 |
| Found | Yes |
| Hash | 72e979d77c27e386e5cc9c39aca0dcb67f26d95834a8b23530238a1be27f6dc9 |
| SimHash | 69605f51c991 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /components |
| Disallow | /404 |
| Disallow | /js |
| Disallow | /academics/campuses.html |
| Disallow | /academics/colleges.html |
| Disallow | /search/gss/index.html |
| Disallow | /search/department/index.html |
| Disallow | /search/departments/index.html |
| Disallow | /abington-rfi/index.html |
| Disallow | /altoona-rfi/index.html |
| Disallow | /beaver-rfi/index.html |
| Disallow | /behrend-rfi/index.html |
| Disallow | /berks-rfi/index.html |
| Disallow | /brandywine-rfi/index.html |
| Disallow | /dubois-rfi/index.html |
| Disallow | /fayette-rfi/index.html |
| Disallow | /great-valley-rfi/index.html |
| Disallow | /greater-allegheny-rfi/index.html |
| Disallow | /harrisburg-rfi/index.html |
| Disallow | /hazleton-rfi/index.html |
| Disallow | /lehigh-valley-rfi/index.html |
| Disallow | /mont-alto-rfi/index.html |
| Disallow | /new-kensington-rfi/index.html |
| Disallow | /schuylkill-rfi/index.html |
| Disallow | /scranton-rfi/index.html |
| Disallow | /shenango-rfi/index.html |
| Disallow | /penn-state-rfi/index.html |
| Disallow | /wilkes-barre-rfi/index.html |
| Disallow | /york-rfi/index.html |
| Disallow | /commonwealth-campuses/index.html |
| Disallow | /commonwealth-campuses/ |
| Disallow | /news/update |
| Disallow | /news/update/ |
| Disallow | /advocate-penn-state/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.psu.edu/sitemap.xml |
Comments