freebsd.org
robots.txt
Robots Exclusion Standard data for freebsd.org
Resource Scan
Scan Details
Site Domain | freebsd.org |
Base Domain | freebsd.org |
Scan Status | Ok |
Last Scan | 2024-09-10T21:48:18+00:00 |
Next Scan | 2024-10-10T21:48:18+00:00 |
Last Scan
Scanned | 2024-09-10T21:48:18+00:00 |
URL | https://freebsd.org/robots.txt |
Redirect | https://www.freebsd.org/robots.txt |
Redirect Domain | www.freebsd.org |
Redirect Base | freebsd.org |
Domain IPs | 2610:1c1:1:606c::50:15, 96.47.72.84 |
Redirect IPs | 203.80.16.154, 2404:a8:3ff::50:3 |
Response IP | 203.80.16.154 |
Found | Yes |
Hash | 6d84161e16dfba34aebee11474ef59b2a5a7574e6ab6769001e00843d45f81eb |
SimHash | 103ddc72e893 |
Groups
*
Rule | Path |
---|---|
Disallow | /cgi/man.cgi?apropos=2& |
Disallow | /cgi/ports.cgi? |
Disallow | /cgi/man-autocomplete.cgi |
Disallow | /cgi/ports-autocomplete.cgi |
Disallow | /statistic |
Comments