hiv.gov
robots.txt
Robots Exclusion Standard data for hiv.gov
Resource Scan
Scan Details
| Site Domain | hiv.gov |
| Base Domain | hiv.gov |
| Scan Status | Ok |
| Last Scan | 2024-11-12T16:33:16+00:00 |
| Next Scan | 2024-12-12T16:33:16+00:00 |
Last Scan
| Scanned | 2024-11-12T16:33:16+00:00 |
| URL | https://hiv.gov/robots.txt |
| Redirect | https://www.hiv.gov:443/robots.txt |
| Redirect Domain | www.hiv.gov |
| Redirect Base | hiv.gov |
| Domain IPs | 15.197.141.195, 3.33.151.151 |
| Redirect IPs | 13.33.183.36, 13.33.183.43, 13.33.183.46, 13.33.183.94, 2600:9000:2816:2400:d:49fb:cd40:93a1, 2600:9000:2816:2a00:d:49fb:cd40:93a1, 2600:9000:2816:4600:d:49fb:cd40:93a1, 2600:9000:2816:4e00:d:49fb:cd40:93a1, 2600:9000:2816:7000:d:49fb:cd40:93a1, 2600:9000:2816:9c00:d:49fb:cd40:93a1, 2600:9000:2816:bc00:d:49fb:cd40:93a1, 2600:9000:2816:f200:d:49fb:cd40:93a1 |
| Response IP | 108.157.254.126 |
| Found | Yes |
| Hash | 1d7d3238b9d5eecbece75e5dd901f4d68b078fe9dcf51a64e84080bc6035688a |
| SimHash | 484ac8300bb0 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Allow | / | - |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-program | - |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-pharmacies | - |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-program-resources | - |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-pharmacies/health-mart-pharmacies | - |
| Disallow | * | query=* |
| Disallow | *%26sortby%3D* | - |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.hiv.gov/sitemap-index.xml |
Warnings
- `host` is not a known field.