aids.gov
robots.txt
Robots Exclusion Standard data for aids.gov
Resource Scan
Scan Details
| Site Domain | aids.gov |
| Base Domain | aids.gov |
| Scan Status | Ok |
| Last Scan | 2024-10-20T13:26:46+00:00 |
| Next Scan | 2024-11-19T13:26:46+00:00 |
Last Scan
| Scanned | 2024-10-20T13:26:46+00:00 |
| URL | https://aids.gov/robots.txt |
| Redirect | https://www.hiv.gov:443/robots.txt |
| Redirect Domain | www.hiv.gov |
| Redirect Base | hiv.gov |
| Domain IPs | 15.197.141.195, 3.33.151.151 |
| Redirect IPs | 108.157.254.108, 108.157.254.114, 108.157.254.126, 108.157.254.85, 2600:9000:2753:3000:d:49fb:cd40:93a1, 2600:9000:2753:5000:d:49fb:cd40:93a1, 2600:9000:2753:5400:d:49fb:cd40:93a1, 2600:9000:2753:9200:d:49fb:cd40:93a1, 2600:9000:2753:ac00:d:49fb:cd40:93a1, 2600:9000:2753:b000:d:49fb:cd40:93a1, 2600:9000:2753:ca00:d:49fb:cd40:93a1, 2600:9000:2753:fc00:d:49fb:cd40:93a1 |
| Response IP | 108.157.254.85 |
| Found | Yes |
| Hash | d2f89ffbfda7fc4de06b4f225c4d01b52d79d396037b79030301fdbc2fc3261c |
| SimHash | 4848d8300fb0 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-program |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-pharmacies |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-program-resources |
| Disallow | /federal-response/ending-the-hiv-epidemic/prep-pharmacies/health-mart-pharmacies |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.hiv.gov/sitemap-index.xml |
Warnings
- `host` is not a known field.