health.gov
robots.txt
Robots Exclusion Standard data for health.gov
Resource Scan
Scan Details
Site Domain | health.gov |
Base Domain | health.gov |
Scan Status | Ok |
Last Scan | 2024-05-09T16:01:42+00:00 |
Next Scan | 2024-05-23T16:01:42+00:00 |
Last Scan
Scanned | 2024-05-09T16:01:42+00:00 |
URL | https://health.gov/robots.txt |
Domain IPs | 18.164.154.104, 18.164.154.115, 18.164.154.21, 18.164.154.92 |
Response IP | 18.165.171.58 |
Found | Yes |
Hash | c6bf0f52695e67c533543332d37a52f29a43ebab040ca2dfc79d9eb8b0eba817 |
SimHash | bc169d1b0554 |
Groups
*
Rule | Path |
---|---|
Disallow | /search/ |
Disallow | /core/ |
Disallow | /node/ |
Disallow | /espanol/node/ |
Disallow | /user/ |
Disallow | /admin/ |
Disallow | */search |
Disallow | /update.php |
Disallow | /composer* |
Disallow | /deploy* |
Disallow | */package.json |
Disallow | */package-lock.json |
Disallow | /moveyourway/widget |
Disallow | /espanol/moveyourway/widget |
Other Records
Field | Value |
---|---|
crawl-delay | 2 |
Comments