atsdr.cdc.gov
robots.txt

Robots Exclusion Standard data for atsdr.cdc.gov

Resource Scan

Scan Details

Site Domain atsdr.cdc.gov
Base Domain cdc.gov
Scan Status Ok
Last Scan2024-05-30T18:01:47+00:00
Next Scan 2024-06-29T18:01:47+00:00

Last Scan

Scanned2024-05-30T18:01:47+00:00
URL https://atsdr.cdc.gov/robots.txt
Domain IPs 198.246.106.34
Response IP 198.246.106.34
Found Yes
Hash 6be560f3a87d5140210d06c1ca83d7fbb0e1977c38c5b9466d71dc919d64fd9f
SimHash 7c00910303d3

Groups

*

Rule Path Comment
Disallow /cgi-bin/ Executables only
Disallow /usage/ Usage statistics pages only
Disallow /jobs/ Job postings - short expiration
Disallow /gsql/ This is an infinite database query space
Disallow /CHEM/ .gif and .xyz chemical files only

Comments

  • robots.txt for http://www.atsdr.cdc.gov/