ars.usda.gov
robots.txt

Robots Exclusion Standard data for ars.usda.gov

Resource Scan

Scan Details

Site Domain ars.usda.gov
Base Domain usda.gov
Scan Status Ok
Last Scan2024-05-26T09:03:43+00:00
Next Scan 2024-06-25T09:03:43+00:00

Last Scan

Scanned2024-05-26T09:03:43+00:00
URL https://ars.usda.gov/robots.txt
Redirect https://www.ars.usda.gov/robots.txt
Redirect Domain www.ars.usda.gov
Redirect Base usda.gov
Domain IPs 52.245.234.40
Redirect IPs 52.245.234.40
Response IP 52.245.234.40
Found Yes
Hash 2cd3c223999996a23641fd82c59f5a7b99f12bba1698a897ad1221ca4a51f7a5
SimHash 3b027c437d90

Groups

*

Rule Path
Disallow /bin/
Disallow /config/
Disallow /css/
Disallow /Jquery/
Disallow /views/
Disallow /scripts/
Disallow /umbraco/
Disallow /umbraco_client/
Disallow /App_Plugins/

Other Records

Field Value
sitemap https://www.ars.usda.gov/sitemap.xml