census.gov
robots.txt
Robots Exclusion Standard data for census.gov
Resource Scan
Scan Details
| Site Domain | census.gov |
| Base Domain | census.gov |
| Scan Status | Ok |
| Last Scan | 2024-09-23T15:07:48+00:00 |
| Next Scan | 2024-10-23T15:07:48+00:00 |
Last Scan
| Scanned | 2024-09-23T15:07:48+00:00 |
| URL | https://census.gov/robots.txt |
| Redirect | https://www.census.gov/robots.txt |
| Redirect Domain | www.census.gov |
| Redirect Base | census.gov |
| Domain IPs | 148.129.75.166, 2610:20:2010:a05:1000:0:9481:4ba6 |
| Redirect IPs | 23.43.177.135, 2600:141a:8000:188::208c, 2600:141a:8000:18f::208c |
| Response IP | 23.5.15.106 |
| Found | Yes |
| Hash | 9b8b03493fe8cc9c455398a27c5fddd4d0b48ab8afc470edb81f8cdbe928030c |
| SimHash | 6d05091144a6 |
Groups
*
w3c-checklink
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /libs/ |
| Disallow | /tmp/ |
| Disallow | /etc/ |
| Disallow | /about/adrm/data-linkage/ |
| Allow | /etc.clientlibs/census/clientlibs |
| Allow | /etc/clientlibs/granite |
| Allow | /etc/clientlibs/foundation |
googlebot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /libs/ |
| Disallow | /tmp/ |
| Disallow | /etc/ |
| Disallow | /about/adrm/data-linkage/ |
| Allow | /etc.clientlibs/census/clientlibs |
| Allow | /etc/clientlibs/granite |
| Allow | /etc/clientlibs/foundation |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
yahoo! slurp
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /libs/ |
| Disallow | /tmp/ |
| Disallow | /etc/ |
| Disallow | /about/adrm/data-linkage/ |
| Allow | /etc.clientlibs/census/clientlibs |
| Allow | /etc/clientlibs/granite |
| Allow | /etc/clientlibs/foundation |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.census.gov/sitemapindex/sitemap.xml |
| sitemap | https://www.census.gov/quickfacts/fact/sitemap/US/PST045217 |