data.ca.gov
robots.txt
Robots Exclusion Standard data for data.ca.gov
Resource Scan
Scan Details
| Site Domain | data.ca.gov |
| Base Domain | ca.gov |
| Scan Status | Ok |
| Last Scan | 2025-02-17T00:46:22+00:00 |
| Next Scan | 2025-03-19T00:46:22+00:00 |
Last Scan
| Scanned | 2025-02-17T00:46:22+00:00 |
| URL | https://data.ca.gov/robots.txt |
| Domain IPs | 104.19.218.112, 104.19.219.112, 2606:4700::6813:da70, 2606:4700::6813:db70 |
| Response IP | 104.19.219.112 |
| Found | Yes |
| Hash | d4e925ba03922494ffe5d71d8925ab5e2a74dbed790f0d0e68e790a73c6e1ab6 |
| SimHash | c2025bc26e72 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /dataset?* |
| Disallow | /dataset/?* |
| Disallow | /dataset/activity/* |
| Disallow | /dataset/groups/* |
| Disallow | /dataset/showcases/* |
| Disallow | /dataset/*/issues/* |
| Disallow | /dataset/*/resource/* |
| Disallow | /datastore/* |
| Disallow | /datarequest/* |
| Disallow | /group/*?* |
| Disallow | /organization/*?* |
| Disallow | /showcase?* |
| Disallow | /issues/ |
| Disallow | /revision/ |
| Disallow | /user/* |
| Disallow | /api/ |
| Disallow | /cgi-bin |
| Disallow | /wp-admin/ |
| Disallow | /wp-content/ |
| Disallow | /wp-includes/ |
| Disallow | /*.php$ |
| Disallow | /*.inc$ |
| Disallow | /*.gz$ |
| Disallow | /*.wmv$ |
| Disallow | /*.cgi$ |
| Disallow | /*.xhtml$ |
| Disallow | /am/ |
| Disallow | /ar/ |
| Disallow | /bg/ |
| Disallow | /ca |
| Disallow | /cs_CZ/ |
| Disallow | /da_DK/ |
| Disallow | /de/ |
| Disallow | /dv/ |
| Disallow | /el/ |
| Disallow | /en/ |
| Disallow | /en_AU/ |
| Disallow | /en_GB/ |
| Disallow | /es/ |
| Disallow | /es_AR/ |
| Disallow | /eu/ |
| Disallow | /fa_IR/ |
| Disallow | /fi/ |
| Disallow | /fr/ |
| Disallow | /he/ |
| Disallow | /hr/ |
| Disallow | /hu/ |
| Disallow | /id/ |
| Disallow | /is/ |
| Disallow | /it/ |
| Disallow | /ja/ |
| Disallow | /km/ |
| Disallow | /ko_KR/ |
| Disallow | /lt/ |
| Disallow | /lv/ |
| Disallow | /mk/ |
| Disallow | /mn_MN/ |
| Disallow | /my_MM/ |
| Disallow | /nb_NO/ |
| Disallow | /ne/ |
| Disallow | /nl/ |
| Disallow | /no/ |
| Disallow | /pl/ |
| Disallow | /pt_BR/ |
| Disallow | /pt_PT/ |
| Disallow | /ro/ |
| Disallow | /ru/ |
| Disallow | /sk/ |
| Disallow | /sl/ |
| Disallow | /sq/ |
| Disallow | /sr/ |
| Disallow | /sr_Latn/ |
| Disallow | /sv/ |
| Disallow | /th/ |
| Disallow | /tl/ |
| Disallow | /tr/ |
| Disallow | /uk/ |
| Disallow | /uk_UA/ |
| Disallow | /vi/ |
| Disallow | /zh_CN/ |
| Disallow | /zh_HK/ |
| Disallow | /zh_TW/ |
| Disallow | /zh_Hant_TW/ |
| Disallow | /zh_Hans_CN/ |
*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 15 |
Warnings
- 2 invalid lines.