Log In Sign Up

insurance.ca.gov
robots.txt

Robots Exclusion Standard data for insurance.ca.gov

Resource Scan

Scan Details

Site Domain insurance.ca.gov
Base Domain ca.gov
Scan Status Ok
Last Scan2024-08-28T04:31:34+00:00
Next Scan 2024-09-27T04:31:34+00:00

Last Scan

Scanned2024-08-28T04:31:34+00:00
URL https://insurance.ca.gov/robots.txt
Redirect https://www.insurance.ca.gov/robots.txt
Redirect Domain www.insurance.ca.gov
Redirect Base ca.gov
Domain IPs 67.136.93.16
Redirect IPs 67.136.93.16
Response IP 67.136.93.16
Found Yes
Hash 06cb1e63d63870c917b0f1fdd103f2f85557f59a4e64436db89d19ed6756d825
SimHash 290fac6393e0

Groups

*

Rule Path
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/reporting-2005/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/reporting-2006/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/reporting-2007/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2010/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2011/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2012/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2013/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2014/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2015/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2016/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2017/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2018/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2019/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2020/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/2021/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/reporting-2008/
Disallow /0400-news/0200-studies-reports/1100-statistical-plans/reporting-2009/
Disallow /0100-consumers/
Disallow /0100-consumers/0030-licensee-info/0031-surplus-lines/lasli.cfm
Disallow /0100-consumers/0030-licensee-info/0031-surplus-lines/
Disallow /loader.cfm
Disallow /login.cfm
Disallow /logout.cfm
Disallow /upload.cfm
Disallow /createpage.htm
Disallow /createpage.cfm
Disallow /index.htm
Disallow /customcf/handler404.cfm
Disallow /0300-fraud/0100-fraud-division-overview/25-wc-conv/upload/January2015.pdf
Disallow /commonspot/
Disallow /0300-fraud/0100-fraud-division-overview/ifab/CrossTraining/
Disallow /0250-insurers/0300-insurers/0100-applications/rsb-forms/2020/2020-3-submissions/
Disallow /rss/

naicspider
naicspider+

Rule Path
Disallow /

bingbot

Rule Path
Disallow /customcf/handler404.cfm

blekkobot

Rule Path
Disallow /0250-insurers/0300-insurers/0100-applications/rsb-forms/

megaindex.ru
megaindex.ru+

Rule Path
Disallow /

megaindex.ru/2.0
megaindex.ru/2.0+

Rule Path
Disallow /

botify

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /
Disallow /0250-insurers/0300-insurers/0100-applications/rsb-forms/2020/2020-3-submissions/
Disallow /0250-insurers/0300-insurers/0100-applications/rsb-forms/2020/2020-8-submissions/
Back to top

Comments

  • robots updated: 12/14/2021 Aiping Jiang
  • prohibit NAICSpider crowling the site
  • prohibit Bingbot crowling the site
  • prohibit Bingbot crowling the site
  • prohibit MegaIndex.ru bot crowling the site
  • prohibit botify bot crowling the site
  • prohibit Yandex bot crowling the site
  • Do not allow Python-urllib to access any part of your site
  • prohibit Semrush bot crowling the site
  • prohibit PaperLiBot crowling the site
  • prohibit Monsidobot crowling the site
  • prohibit bingbot crowling the site
Back to top