Log In Sign Up

www4.cbs.state.or.us
robots.txt

Robots Exclusion Standard data for www4.cbs.state.or.us

Resource Scan

Scan Details

Site Domain www4.cbs.state.or.us
Base Domain state.or.us
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-07-02T21:26:45+00:00
Next Scan 2024-07-09T21:26:45+00:00

Last Successful Scan

Scanned2024-06-01T21:25:10+00:00
URL https://www4.cbs.state.or.us/robots.txt
Domain IPs 159.121.182.9
Response IP 159.121.182.9
Found Yes
Hash 03f26bf7a7dbbdeadcd26989276abbfc9c9a2a8623f5fb08fa259cde21404fc2
SimHash a2a65b0955e5

Groups

w3c-checklink

Rule Path
Disallow

dcbs-google

Rule Path
Allow /

googlebot

Rule Path
Disallow /exs/imd/survey/
Disallow /ex/dfcs/dfcslic/mortgage_lender/
Allow /ex/imd/reports/
Allow /ex/osha/film/
Allow /ex/osha/training/training/
Allow /exs/bcd/minor_labels/

msnbot

Rule Path
Disallow /exs/imd/survey/
Disallow /ex/dfcs/dfcslic/mortgage_lender/
Allow /ex/imd/reports/
Allow /ex/osha/film/
Allow /ex/osha/training/training/
Allow /exs/bcd/minor_labels/

Other Records

Field Value
crawl-delay 120

slurp

Rule Path
Disallow /exs/imd/survey/
Disallow /ex/dfcs/dfcslic/mortgage_lender/
Allow /ex/imd/reports/
Allow /ex/osha/film/
Allow /ex/osha/training/training/
Allow /exs/bcd/minor_labels/

Other Records

Field Value
crawl-delay 120

siteimprovebot-crawler

Rule Path
Allow /

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /
Disallow /exs/imd/survey/
Disallow /ex/dfcs/dfcslic/mortgage_lender/
Back to top

Other Records

Field Value
sitemap http://licenseinfo.oregon.gov/siteinfo.xml
Back to top

Comments

  • This robots.txt file is being used to keep robots from indexing
  • It is placed at the document root of the server.
  • Created by: Royann Janus Date: 07/26/98
  • Updated by: Glenn J. Schworak Date: 03/13/09
  • Let the world know about our site maps
  • Allow linkchecker at http://validator.w3.org/docs/checklink.html walk our entire site
  • Allow our internal GOOGLE to walk all folders
  • Allow Google into some select folders
  • Keep MSN from searching too often and specifically out of the survey
  • Keep Yahoo! from searching too often and specifically out of the survey
  • Allow these specific bots to crawl our site
  • SiteimproveBot-Crawler is for Tyler Oregon to check for Broken Links on Sharepoint
  • Block Huawei from crawling our sites
  • Keep all bots not specificlally mentioned above out of any folders
  • that are not specifically allowed in the list above.
Back to top