The nationwide adherence to a common clinical vocabulary for emrs will result in

NPJ Digit Med. 2020; 3: 17.

Abstract

Computerized clinical decision support systems, or CDSS, represent a paradigm shift in healthcare today. CDSS are used to augment clinicians in their complex decision-making processes. Since their first use in the 1980s, CDSS have seen a rapid evolution. They are now commonly administered through electronic medical records and other computerized clinical workflows, which has been facilitated by increasing global adoption of electronic medical records with advanced capabilities. Despite these advances, there remain unknowns regarding the effect CDSS have on the providers who use them, patient outcomes, and costs. There have been numerous published examples in the past decade(s) of CDSS success stories, but notable setbacks have also shown us that CDSS are not without risks. In this paper, we provide a state-of-the-art overview on the use of clinical decision support systems in medicine, including the different types, current use cases with proven efficacy, common pitfalls, and potential harms. We conclude with evidence-based recommendations for minimizing risk in CDSS design, implementation, evaluation, and maintenance.

Subject terms: Health services, Diagnosis, Drug regulation, Medical imaging

Introduction: What is a clinical decision support system?

A clinical decision support system (CDSS) is intended to improve healthcare delivery by enhancing medical decisions with targeted clinical knowledge, patient information, and other health information.1 A traditional CDSS is comprised of software designed to be a direct aid to clinical-decision making, in which the characteristics of an individual patient are matched to a computerized clinical knowledge base and patient-specific assessments or recommendations are then presented to the clinician for a decision.2 CDSSs today are primarily used at the point-of-care, for the clinician to combine their knowledge with information or suggestions provided by the CDSS. Increasingly however, there are CDSS being developed with the capability to leverage data and observations otherwise unobtainable or uninterpretable by humans.

Computer-based CDSSs can be traced to the 1970s. At the time, they had poor system integration, were time intensive and often limited to academic pursuits.3,4 There were also ethical and legal issues raised around the use of computers in medicine, physician autonomy, and who would be at fault when using the recommendation of a system with imperfect ‘explainability’.5 Presently, CDSS often make use of web-applications or integration with electronic health records (EHR) and computerized provider order entry (CPOE) systems. They can be administered through desktop, tablet, smartphone, but also other devices such as biometric monitoring and wearable health technology. These devices may or may not produce outputs directly on the device or be linked into EHR databases.6

CDSSs have been classified and subdivided into various categories and types, including intervention timing, and whether they have active or passive delivery.7,8 CDSS are frequently classified as knowledge-based or non-knowledge based. In knowledge-based systems, rules (IF-THEN statements) are created, with the system retrieving data to evaluate the rule, and producing an action or output7; Rules can be made using literature-based, practice-based, or patient-directed evidence.2 CDSS that are non-knowledge based still require a data source, but the decision leverages artificial intelligence (AI), machine learning (ML), or statistical pattern recognition, rather than being programmed to follow expert medical knowledge.7 Non-knowledge based CDSS, although a rapidly growing use case for AI in medicine, are rife with challenges including problems understanding the logic that AI uses to produce recommendations (black boxes), and problems with data availability.9 They have yet to reach widespread implementation. Both types of CDSS have common components with subtle differences, illustrated in Fig. ​1.

Diagram of key interactions in knowledge-based and non-knowledge based CDSS.

They are composed of (1) base: the rules that are programmed into the system (knowledge-based), the algorithm used to model the decision (non-knowledge based), as well as the data available, (2) inference engine: takes the programmed or AI-determined rules, and data structures, and applies them to the patient’s clinical data to generate an output or action, which is presented to the end user (eg. physician) through the (3) communication mechanism: the website, application, or EHR frontend interface, with which the end user interacts with the system9.

CDSS have been endorsed by the US Government’s Health and Medicare acts, financially incentivizing CDS implementation into EHRs.10 In 2013, an estimated 41% of U.S. hospitals with an EHR, also had a CDSS, and in 2017, 40.2% of US hospitals had advanced CDS capability (HIMSS Stage 6).11 Elsewhere, adoption rates of EMRs have been promising, with approximately 62% of practitioners in Canada in 2013.12 Canada has had significant endorsement from the government level, as well as Infoway, a not-for-profit corporation.13 England has also been a world leader in healthcare IT investment, with up to 20 billion euros invested back in 2010.13 Several countries have also managed to implement national health records, at least for patient-facing data, including Denmark, Estonia, Australia, and others.14

The scope of functions provided by CDSS is vast, including diagnostics, alarm systems, disease management, prescription (Rx), drug control, and much more.15 They can manifest as computerized alerts and reminders, computerized guidelines, order sets, patient data reports, documentation templates, and clinical workflow tools.16 Each CDSS function will be discussed in detail throughout this review, with the potential and realized benefits of these functions, as well as unintended negative consequences, and strategies to avoid harm from CDSS. Methodology used to inform the review is shown in Box 1.

Box 1. Methods and sources used for this overview

• MEDLINE search 1980-January 2018. Key words: CDSS, diagnostic decision support system/DDSS, personal health record/PHR decision support, EHR decision support

• Hand searches of the references of retrieved literature

• University libraries searching for texts on clinical decision support systems and other keywords mentioned above

• Personal and local experience working with healthcare technology and decision support systems

Functions and advantages of CDSS

Patient safety

Strategies to reduce medication errors commonly make use of CDSS (Table ​1). Errors involving drug-drug interactions (DDI) are cited as common and preventable, with up to 65% of inpatients being exposed to one or more potentially harmful combinations.17 CPOE systems are now designed with drug safety software that has safeguards for dosing, duplication of therapies, and DDI checking.18 The types of alerts generated by these systems are among the most disseminated kind of decision support.19 However, studies have found a high level of variability between how alerts for DDIs are displayed (e.g., passive or active/disruptive), which are prioritized,20,21 and in the algorithms used to identify DDIs.18,22 Systems often have varying degrees of irrelevant alerts presented, and there is no standard for how best to implement which alerts to providers. The US Office of the National Coordinator for Health Information Technology has developed a list of ‘high-priority’ list of DDIs for CDS, which has reached various levels of endorsement and deployment in CDSS’ of other countries including the U.K., Belgium, and Korea.20,21,23

Table 1

Benefits of clinical decision support systems (CDSS), possible harms, and evidence-based mitigation strategies.

Other systems targeting patient safety include electronic drug dispensing systems (EDDS), and bar-code point-of-care (BPOC) medication administration systems.24 These are often implemented together to create a ‘closed loop’, where each step of the process (prescribing, transcribing, dispensing, administering) is computerized and occurs within a connected system. At administration, the medication is automatically identified through radio-frequency identification (RFID) or barcodes and crosschecked with patient information and prescriptions. This presents another target for CDSS and the potential benefit is the prevention of medication administration errors occurring at the ‘bedside’ (opposed to further upstream). Adoption is relatively low, partly due to high technology requirements and costs.25 However; studies show good efficacy for these systems in reducing errors.26 Mohoney et al. showed that many of these systems can be combined with CPOE and CDSS simultaneously, with reduced prescribing error rates for drug allergy detection, excessive dosing, and incomplete or unclear ordering.24 As with most CDSS, errors can still be made if providers omit or deliberately work around the technology.27

CDSS also improve patient safety through reminder systems for other medical events, and not just those that are medication related. Among numerous examples, a CDSS for blood glucose measurement in the ICU was able to decrease the number of hypoglycemia events.28 This CDSS automatically prompted nurses to take a glucose measurement according to a local glucose monitoring protocol, which specified how often measurements should be done according to specific patient demographics and previous glucose levels/trends.28

Overall, CDSS targeting patient safety through CPOE and other systems have generally been successful in reducing prescribing and dosing errors, contraindications through automated warnings, drug-event monitoring and more.29 Patient safety can be considered a secondary objective (or requirement) of almost all types of CDSS, no matter the primary purpose for their implementation.

Clinical management

Studies have shown CDSS can increase adherence to clinical guidelines.30 This is significant because traditional clinical guidelines and care pathways have been shown to be difficult to implement in practice with low clinician adherance.31,32 The assumption that practitioners will read, internalize, and implement new guidelines has not held true.33 However, the rules implicitly encoded in guidelines can be literally encoded into CDSS. Such CDSS can take a variety of forms, from standardized order sets for a targeted case, alerts to a specific protocol for the patients it pertains to, reminders for testing, etc. Furthermore, CDSS can assist with managing patients on research/treatment protocols,34 tracking and placing orders, follow-up for referrals, as well as ensuring preventative care.35

CDSS can also alert clinicians to reach out to patients who have not followed management plans, or are due for follow-up, and help identify patients eligible for research based on specific criteria.36 A CDSS designed and implemented at Cleveland Clinic provides a point-of-care alert to physicians when a patient’s record matches clinical trial criteria.37 The alert prompts the user to complete a form which establishes eligibility and consent-to-contact, forwards the patient’s chart to the study coordinator, and prints a clinical trial patient information sheet.

Cost containment

CDSS can be cost-effective for health systems, through clinical interventions,38 decreasing inpatient length-of-stay, CPOE-integrated systems suggesting cheaper medication alternatives,39 or reducing test duplication. A CPOE-rule was implemented in a pediatric cardiovascular intensive care unit (ICU) that limited the scheduling of blood count, chemistry and coagulation panels to a 24-h interval.40 This reduced laboratory resource utilization with a projected cost savings of $717,538 per year, without increasing length of stay (LOS), or mortality.

CDSS can notify the user of cheaper alternatives to drugs, or conditions that insurance companies will cover. In Germany, many inpatients are switched to drugs on hospital drug formularies. After finding that 1 in 5 substitutions were incorrect, Heidelberg hospital developed a drug-switch algorithm and integrated it into their existing CPOE system.41 The CDSS could switch 91.6% of 202 medication consultations automatically, with no errors, increasing safety, reducing workload and reducing cost for providers.

Administrative functions

CDSS provide support for clinical and diagnostic coding, ordering of procedures and tests, and patient triage. Designed algorithms can suggest a refined list of diagnostics codes to aid physicians in selecting the most suitable one(s). A CDSS was conceived to address inaccuracy of ICD-9 emergency department(ED) admission coding (ICD is International Statistical Classification of Diseases, standardized codes used to represent diseases and diagnoses).42 The tool used an anatomographical interface (visual, interactive representation of the human body) linked to ICD codes to help ED physicians accurately find diagnostic admission codes faster.

CDSS can directly improve quality of clinical documentation. An obstetric CDSS featured an enhanced prompting system, significantly improving documentation of indications for labor induction and estimated fetal weight, compared to control hospital.43 Documentation accuracy is important because it can directly aid clinical protocols. For example, a CDSS was implemented to ensure patients were properly vaccinated following splenectomy, to combat the increased risk of infections (including pneumococcal, Haemophilus influenzae, meningococcal, etc.) that comes with spleen removal. However, the authors found that 71% of patients with the term ‘splenectomy’ in their EHR did not have it documented on their problem list (which was what triggers the CDSS alert).44 A supplemental CDSS was then developed to enhance problem list documentation of splenectomy,45 and improve the utility of the original vaccination CDSS.

Diagnostics support

CDSS for clinical diagnosis are known as diagnostic decision support systems (DDSS). These systems have traditionally provided a computerized ‘consultation’ or filtering step, whereby they might be provided data/user selections, and then output a list of possible or probable diagnoses.46 Unfortunately, DDSS have not had as much influence as other types of CDSS (yet) for reasons including negative physician perceptions and biases, poor accuracy (often due to gaps in data availability), and poor system integration requiring manual data entry.47,48 The latter is improving with better EHR-integration and standardized vocabulary like Snomed Clinical Terms.

A good example of an effective DDSS is one which was created by Kunhimangalam et al.49 for diagnosis of peripheral neuropathy using fuzzy logic. Through 24 input fields which include symptoms and diagnostic test outputs, they achieved 93% accuracy compared to experts at identifying motor, sensory, mixed neuropathies, or normal cases. While this has great utility, especially in countries with less access to established clinical experts, there is also a desire for systems that can supplement specialist diagnostics. DXplain is an electronic reference based DDSS that provides probable diagnosis based on clinical manifestations.50 In a randomized control trial involving 87 family medicine residents, those randomized to use the system showed significantly higher accuracy (84% vs. 74%) on a validated diagnosis test involving 30 clinical cases.50

Given the known incidence of diagnostic errors, particularly in primary care,51 there is a lot of hope for CDSS and IT solutions to bring improvements to diagnosis.52 We are now seeing diagnostic systems being developed with non-knowledge-based techniques like machine learning, which may pave the way for more accurate diagnosis. The Babylon AI powered Triage and Diagnostic System in the U.K. is a good example of the potential, but also of the work that still has to be done before these systems are ready for primetime.53,54

Diagnostics support: imaging

Knowledge-based imaging CDSS are typically used for image ordering, where CDSS can aid radiologists in selecting the most appropriate test to run, providing reminders of best practice guidelines, or alerting contraindications to contrast, for example.55 An interventional CDS for image ordering at Virginia Mason Medical Center was shown to substantially decrease the utilization rate of lumbar MRI for low back pain, head MRI for headache, and sinus CT for sinusitis.56 The CDS required a series of questions to be answered by providers prior to image ordering (POC), to verify appropriateness. Importantly, if an image was denied, an alternative was suggested by the system. Another commercialized example is RadWise®, which guides clinicians to the most relevant imaging order by analyzing patient symptoms and matching them with a large database of diagnoses, while also providing appropriate use recommendations at the point of care.57

There is great interest in non-knowledge based CDS for enhanced imaging and precision radiology (‘radiomics’).58,59 With images accounting for increasing amounts of medical data, but requiring extensive manual interpretation, providers need technologies to aid them in extracting, visualizing, and interpreting.60 AI technologies are proving capable of providing insights into data beyond what humans can.61 To do so, these technologies make use of advanced pixel recognition and image classification algorithms, most prominently: deep learning (DL).62 IBM Watson Health, DeepMind, Google, and other companies are at the forefront, developing products for use in tumor detection,63 medical imaging interpretation,64 diabetic retinopathy diagnosis,65 Alzheimer’s diagnosis through multimodal feature learning,62 and countless more. IBM Watson’s ‘Eyes of Watson’, has been able to combine image recognition of a brain scan with text recognition of case descriptions to provide comprehensive decision support (or what IBM describes as a ‘cognitive assistant’).60

Several projects have been able to demonstrate performance that is disputably ‘on par’ with human experts.65–68 For example, Google’s team trained a deep convolutional neural network (CNN) to detect diabetic retinopathy (blood vessel damage in the eye) from a dataset of 130,000 retinal images with a very high sensitivity and specificity.65 The algorithms performance was on par with US board certified ophthalmologists. Another study just recently published by the Stanford group demonstrated a CNN for detecting arrhythmias on electrocardiogram that exceeded the accuracy (F1 and sensitivity with matched specificity) of the average cardiologist on all rhythm classes.68 With the current rate of progress, some experts controversially speculate that in 15–20 years, the majority of diagnostic imaging interpretation will be done (or at least pre-processed) by computers.69 For the time being however, we should think of these early systems as an addition or augmentation to a clinician’s available toolset.

Diagnostics support: laboratory and pathology

Another subset of diagnostics where CDSS can be useful is laboratory testing and interpretation. Alerts and reminders for abnormal lab results are simple and ubiquitous in EHR systems. CDSS can also extend the utility of lab-based tests for the purpose of avoiding riskier or more invasive diagnostics. In Hepatitis B and C testing, liver biopsies are considered the gold standard for diagnosis, while non-invasive lab tests are not accurate enough to be accepted. However; AI models are being developed that combine multiple tests (serum markers, imaging, and gene tests) to produce much greater accuracy.70 There is also application for CDSS as an interpretation tool where a test’s reference ranges are highly personalized, for example age, sex, or disease subtypes.71

Pathology reports are crucial as decision points for many other medical specialties. Some CDSS can be used for automated tumor grading. This was done for urinary bladder tumor grading and estimating recurrence, with up to 93% accuracy.72 The same has been done for brain tumor classification and grading.73 There are many other examples including computerized ECG analysis, automated arterial blood gas interpretation, protein electrophoresis reports, and CDSS for blood cell counting.46

Patient-facing decision support

With the advent of the ‘Personal Health Record’ (PHR), we are seeing CDS functionality integrated, similar to EHRs, with the patient as the end user or ‘manager’ of the data. This is a great step towards patient-focused care, and CDS-supported PHRs are the ideal tool to implement shared decision-making between patient and provider, specifically because CDSS can remove a ‘lack of information’ as a barrier to a patient’s participation in their own care.74 PHRs are frequently designed as an extension of commercial EHR software, or as standalone web-based or mobile-based applications.75 When connected to EHRs, PHRs can have a two way relationship, whereby information entered directly by the patient can be available to their providers, and also information in the EHR can be transmitted to the PHR for patients to view.76

One of the earliest PHRs, the “Patient Gateway”, was simply a dashboard for patients to view medications and labs, and communicate with their physicians.77 This has expanded and some systems now allow patients to modify their own record of care, effecting the EHR data as well.78 Another example is Vanderbilt University’s MyHealthAtVanderbilt, a PHR fully integrated into the institutional EHR. In addition to disease-targeted delivery of patient educational materials, they incorporated a Flu Tool for patients with flu-like symptoms to decide the level of care they need and then help them seek treatment.79 Symptom tracking is a useful and common feature of PHRs, but the variety of collected data is virtually limitless, from allergies to insurance coverage to prescription and medication information.80 Furthermore, PHRs and other patient monitoring applications can be designed to collect information from health devices and other wearables, to create actionable insights for providers. An excellent example exists in diabetes care. Many systems are already in use,81 but one in particular pioneered by the Stanford School of Medicine uses a wearable glucose monitor which transmits data to an Apple device (HealthKit).82 Apple has made HealthKit interoperable with the Epic EHR and Epic PHR, “MyChart”. This successfully allows providers to monitor glucose trends in their patients in between visits, and contact them through MyChart for follow up or urgent recommendations. The pilot study demonstrated improved provider workflow, communication with patients, and ultimately quality of care.82 Various other medical fields are deploying similar systems for monitoring that combines PHR/EHR, wearable technologies, and CDSS, including but not limited to heart failure (cardiology), hypertension, sleep apnea, palliative/elder care, and more.

It is worth noting that as PHRs have become more advanced with CDSS capabilities, there has also been increasing emphasis on the design of these systems to serve shared decision making between patient and provider, and to be interactive tools to make patients more knowledgeable/involved in their own care. PHRs that only serve as a repository for health information are now seen as missing the mark, particularly by patients themselves.75

Pitfalls of CDSS

Fragmented workflows

CDSS can disrupt clinician workflow, especially in the case of stand-alone systems. Many early CDSS were designed as systems that required the provider to document or source information outside their typical workspace. CDSS also disrupt workflow if designed without human information processing and behaviors in mind. In response, CDSS have been designed using the ‘think-aloud’ method to model practitioners’ workflow and create a system with better usability.83

Disrupted workflow can lead to increased cognitive effort, more time required to complete tasks, and less time face-to-face with patients. Even when CDSS are well integrated within existing information systems, there can be disconnect between face-to-face interactions and interaction with a computer workstation. Studies have found that practitioners with more experiential knowledge are less likely to use, and more likely to override CDSS.84

Alert fatigue and inappropriate alerts

Studies have found up to 95% of CDSS alerts are inconsequential, and often times physicians disagree with or distrust alerts.85 Other times they just do not read them. If physicians are presented with excessive/unimportant alerts, they can suffer from alert fatigue.86

Disruptive alerts should be limited to more life-threatening or consequential contraindications, such as serious allergies. However; even allergy alerts can be incorrect, and clinicians will often verify themselves, especially if the source is another site/hospital/practitioner.85,87 Medication alerts can also be specialty specific, but irrelevant when taken out of context. For example, an alert against using broad-spectrum antibiotics such as vancomycin may be inappropriate in ICU.85 An alert against duplicate medications may be inappropriate in inflammatory bowel disease clinics, where the same class of drug can be applied through different administration routes for increased effect.

Impact on user skill

Prior to CPOE and CDSS, healthcare providers, pharmacists, and nurses were relied upon exclusively to double-check orders. CDSS can create the impression that verifying the accuracy of an order is unnecessary or automatic.85 This is an important myth to dispel.

It is also important to consider the potential long-term effect of a CDSS on users. Over time a CDSS can exert a training effect, so that the CDSS itself may no longer be required. Coined the “carry-over effect”, it is most likely with CDSS that are educational in nature.88 Conversely, providers may develop too much reliance or trust on a CDSS for a specific task.89 This could be compared to using a calculator for mathematical operations over a long period of time, and then having poorer mental math skills. It is potentially problematic as the user has less independence and will be less equipped for that task should they switch to an environment without the CDSS.

CDSS may be dependent on computer literacy

Lack of technological proficiency can be hindering when engaging with a CDSS.90,91 This can vary by the design details of the CDSS, but some have been found to be overly complex, relying too much on user skill.90,92 Systems should aim to stay as close to the core functionality of the pre-existing system as possible. Regardless, all new systems have a learning period, and so baseline evaluations of users’ technological competence may be appropriate. Further training can then be provided to facilitate full use of CDSS capabilities,93 or more explicit guidance incorporated into the CDSS’ recommendations themselves.94 This information could be implemented as info buttons to be non-disruptive.95

System and content maintenance

Maintenance of CDSS is an important but often neglected part of the CDSS life-cycle. This includes technical maintenance of systems, applications and databases that power the CDSS. Another challenge is the maintenance of knowledge-base and its rules, which must keep apace with the fast-changing nature of medical practice and clinical guidelines. Even the most advanced healthcare institutions report difficulty keeping their systems up to date as knowledge inevitably changes.85 Order sets and the algorithmic rules behind the CDSS have been identified as particularly difficult.85

Operational impact of poor data quality and incorrect content

EHRs and CDSSs rely on data from external, dynamic systems and this can create novel deficiencies. As an example, some CDSS modules might encourage ordering even when the hospital lacks adequate supplies. In a study by Ash et al.85, a number of experts indicated that at their hospital, Hemoccult tests or pneumococcal vaccine inventories run out quickly, but this is not communicated to the CDSS.

Medication and problem lists can be problematic, if not updated or used appropriately. At one site, the medication list might be a list of dispensations, which means patients may or may not be taking them(and thus must still be asked in person).85 Other medication lists are generated from CPOE orders only, thus still requiring manual confirmation that patients are taking the medication. Systems that make it easy to distinguish these are ideal. It is also a major area where PHRs could create a solution, by collecting medication adherence data directly from patients.

In poorly designed systems, users may develop workarounds that compromise data, such as entering generic or incorrect data.85 The knowledge base of CDSS is dependent on a centralized, large clinical data repository. Quality of data can affect quality of decision support. If data collection or input into the system is unstandardized, the data is effectively corrupted. You may design a system for use at the point-of-care, but when applied to real world environments and data, will not be utilized properly. The importance of using informational standards such as ICD, SNOMED, and others, cannot be understated.

Lack of transportability and interoperability

Despite ongoing development for the better part of three decades, CDSS (and even EHRs in general) suffer from interoperability issues. Many CDSS exist as cumbersome stand-alone systems, or exist in a system that cannot communicate effectively with other systems.

What makes transportability so difficult to achieve? Beyond programming complexities that can make integration difficult, the diversity of clinical data sources is a challenge.96 There is a reluctance or perceived risk associated with transporting sensitive patient information. Positively, interoperability standards are continuously being developed and improved, such as Health Level 7 (HL7) and Fast Healthcare Interoperability Resources (FHIR). These are already being utilized in commercial EHR vendors.97 Several government agencies, medical organizations and informatics bodies are actively supporting and some even mandating the use of these interoperability standards in health systems.98–100

The cloud also offers a potential solution to interoperability (and other EHR ailments such as data sync, software updating, etc.101). Cloud EHRs have open architecture, newer standards, and more flexible connectivity to other systems.102 It is also a common misconception that data stored on a cloud is more vulnerable. This is not necessarily true. Web-based EHRs are required to store data in high-level storage centers with advanced encryption and other safeguards. They must comply with national data security standards including the Health Insurance Portability and Accountability Act (HIPAA) in the USA, Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada, or the Data Protection Directive and General Data Protection Regulation (GDPR) in Europe, to name a few.103 They can be just as safe (or just as vulnerable) as traditional, server-based architecture.103 In fact, there are often fewer people who have access to unencrypted data in cloud storage centers vs. server-based records.103

Financial challenges

Up to 74% of those with a CDSS said that financial viability remains a struggle.104 Outset costs to set up and integrate new systems can be substantial. Ongoing costs can continue to be an issue indefinitely as new staff need to be trained to use the system, and system updates are required to keep pace with current knowledge.

Results from cost analyses of CDSS implementations are mixed, controversial, and sparse.105–108 Whether an intervention is cost-effective depends on a wide range of factors, including those specific to the environment, both political and technological.105 Cost benefit assessment in itself can be limited, with challenges such as a lack of standardized metrics.107 This is an emerging research area and much work needs to be done to advance our understanding of the financial effects of CDSS.

Conclusion

CDSS have been shown to augment healthcare providers in a variety of decisions and patient care tasks, and today they actively and ubiquitously support delivery of quality care. Some applications of CDSS have more evidence behind them, especially those based on CPOE. Support for CDSS continues to mount in the age of the electronic medical record, and there are still more advances to be made including interoperability, speed and ease of deployment, and affordability. At the same time, we must stay vigilant for potential downfalls of CDSS, which range from simply not working and wasting resources, to fatiguing providers and compromising quality of patient care. Extra precautions and conscientious design must be taken when building, implementing, and maintaining CDSS. A portion of these considerations were covered in this review, but further review will be required in practice, especially as CDSS continue to evolve in complexity through advances in AI, interoperability, and new sources of data.

Acknowledgements

The author’s (R.T.S.) work was supported by a Canada Institute of Health (CIHR) Research Graduate Scholarship (CGS-M). Thank you to Nathan Stern for discussion and initial review of the manuscript.

Author contributions

R.T.S. reviewed the literature, drafted and revised the manuscript. D.P., D.C.S., D.C.B., K.I.K., and R.N.F. provided critical input, discussion, and revision of the manuscript. All authors approved the final submitted version of the manuscript.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

1. Osheroff, J. et al. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. (HIMSS Publishing, 2012).

2. Sim I, et al. Clinical decision support systems for the practice of evidence-based medicine. J. Am. Med Inf. Assoc. Jamia. 2001;8:527–534. doi: 10.1136/jamia.2001.0080527. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

3. De Dombal, F. Computers, diagnoses and patients with acute abdominal pain. Arch. Emerg. Med. 9, 267–270 (1992). [PMC free article] [PubMed]

4. Shortliffe EH, Buchanan BG. A model of inexact resoning in medicine. Math. Biosci. 1975;379:233–262. [Google Scholar]

5. Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb. Med. Inform. 2016;25(S 01):S103–S116. doi: 10.15265/IYS-2016-s034. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

6. Dias Duarte, Paulo Silva Cunha João. Wearable Health Devices—Vital Sign Monitoring, Systems and Technologies. Sensors. 2018;18(8):2414. doi: 10.3390/s18082414. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

7. Berner, E. S. (Ed.). Clinical Decision Support Systems (Springer, New York, NY, 2007).

8. Osheroff, J., Pifer, E., Teigh, J., Sittig, D. & Jenders, R. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. (HIMS, 2005).

12. Chang, F. & Gupta, N. Progress in electronic medical record adoption in Canada Recherche Les progrès dans l’ adoption du dossier médical électronique au Canada. Canadian Family Physician.61, 1076–1084 (2015).

14. Nøhr, C. et al. Nationwide citizen access to their health data: analysing and comparing experiences in Denmark, Estonia and Australia. 1–11. 10.1186/s12913-017-2482-y (2017). [PMC free article] [PubMed]

15. Omididan Z, Hadianfar A. The role of clinical decision support systems in healthcare (1980-2010): a systematic review study. Jentashapir Sci.-Res Q. 2011;2:125–134. [Google Scholar]

16. Kabane, S. M. Healthcare and the Effect of Technology: Developments, Challenges and Advancements: Developments, Challenges and Advancements. Medical Information Science Reference (2010).

17. Vonbach P, Dubied A, Krähenbühl S, Beer JH. Prevalence of drug-drug interactions at hospital entry and during hospital stay of patients in internal medicine. Eur. J. Intern. Med. 2008;19:413–420. doi: 10.1016/j.ejim.2007.12.002. [PubMed] [CrossRef] [Google Scholar]

18. Helmons PJ, Suijkerbuijk BO, Nannan Panday PV, Kosterink JGW. Drug-drug interaction checking assisted by clinical decision support: a return on investment analysis. J. Am. Med. Inf. Assoc. Jamia. 2015;22:764–772. doi: 10.1093/jamia/ocu010. [PubMed] [CrossRef] [Google Scholar]

19. Koutkias V, Bouaud J. Contributions from the 2017 Literature on Clinical Decision Support. Yearb. Med. Inf. 2018;27:122–128. doi: 10.1055/s-0038-1641222. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

20. Phansalkar Shobha, Desai Amrita A, Bell Douglas, Yoshida Eileen, Doole John, Czochanski Melissa, Middleton Blackford, Bates David W. High-priority drug–drug interactions for use in electronic health records. Journal of the American Medical Informatics Association. 2012;19(5):735–743. doi: 10.1136/amiajnl-2011-000612. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

21. Cornu P, Phansalkar S, Seger DL, Cho I, Pontefract S. International Journal of Medical Informatics High-priority and low-priority drug – drug interactions in di ff erent international electronic health record systems: a comparative study. Int. J. Med. Inf. 2018;111:165–171. doi: 10.1016/j.ijmedinf.2017.12.027. [PubMed] [CrossRef] [Google Scholar]

22. McEvoy DS, et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J. Am. Med. Inf. Assoc. 2017;24:331–338. [PMC free article] [PubMed] [Google Scholar]

23. Cho I, Lee J, Choi J, Hwang H, Bates DW. National rules for drug-drug interactions: are they appropriate for tertiary hospitals? J. Korean Med. Sci. 2016;31:1887–1896. doi: 10.3346/jkms.2016.31.12.1887. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

24. Mahoney CD, Berard-Collins CM, Coleman R, Amaral JF, Cotter CM. Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am. J. Health Syst. Pharm. 2007;64:1969–1977. doi: 10.2146/ajhp060617. [PubMed] [CrossRef] [Google Scholar]

25. Peris-Lopez P, Orfila A, Mitrokotsa A, van der Lubbe JCA. A comprehensive RFID solution to enhance inpatient medication safety. Int. J. Med. Inf. 2011;80:13–24. doi: 10.1016/j.ijmedinf.2010.10.008. [PubMed] [CrossRef] [Google Scholar]

26. Levtzion-korach O, et al. Effect of bar-code technology on the safety of medication administration. N. Engl. J. Med. 2010;362:1698–1707. doi: 10.1056/NEJMsa0907115. [PubMed] [CrossRef] [Google Scholar]

27. van der Veen W, et al. Association between workarounds and medication administration errors in bar-code-assisted medication administration in hospitals. J. Am. Med. Inf. Assoc. 2018;25:385–392. doi: 10.1093/jamia/ocx077. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

28. Eslami S, et al. Effects of two different levels of computerized decision support on blood glucose regulation in critically ill patients. Int J. Med. Inf. 2012;81:53–60. doi: 10.1016/j.ijmedinf.2011.10.004. [PubMed] [CrossRef] [Google Scholar]

29. Jia P, Zhang L, Chen J, Zhao P, Zhang M. The effects of clinical decision support systems on medication safety: an overview. PLoS ONE. 2016;11:1–17. [PMC free article] [PubMed] [Google Scholar]

30. Kwok R, Dinh M, Dinh D, Chu M. Improving adherence to asthma clinical guidelines and discharge documentation from emergency departments: Implementation of a dynamic and integrated electronic decision support system. Emerg. Med. Australas. 2009;21:31–37. [PubMed] [Google Scholar]

31. Davis DA, Taylor-Vaisey A. Translating guidelines into practice: a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. Can. Med. Assoc. J. 1997;157:408–416. [PMC free article] [PubMed] [Google Scholar]

32. Michael C, Rand CS, Powe NR, Wu AW, Wilson MH. Why don’ t physicians follow clinical practice guidelines? a framework for improvement. Jama,. 1999;282:1458–1465. doi: 10.1001/jama.282.15.1458. [PubMed] [CrossRef] [Google Scholar]

34. Lipton JA, et al. Impact of an alerting clinical decision support system for glucose control on protocol compliance and glycemic control in the intensive cardiac care unit. Diabetes Technol. Ther. 2011;13:343–349. doi: 10.1089/dia.2010.0100. [PubMed] [CrossRef] [Google Scholar]

35. Salem Hesham A, Caddeo Giacomo, McFarlane Jon, Patel Kunjan, Cochrane Lynda, Soria Daniele, Henley Mike, Lund Jonathan. A multicentre integration of a computer-led follow-up of prostate cancer is valid and safe. BJU International. 2018;122(3):418–426. doi: 10.1111/bju.14157. [PubMed] [CrossRef] [Google Scholar]

37. Embi, P. J., Jain, A., Clark, J. & Harris, C. M. Development of an electronic health record-based Clinical Trial Alert system to enhance recruitment at the point of care. AMIA Annu. Symp. Proc.2005, 231–235 (2005). [PMC free article] [PubMed]

38. Calloway S, Akilo H, Bierman K. Impact of a clinical decision support system on pharmacy clinical interventions, documentation efforts, and costs. Hosp. Pharm. 2013;48:744–752. doi: 10.1310/hpj4809-744. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

39. McMullin ST, et al. Impact of an evidence-based computerized decision support system on primary care prescription costs. Ann. Fam. Med. 2004;2:494–498. doi: 10.1370/afm.233. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

40. Algaze CA, et al. Use of a checklist and clinical decision support tool reduces laboratory use and improves cost. Pediatrics. 2016;137:e20143019. doi: 10.1542/peds.2014-3019. [PubMed] [CrossRef] [Google Scholar]

41. Pruszydlo MG, Walk-Fritz SU, Hoppe-Tichy T, Kaltschmidt J, Haefeli WE. Development and evaluation of a computerised clinical decision support system for switching drugs at the interface between primary and tertiary care. BMC Med. Inf. Decis. Mak. 2012;12:1. doi: 10.1186/1472-6947-12-1. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

42. Bell CM, Jalali A, Mensah E. A decision support tool for using an ICD-10 anatomographer to address admission coding inaccuracies: a commentary. Online J. Public Health Inform. 2013;5:222. doi: 10.5210/ojphi.v5i2.4813. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

43. Haberman S, et al. Effect of clinical-decision support on documentation compliance in an electronic medical record. Obstet. Gynecol. 2009;114:311–317. doi: 10.1097/AOG.0b013e3181af2cb0. [PubMed] [CrossRef] [Google Scholar]

44. Turchin, A., Shubina, M. & Gandhi, T. NLP for patient safety: splenectomy and pneumovax. In Proc. AMIA 2010 Annual Symposium (2010).

45. McEvoy Dustin, Gandhi Tejal K, Turchin Alexander, Wright Adam. Enhancing problem list documentation in electronic health records using two methods: the example of prior splenectomy. BMJ Quality & Safety. 2017;27(1):40–47. doi: 10.1136/bmjqs-2017-006707. [PubMed] [CrossRef] [Google Scholar]

46. Berner Eta S., editor. Clinical Decision Support Systems. New York, NY: Springer New York; 2007. [Google Scholar]

47. Berner, E. S. Diagnostic decision support systems: why aren’t they used more and what can we do about it? AMIA Annu. Symp. Proc. 2006, 1167–1168 (2006). [PMC free article] [PubMed]

48. Segal MM, et al. Experience with integrating diagnostic decision support software with electronic health records: benefits versus risks of information sharing. EGEMs Gener. Evid. Methods Improv. Patient Outcomes. 2017;5:23. doi: 10.5334/egems.244. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

49. Kunhimangalam R, Ovallath S, Joseph PK. A clinical decision support system with an integrated EMR for diagnosis of peripheral neuropathy. J. Med. Syst. 2014;38:38. doi: 10.1007/s10916-014-0038-9. [PubMed] [CrossRef] [Google Scholar]

50. Martinez-Franco AI, et al. Diagnostic accuracy in Family Medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial. Diagn. Berl. Ger. 2018;5:71–76. doi: 10.1515/dx-2017-0045. [PubMed] [CrossRef] [Google Scholar]

51. Singh H, Schiff GD, Graber ML, Onakpoya I, Thompson MJ. The global burden of diagnostic errors in primary care. BMJ Qual. Saf. 2017;26:484–494. doi: 10.1136/bmjqs-2016-005401. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

52. Singh H, Meyer AND, Thomas EJ. The frequency of diagnostic errors in outpatient care: Estimations from three large observational studies involving US adult populations. BMJ Qual. Saf. 2014;23:727–731. doi: 10.1136/bmjqs-2013-002627. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

53. Razzaki, S. et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis. arXiv preprint arXiv:1806.10698 (2018).

54. Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. Lancet. 2018;392:2263–2264. doi: 10.1016/S0140-6736(18)32819-8. [PubMed] [CrossRef] [Google Scholar]

55. Georgiou A, Prgomet M, Markewycz A, Adams E, Westbrook JI. The impact of computerized provider order entry systems on medical-imaging services: a systematic review. J. Am. Med. Inf. Assoc. 2011;18:335–340. doi: 10.1136/amiajnl-2010-000043. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

56. Blackmore CC, Mecklenburg RS, Kaplan GS. Effectiveness of clinical decision support in controlling inappropriate imaging. JACR. 2019;8:19–25. [PubMed] [Google Scholar]

58. Giardino A, et al. Role of imaging in the era of precision medicine. Acad. Radiol. 2017;24:639–649. doi: 10.1016/j.acra.2016.11.021. [PubMed] [CrossRef] [Google Scholar]

59. Oakden-rayner, L. et al. Precision Radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework. Sci. Rep. 10.1038/s41598-017-01931-w (2017). [PMC free article] [PubMed]

61. Greenspan H, Ginneken van B, Summers RM. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging. 2016;35:1153–1159. doi: 10.1109/TMI.2016.2553401. [CrossRef] [Google Scholar]

62. Suzuki Kenji, Chen Yisong., editors. Artificial Intelligence in Decision Support Systems for Diagnosis in Medical Imaging. Cham: Springer International Publishing; 2018. [Google Scholar]

65. Gulshan V, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA - J. Am. Med. Assoc. 2016;316:2402–2410. doi: 10.1001/jama.2016.17216. [PubMed] [CrossRef] [Google Scholar]

66. Esteva A, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115. doi: 10.1038/nature21056. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

67. Rajpurkar, P. et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017).

68. Hannun Awni Y., Rajpurkar Pranav, Haghpanahi Masoumeh, Tison Geoffrey H., Bourn Codie, Turakhia Mintu P., Ng Andrew Y. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine. 2019;25(1):65–69. doi: 10.1038/s41591-018-0268-3. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

69. Erickson, B. J. Machine Intelligence in Medical Imaging (Society for Imaging Informatics, SIIM, 2016).

70. Keltch, B., Lin, Y. & Bayrak, C. Comparison of AI techniques for prediction of liver fibrosis in hepatitis patients patient facing systems. J. Med. Syst. 10.1007/s10916-014-0060-y (2014). [PubMed]

71. Mørkrid L, et al. Continuous age- and sex-adjusted reference intervals of urinary markers for cerebral creatine deficiency syndromes: a novel approach to the definition of reference intervals. Clin. Chem. 2015;61:760–768. doi: 10.1373/clinchem.2014.235564. [PubMed] [CrossRef] [Google Scholar]

72. Spyridonos P, Cavouras D, Ravazoula P, Nikiforidis G. A computer-based diagnostic and prognostic system for assessing urinary bladder tumour grade and predicting cancer recurrence. Med. Inf. Internet Med. 2002;27:111–122. doi: 10.1080/1463923021000043723. [PubMed] [CrossRef] [Google Scholar]

73. Tsolaki E, et al. Fast spectroscopic multiple analysis (FASMA) for brain tumor classification: a clinical decision support system utilizing multi-parametric 3T MR data. Int. J. Comput. Assist Radio. Surg. 2015;10:1149–1166. doi: 10.1007/s11548-014-1088-7. [PubMed] [CrossRef] [Google Scholar]

74. Davis S, Roudsari A, Raworth R, Courtney KL, Mackay L. Shared decision-making using personal health record technology: a scoping review at the crossroads. J. Am. Med. Inf. Assoc. 2017;24:857–866. doi: 10.1093/jamia/ocw172. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

75. Fuji KT, et al. Standalone personal health records in the United States: meeting patient desires. Health Technol. 2012;2:197–205. doi: 10.1007/s12553-012-0028-1. [CrossRef] [Google Scholar]

76. Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J. Am. Med. Inf. Assoc. 2006;13:121–126. doi: 10.1197/jamia.M2025. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

77. Wald JS, et al. A patient-controlled journal for an electronic medical record: issues and challenges. Stud. Health Technol. Inform. 2004;107(Pt 2):1166–1170. [PubMed] [Google Scholar]

78. Hanauer DA, Preib R, Zheng K, Choi SW. Patient-initiated electronic health record amendment requests. J. Am. Med. Inf. Assoc. 2014;21:992–1000. doi: 10.1136/amiajnl-2013-002574. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

79. Rosenbloom ST, et al. Triaging patients at risk of influenza using a patient portal. J. Am. Med. Inf. Assoc. 2012;19:549–554. doi: 10.1136/amiajnl-2011-000382. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

80. Roehrs Alex, da Costa Cristiano André, Righi Rodrigo da Rosa, de Oliveira Kleinner Silva Farias. Personal Health Records: A Systematic Literature Review. Journal of Medical Internet Research. 2017;19(1):e13. doi: 10.2196/jmir.5876. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

81. Benhamou PY. Improving diabetes management with electronic health records and patients’ health records. Diabetes Metab. 2011;37(Suppl. 4):S53–S56. doi: 10.1016/S1262-3636(11)70966-1. [PubMed] [CrossRef] [Google Scholar]

82. Kumar RB, Goren ND, Stark DE, Wall DP, Longhurst CA. Automated integration of continuous glucose monitor data in the electronic health record using consumer technology. J. Am. Med. Inf. Assoc. 2016;23:532–537. doi: 10.1093/jamia/ocv206. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

83. Kilsdonk E, Peute LW, Riezebos RJ, Kremer LC, Jaspers MWM. Uncovering healthcare practitioners’ information processing using the think-aloud method: From paper-based guideline to clinical decision support system. Int. J. Med. Inf. 2016;86:10–19. doi: 10.1016/j.ijmedinf.2015.11.011. [PubMed] [CrossRef] [Google Scholar]

84. Dowding D, et al. Nurses’ use of computerised clinical decision support systems: A case site analysis. J. Clin. Nurs. 2009;18:1159–1167. doi: 10.1111/j.1365-2702.2008.02607.x. [PubMed] [CrossRef] [Google Scholar]

85. Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp. Proc. AMIA Symp. AMIA Symp. 2007;2007:26–30. [PMC free article] [PubMed] [Google Scholar]

86. Khalifa M, Zabani I. Improving utilization of clinical decision support systems by reducing alert fatigue: Strategies and recommendations. Stud. Health Technol. Inform. 2016;226:51–54. [PubMed] [Google Scholar]

87. Légat Laura, Van Laere Sven, Nyssen Marc, Steurbaut Stephane, Dupont Alain G, Cornu Pieter. Clinical Decision Support Systems for Drug Allergy Checking: Systematic Review. Journal of Medical Internet Research. 2018;20(9):e258. doi: 10.2196/jmir.8206. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

89. Goddard K, Roudsari A, Wyatt J. Automation bias - A hidden issue for clinical decision support system use. Stud. Health Technol. Inform. 2011;164:17–22. [PubMed] [Google Scholar]

90. Devaraj, S., Sharma, S. K., Fausto, D. J., Viernes, S. & Kharrazi, H. Barriers and facilitators to clinical decision support systems adoption: a systematic review. J. Bus Adm. Res. 10.5430/jbar.v3n2p36 (2014).

91. Leslie SJ, et al. Clinical decision support software for management of chronic heart failure: Development and evaluation. Comput Biol. Med. 2006;36:495–506. doi: 10.1016/j.compbiomed.2005.02.002. [PubMed] [CrossRef] [Google Scholar]

92. Murray E, et al. Why is it difficult to implement e-health initiatives? A qualitative study. Implement Sci. 2011;6:6. doi: 10.1186/1748-5908-6-6. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

93. Lai F, Macmillan J, Daudelin DH, Kent DM. The potential of training to increase acceptance and use of computerized decision support systems for medical diagnosis. Hum. Factors J. Hum. Factors Erg. Soc. 2006;48:95–108. doi: 10.1518/001872006776412306. [PubMed] [CrossRef] [Google Scholar]

94. Ojeleye L. Ensuring effective computerised clinical decision support. Prescriber. 2016;27:54–56. doi: 10.1002/psb.1464. [CrossRef] [Google Scholar]

95. Cook DA, Teixeira MT, Heale BSE, Cimino JJ, Del Fiol G. Context-sensitive decision support (infobuttons) in electronic health records: a systematic review. J. Am. Med Inf. Assoc. 2017;24:460–468. doi: 10.1093/jamia/ocx002. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

96. Sujansky W. Heterogeneous database integration in biomedicine. J. Biomed. Inform. 2001;34:285–298. doi: 10.1006/jbin.2001.1024. [PubMed] [CrossRef] [Google Scholar]

98. Katehakis, D. G. Towards the Development of a National eHealth Interoperability Framework to Address Public Health Challenges in Greece. Proceedings of the First International Workshop on Semantic Web Technologies for Health Data Management, SWH@ISWC.2164, 1–9 (2018).

102. Fernández-Cardeñosa G, De La Torre-Díez I, López-Coronado M, Rodrigues JJPC. Analysis of cloud-based solutions on EHRs systems in different scenarios. J. Med. Syst. 2012;36:3777–3782. doi: 10.1007/s10916-012-9850-2. [PubMed] [CrossRef] [Google Scholar]

103. Rodrigues, J. J. P. C., de la Torre, I., Fernández, G. & López-Coronado, M. Analysis of the security and privacy requirements of cloud-based electronic health records systems. J. Med. Internet Res. 15, e186 (2013). [PMC free article] [PubMed]

104. Kabachinski J. A look at clinical decision support systems. Biomed. Instrum. Technol. 2013;47:432–434. doi: 10.2345/0899-8205-47.5.432. [PubMed] [CrossRef] [Google Scholar]

105. O’Reilly D, Tarride J-E, Goeree R, Lokker C, McKibbon KA. The economics of health information technology in medication management: a systematic review of economic evaluations. J. Am. Med. Inf. Assoc. 2012;19:423–438. doi: 10.1136/amiajnl-2011-000310. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

106. Bright TJ, et al. Effect of clinical decision-support systems: a systematic review. Ann. Intern Med. 2012;157:29–43. doi: 10.7326/0003-4819-157-1-201207030-00450. [PubMed] [CrossRef] [Google Scholar]

107. Jacob V, et al. Cost and economic benefit of clinical decision support systems (CDSS) for cardiovascular disease prevention: a community guide systematic review. J. Am. Med. Inf. Assoc. Jamia. 2017;24:669–676. [PMC free article] [PubMed] [Google Scholar]

108. Main C, et al. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems. Health Technol. Assess. 2010;14:1–227. [PubMed] [Google Scholar]

109. Scheepers-Hoeks, A. M., Grouls, R. J., Neef, C. & Korsten, H. H. Strategy for implementation and first results of advanced clinical decision support in hospital pharmacy practice. Stud. Health Technol. Inform. 148, 142–148 (2009). [PubMed]

110. Edlin Richard, McCabe Christopher, Hulme Claire, Hall Peter, Wright Judy. Cost Effectiveness Modelling for Health Technology Assessment. Cham: Springer International Publishing; 2015. [Google Scholar]

111. Vermeulen KM, et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J. Med, Inf. 2014;83:572–580. doi: 10.1016/j.ijmedinf.2014.05.003. [PubMed] [CrossRef] [Google Scholar]

112. Okumura LM, Veroneze I, Burgardt CI, Fragoso MF. Effects of a computerized provider order entry and a clinical decision support system to improve cefazolin use in surgical prophylaxis: a cost saving analysis. Pharm. Pract. 2016;14:1–7. [PMC free article] [PubMed] [Google Scholar]

113. Osheroff JA, et al. A roadmap for national action on clinical decision support. J. Am. Med. Inf. Assoc. 2007;14:141–145. doi: 10.1197/jamia.M2334. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

114. Greenes, R. A. Clinical Decision Support 2nd edn. The Road to Broad Adoption10.1016/B978-0-12-398476-0.00035-X (2014).

115. Pesek Ksenija., editor. Atherosclerotic Cardiovascular Disease. 2011. [Google Scholar]

116. Bates DW, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J. Am. Med. Inf. Assoc. 2003;10:523–530. doi: 10.1197/jamia.M1370. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

117. Kawamoto K. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. Bmj. 2005;330:765–0. doi: 10.1136/bmj.38398.500764.8F. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

118. Health Level Seven International - Homepage. http://www.hl7.org/. Accessed 29 Aug 2019 (2018).

120. Marco-Ruiz L, Bellika JG. Semantic Interoperability in Clinical Decision Support Systems: A Systematic Review. Stud. Health Technol. Inf. 2015;216:958. [PubMed] [Google Scholar]

121. Angraal, S., Krumholz, H. M. & Schulz, W. L. Blockchain technology: applications in health care. Circ. Cardiovasc. Qual. Outcomes10.1161/CIRCOUTCOMES.117.003800 (2017). [PubMed]

122. Ivan, D. Moving toward a blockchain-based method for the secure storage of patient records. ONC/NIST Use of Blockchain for Healthcare and Research Workshop. Gaithersburg, Maryland, United States: ONC/NIST (2016).

124. Khalifa M. Clinical decision support: strategies for success. Procedia Comput. Sci. 2014;37:422–427. doi: 10.1016/j.procs.2014.08.063. [CrossRef] [Google Scholar]

125. Sittig, D. F. Electronic Health Records: Challenges in Design and Implementation. (CRC Press, 2014).

126. Harper, B. D. & Norman, K. L. Improving User Satisfaction: The Questionnaire for User Interaction Satisfaction Version 5.5. Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, (pp. 224–228), Virginia Beach, VA. (1993).

127. Lewis JR. The system usability scale: past, present, and future. Int J. Hum.-Comput. Interact. 2018;34:577–590. doi: 10.1080/10447318.2018.1455307. [CrossRef] [Google Scholar]

128. Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J. Hum.-Comput. Interact. 1995;7:57–78. doi: 10.1080/10447319509526110. [CrossRef] [Google Scholar]

129. Lewis JR. Measuring perceived usability: the CSUQ, SUS, and UMUX. Int J. Hum.-Comput Interact. 2018;34:1148–1156. doi: 10.1080/10447318.2017.1418805. [CrossRef] [Google Scholar]

Articles from NPJ Digital Medicine are provided here courtesy of Nature Publishing Group