Machine learning supports clinicians to safely discharge chest pain patients from the emergency department
Revision - now patented!
Chest pain is one of the most common reasons for a patient to visit the emergency department. Clinicians are understandably risk averse to discharging these patients because they are expected to have less than a 2% miss rate for major adverse cardiac events. This means that clinicians admit many patients who, with hindsight, could have been safely discharged home.
Our goal in this project is to support clinicians with a new discharge protocol to help reduce the number of unnecessary hospital admissions. We find that it’s helpful to formulate this goal in a problem statement.
As an emergency department clinician, I want to know which patients are not likely to experience major adverse cardiac events so that I can safely discharge them home.
Assigning points won’t give you an accurate picture
Many decision tools provide a second opinion for clinicians that are considering a discharge for a patient presenting with chest pain. They are frequently delivered as acronyms (GRACE! TIMI! FRISC! HEART!) to help a clinician remember all of the relevant decision criteria during their busy shift.
This strict focus on simplicity has enabled clinical adoption of these tools. Unfortunately, these stringent constraints result in a much less accurate prediction of major adverse cardiac events. Vituity clinicians implemented the HEART score pathway in a large public health district in California for low risk patients that were presenting with chest pain in 2016. The HEART score by itself had a miss rate in this population of ~ 3,000 encounters of 5.4% (or a sensitivity of 0.946). This is nearly triple the standard to which clinicians are held!
Only computers can handle this level of complexity
This lack of sensitivity led us to a more flexible machine learning approach to identify major adverse cardiac events in our emergency rooms. There is a wealth of information in the electronic medical record that is not incorporated into HEART score, and so we started there. Goodbye checklists! Hello computer!
Offloading the risk calculation away from a human brain means that we can consider thousands of variables that influence a patient’s risk of major adverse cardiac events. For example, the R in HEART score stands for a ‘risk factors’. If a patient has been diagnosed with high cholesterol they are given 1 point. With a computer we can zoom in and build a much more resolved picture. It’s reasonable to believe that having high cholesterol for a number of years would contribute more to cardiac problems than a more recent diagnosis. With a computer we can keep track of the exact number of days between the present and the patient’s diagnosis of high cholesterol. Then, we can include this number of days directly in our machine learning model.
We trained a machine learning model to identify major adverse cardiac events with 700,000 historical encounters of a large public health district in California (2010–2016). We validated this model’s performance retrospectively by applying it to the population of patients from the HEART score pathway (2016-present). In these ~ 3,000 distinct encounters we made the exact same number of high risk predictions to make the comparison fair to the HEART score. We were thrilled to see that our machine learning model’s miss rate was only 1.6% (sensitivity of 0.984). This is lower than the 2% expected miss rate for clinicians and suggests that we could address a gap in the current discharge protocol.
Only timely predictions are useful to physicians
No computer has ever changed a health outcome by itself. In order for our model to be useful we have to get a prediction out of the computer and into the hand of a clinician. For our problem these predictions need to be served to a clinician during their first exam with a patient in the emergency room. We accomplished this goal by building a real-time data warehouse to keep track of what is happening in the emergency department. This warehouse sits on top of streams health level 7 (HL7) messages that come directly from the electronic medical record. This warehouse was a major feat of data engineering that other’s in our team will describe in later posts. From the perspective of a data scientist, it allows me to serve a prediction at the appropriate moment in a patient’s emergency room visit.
We are currently making predictions in real-time in order to prospectively validate that our model maintains the sensitivity we saw in our retrospective validation. These predictions are silent in that they are not given to the clinician, and this ensures our sample won’t be contaminated with clinician interventions. We will integrate into clinical practice when this period is finished by surfacing model risk to clinicians in our real-time ED trackerboard.
All models are wrong, but some are more useful than others
There has been a progression of models that identify major adverse cardiac events in patients presenting with chest pain. Clinician gestalt is a cognitive model that individuals acquire over years as they see similar examples. Electrocardiograms and cardiac enzymes are biological models that help stratify risk when interpreted by a clinician. Rule-based models made expert decisions more accessible to clinical practice by distilling many possible scenarios into a set of rules that can be easily followed. Here, we argue that statistical models of risk are a logical extension of rule-based models in that they are both more accurate and take the burden of calculation away from busy clinicians. We do not expect statistical models to supplant years of research in biological models or clinician gestalt, but instead to support the decision making process with more flexible approximations of risk.
MedAmerica Data Services, a Vituity data company, provides customized data tools and analytic solutions for health care providers and organizations. For more information on MedAmerica Data Services tools and solutions, please send an inquiry to Data@vituity.com.