Our news ands science feeds are so filled these days with everything artificial intelligence (AI), from large language models to their impacts on society, that we may miss important studies on other informatics topics, some of which may have lessons for AI. This is the case for a recent randomized controlled trial (RCT) on a hospital readmissions initiative (Donzé, 2023) and an accompanying editorial putting it in larger perspective.(Wachter, 2023)
Some may remember about a decade ago, when “data analytics” was the rage, and health insurance payors were noting with alarm the growing rate of hospital readmissions. The cost and frequency of readmissions was highlighted in a study finding as many as 20% of hospital admissions were readmissions within a month of previous discharge.(Jencks, 2009) Before this, several hospital discharge planning programs had been studied and found to reduce readmissions.(Naylor, 1994; Coleman, 2006; Jack, 2009) This situation led the US Centers for Medicare and Medicaid Services (CMS) to implement the Hospital Readmissions Reduction Program (HRRP) as part of the Affordable Care Act. Starting in 2012, the HRRP required public reporting of readmission rates for three common diseases: myocardial infarction, heart failure, and pneumonia, with hospitals having unusually high rates of readmissions being penalized.
Around the time that the HRRP was implemented, the Health Information Technology for Economic and Clinical Health (HITECH) Act was incentivizing the adoption of the electronic health record (EHR). This provided new unprecedented sources of data, and every predictive analyst set out to find models that used EHR data to predict patients most likely to be readmitted, with the goal of identifying those who could presumably be followed more closely and have readmissions averted. Numerous studies were published using models based on EHR data to predict patients at risk for readmission.(Amarasingham, 2010; Donzé, 2013; Gildersleeve, 2013; Shadmi, 2015)
Despite the plethora of predictive models, few interventions have actually been undertaken that demonstrate improved outcomes of care. One study found that the use of a readmission risk tool intervention reduced risk of readmission for patients with congestive heart failure but not those with acute myocardial infarction or pneumonia (Amarasingham, 2013) Another observational study found that readmissions did decline with the implementation of the new rule initially and for the targeted diseases, but less so for other conditions.(Zuckerman 2016) Other have noted that the program has had marginal benefit (Ody, 2019) and redirecting resources that might be devoted to other quality improvement efforts.(Cram, 2022)
Earlier this year, an RCT was published that assessed a multimodal care intervention aimed at reducing readmissions (Donzé, 2023). Carried out in four medium-to-large teaching hospitals in Switzerland, the study implemented the best-known predictive model for risk of readmission yet found no benefit for an intervention that included it. As noted in the accompanying editorial, just because we can predict something does not mean we can necessarily do something about it.(Wachter, 2023)
Why is this RCT pertinent to AI? Mainly because just being able to predict diagnoses or outcomes is not enough. I have written about this myself in this blog over the years. Whether we are talking about predictive analytics, next-generation data science, or AI, no matter how sophisticated our models or compelling our predictive abilities, we must demonstrate how these systems impact outcomes, whether improved patient health or healthcare system processes.
How do we demonstrate the value of AI in health and healthcare? First, we must implement these systems in the real world. There is a great deal being written about the promise and challenges for implementing AI in clinical settings.(Hightower, 2023) But even implementing AI in the real world is not enough. We must also demonstrate that AI can lead to better outcomes, whether improved health or treatment of disease of patients or better delivery of healthcare services. One perspective to think about this is the continuum of translational research. As with all biomedical advances, we start with the basic science, and demonstrating value in the laboratory, which in this case is the use of curated data sets. The next step is to implement systems in real-world healthcare or community settings. Clearly these are complex interventions.
Ultimately, however, we must demonstrate experimentally that health or healthcare is improved by the AI intervention. The best experimental evidence comes from controlled experiments, ideally RCTs. And granted, such trials may be more complicated than the classic RCT of comparing a medication versus a placebo. These RCTs may involve complex designs, and results may be difficult to interpret if the trial does not show benefit. But building the evidence base for AI is essential, and studies like this from Donzé et al. demonstrate that even the best predictive models may not translate into better outcomes.
References
Amarasingham, R., Moore, B.J., Tabak, Y.P., Drazner, M.H., Clark, C.A., Zhang, S., Reed, W.G., Swanson, T.S., Ma, Y., Halm, E.A., 2010. An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data. Med Care 48, 981–988.
Amarasingham, R., Patel, P.C., Toto, K., Nelson, L.L., Swanson, T.S., Moore, B.J., Xie, B., Zhang, S., Alvarez, K.S., Ma, Y., Drazner, M.H., Kollipara, U., Halm, E.A., 2013. Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf 22, 998–1005.
Coleman, E.A., Parry, C., Chalmers, S., Min, S.-J., 2006. The care transitions intervention: results of a randomized controlled trial. Arch Intern Med 166, 1822–1828.
Cram, P., Wachter, R.M., Landon, B.E., 2022. Readmission Reduction as a Hospital Quality Measure: Time to Move on to More Pressing Concerns? JAMA 328, 1589–1590.
Donzé, J., Aujesky, D., Williams, D., Schnipper, J.L., 2013. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med 173, 632–638.
Donzé, J., John, G., Genné, D., Mancinetti, M., Gouveia, A., Méan, M., Bütikofer, L., Aujesky, D., Schnipper, J., 2023. Effects of a Multimodal Transitional Care Intervention in Patients at High Risk of Readmission: The TARGET-READ Randomized Clinical Trial. JAMA Intern Med 183, 658–668.
Gildersleeve, R., Cooper, P., 2013. Development of an automated, real time surveillance tool for predicting readmissions at a community hospital. Appl Clin Inform 4, 153–169.
Hightower, M., 2023. Is Medicine Ready for AI? N Engl J Med 388, e49.
Jack, B.W., Chetty, V.K., Anthony, D., Greenwald, J.L., Sanchez, G.M., Johnson, A.E., Forsythe, S.R., O’Donnell, J.K., Paasche-Orlow, M.K., Manasseh, C., Martin, S., Culpepper, L., 2009. A reengineered hospital discharge program to decrease rehospitalization: a randomized trial. Ann Intern Med 150, 178–187.
Jencks, S.F., Williams, M.V., Coleman, E.A., 2009. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med 360, 1418–1428.
Naylor, M., Brooten, D., Jones, R., Lavizzo-Mourey, R., Mezey, M., Pauly, M., 1994. Comprehensive discharge planning for the hospitalized elderly. A randomized clinical trial. Ann Intern Med 120, 999–1006.
Ody, C., Msall, L., Dafny, L.S., Grabowski, D.C., Cutler, D.M., 2019. Decreases In Readmissions Credited To Medicare’s Program To Reduce Hospital Readmissions Have Been Overstated. Health Aff (Millwood) 38, 36–43.
Shadmi, E., Flaks-Manov, N., Hoshen, M., Goldman, O., Bitterman, H., Balicer, R.D., 2015. Predicting 30-day readmissions with preadmission electronic health record data. Med Care 53, 283–289.
Wachter, R.M., 2023. The Disappointing Impact of Interventions to Prevent Hospital Readmissions. JAMA Intern Med 183, 668–669.
Zuckerman, R.B., Sheingold, S.H., Orav, E.J., Ruhter, J., Epstein, A.M., 2016. Readmissions, Observation, and the Hospital Readmissions Reduction Program. N Engl J Med 374, 1543–1551.
