Natural Language Processing: A Dream That Won't Die … and Shouldn't

One of the longest-standing dreams of informatics, dating back to the early (i.e, 1960s) era of artificial intelligence, is the use of natural language processing (NLP) to extract data about patients from clinical narrative data (e.g., progress notes, discharge summaries, etc.) in the electronic health record (EHR). The notion that you can take the narrative language of clinicians and turn it into concrete facts that can be used for clinical decision support, clinical research, quality measurement, surveillance, etc. is immensely appealing.

Alas, that dream, at least in a generalizable way, is still a dream. You can count a number of my published papers over the years as a few among the many valiant efforts. Unfortunately, the variability (or some might say mangling, especially by physicians) of language, along with the hidden context and meaning "between the lines," makes NLP a very difficult task to program in a computer.

Some, however, have managed to succeed in focused ways. For example, generalizable decision support also never succeeded but it has been found that focused decision support works quite well and is used in EHRs daily. Likewise, there have a number of focused areas where NLP has provided useful data for clinical processes.

It is in this context that I am pleased to report on another contribution to the literature of clinical NLP, which is a paper that appeared in a recent issue of the Journal of the American Medical Informatics Association (JAMIA) and is lead-authored by a former student, Mary Stanfill [1]. I am a co-author. It is always a thrill to see a student publish a peer-reviewed paper, especially one that started as a term paper in one of my courses, advanced to a capstone project in a master's degree, and ultimately ended up in one of the leading journals in our field.

This paper also makes a valuable contribution of being a systematic review of all studies that report results of "automated coding and classification." The analysis shows that there have been many efforts performed using many methods in a variety of clinical domains, with a wide range of results. Of course, this gets to a gripe I have had with clinical NLP and related text mining researchers over the years, which is that evaluation studies have not advanced much beyond measuring the accuracy (e.g., recall, precision, sensitivity, specificity, etc.) of how well systems identify concepts in the text [2]. I would prefer to see the next step in systems being evaluated, such as how well NLP can impact the tasks it might be used for, such as quality measurement programs or facilitating clinical research studies. This would be akin to the "task-oriented" studies of information retrieval systems I performed years ago, which focused on how well searchers completed tasks using retrieval systems rather than just measuring how many relevant articles they retrieved [3].

The good news is that systems using NLP are starting to be deployed in operational clinical settings or clinical and translational research programs, and there is an ever-increasing amount of real data in electronic form for them to use. In addition to a growing number of individual studies, there are also large-scale projects of which NLP is a significant part. There include:

• Informatics for Integrating Biology and the Bedside (i2b2) - a long-standing project to facilitate the use of clinical data for genomic and clinical research. One of its activities includes a yearly challenge evaluation that allows research to compare systems and results on a common task. The i2b2 challenge has looked at automatic de-identification [4], identification of smoking status [5], recognition of obesity and co-morbidities [6], and extraction of medication information [7].
• Electronic Medical Records and Genomics (eMERGE) Network - a multi-center project focused on the use of data in EHRs to facilitate the study of how genetic variability contributes to health and disease [8]. One of the foci includes the use of NLP for extracting data from clinical narratives and integrating it with other data in the clinical record. One accomplishment of this research to date has been the ability to replicate four of seven known gene-disease associations [9].
• SHARP 4 - one of the four collaborative research centers being funded under the HITECH Program to facilitate meaningful use of EHRs, with a focus on secondary use of EHR data.

Another development is the launching this year of a medical records track in the Text Retrieval Conference (TREC) annual information retrieval challenge evaluation. The track will use de-identified records developed by Wendy Chapman and colleagues.

It is also impossible to discuss this topic without acknowledging the discussion around the IBM Watson question-answering system, which recently proved its mettle in a television game show Jeopardy match [10]. IBM has announced some research partnerships that will apply Watson to medical data. This is an interesting research area, but we will need to see real research results to back up the hype [11].

While there are still challenges for clinical NLP, I believe we are seeing a convergence of new methods coupled with growing needs to make use of the increasing volume of clinical data as well as our desire to facilitate re-use of that data for many purposes, such as clinical decision support, quality measurement and improvement, clinical research, and public health reporting and surveillance. While there may be generalizable approaches yet to be discovered, I suspect that evolution will be much like clinical decision support, which has been more successful when engineered to specific domain areas. But as we have also seen with clinical decision support, the ability to perform those specific tasks successfully will be highly valuable to healthcare.

References

1. Stanfill, M., Williams, M., et al. (2010). A systematic literature review of automated clinical coding and classification systems. Journal of the American Medical Informatics Association, 17: 646-651.
2. Hersh, W. (2005). Evaluation of biomedical text mining systems: lessons learned from information retrieval. Briefings in Bioinformatics, 6: 344-356.
3. Hersh, W., Crabtree, M., et al. (2002). Factors associated with success for searching MEDLINE and applying evidence to answer clinical questions. Journal of the American Medical Informatics Association, 9: 283-293.
4. Uzuner, O., Luo, Y., et al. (2007). Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association, 14: 550-563.
5. Uzuner, O., Goldstein, I., et al. (2008). Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association, 15: 14-24.
6. Uzuner, O. (2009). Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association, 16: 561-570.
7. Uzuner, O., Solti, I., et al. (2010). Extracting medication information from clinical text. Journal of the American Medical Informatics Association, 17: 514-518.
8. McCarty, C., Chisholm, R., et al. (2010). The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Genomics, 4(1): 13. http://www.biomedcentral.com/1755-8794/4/13.
9. Denny, J., Ritchie, M., et al. (2010). PheWAS: Demonstrating the feasibility of a phenome-wide scan to discover gene-disease associations. Bioinformatics, 26: 1205-1210.
10. Ferrucci, D., Brown, E., et al. (2010). Building Watson: an overview of the DeepQA Project. AI Magazine, 31(3): 59-79. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303.
11. Anonymous (2011). IBM to Collaborate with Nuance to Apply IBM’s "Watson" Analytics Technology to Healthcare. Armonk, NY, IBM. http://www-03.ibm.com/press/us/en/pressrelease/33726.wss.

Electronic Health Records Do Not Impact the Quality of Healthcare?

The headlines have blared the news this week that a new study published in Archives of Internal Medicine showed that electronic health records (EHRs) in the ambulatory setting do not appear to lead to higher quality patient care [1]. This in turn has led many leading news organizations to have stories with headlines such as, Stanford researchers find EHRs don't boost care quality.

For those of us who work in informatics, this is a pretty serious finding. As responsible scientists and citizens, we cannot ignore negative results about the work we do. However, we also have an obligation to place this work in the larger context of all research on the relationship between health information technology (HIT) and quality of medical care.

Like almost all science that gets reported in the general media, there is more to this study than what is described in the headlines and news reports. The study was published in a prestigious medical journal by two Stanford researchers. The implementation of the research methods they used appears to be sound. There is no reason to believe that the results obtained do not derive from the methods used.

However, there are serious limitations to this type of study and to the data resources used to answer the researchers' question, which was whether ambulatory EHRs that include clinical decision support (CDS) lead to improved quality of medical care delivered. While I do believe this study has a place in the evidence base of HIT, it suffers from limitations that are inherent in studies like this that that are observational, correlational, and retrospective. This study used a data source collected for other purposes, the National Ambulatory Medical Care Survey, and compared physicians who were identified users of CDS with those who were not to see if there were differences in the quality of care they provided based on 20 process quality measures. The results found there were no differences between the groups, i.e., those using EHRs and CDS did not deliver higher quality care than those not using them.

Before delving into the details, it is also worth noting that this study is not the first to apply this methodology. Within the last couple months, two other studies have used a similar approach to assess associations between quality measures and hospital EHR adoption [2] and computerized provider order entry [3], giving mixed results, i.e., some measures showing benefit and others not. In addition, Archives of Internal Medicine published another study using this sort of approach in 2009 showing that hospital notes, test results, order entry, and decision support were variably associated with improved patient outcomes and reduced costs [4]. (It was surprising to see the latter not referenced in the article.) If we were to take all of these studies as definitive, then we might conclude that EHRs usage in hospitals improves quality of care, even if EHR usage does not in ambulatory settings.

But whether the results are favorable or not, it is important to understand some serious limitations in these types of studies and this one in particular. A first limitation is that the study looks at correlation, which does not mean causality. This was an observational and not an experimental study. The data used for the study was not collected for the purposes of assessing the quality of care by EHRs. As with any correlational study, there may be confounders that cause the correlation or lack of it. As we know from evidence-based medicine, the best study design to assess causality is an experimental randomized controlled trial. Indeed, such studies have been done and many have found that EHRs do lead to improvements in quality of care. There have been several systematic reviews of such studies noting that while some of the studies suffer from methodologic limitations, others are well-designed and do demonstrate positive value for various aspects of EHR and CDS [5-7]. There is a continuing stream of such studies and two have been published in the last couple months. One showed that an EHR with targeted CDS led to increased improvements in glucose control in diabetics [8], while another found that a real-time alert cut inappropriate use of D-dimer testing by 70% [9]. Not all such studies demonstrate positive results, but enough to do to show that there is value in the well-informed use of HIT.

A second limitation of this study is the quality measures used. Quality measures are of two general types, process and outcome. Process measures look at what was done, such as ordering a certain test or prescribing a specific treatment. Outcome measures look at the actual clinical outcomes of the patient, e.g., whether there was a reduction of mortality, complications, or cost. It is a fair criticism of the current state of the healthcare quality movement that most measures used (including those in the meaningful use criteria) are process measures that may or may not result in improved patient outcomes.

A third limitation of this study is that we do not know whether the physicians using EHRs and CDS had decision support in place to impact the specific quality measures that the researchers studied. While the quality measures are important process measures for physicians to adhere to, they may not be amenable to CDS generally or the CDS used in these systems.

A fourth limitation is that the study assesses episodes of care and not longitudinal care over time. This may not portray an accurate picture of a physician's practice.

A fifth limitation is that the data analyzed was collected in 2005-2007. While EHRs and CDS were available at that time, they were less widely used and less mature at that time.

Finally, we have no idea how well trained these physicians were at using the CDS that they had. We know that success with HIT is based on many factors that go well beyond the technology itself, such as proper implementation and training. Well-designed research must address these factors too.

There are also some other limitations to the study that are discussed in an accompanying editorial that unfortunately few people, especially those who get news of the study from news reports as opposed to the journal itself, will read. The editorial writers appropriately point out that other studies, including experimental ones, have shown value for HIT interventions.

The results of this study are a legitimate addition to the evidence base of informatics and cannot be dismissed out of hand. However, these findings must take their place in the proper context of all research on HIT. If nothing else, this study highlights the need for more and better research to truly identify where HIT helps, has no impact, or outright harms patients.

References
1. Romano MJ and Stafford RS, Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Archives of Internal Medicine, 2011: Epub ahead of print.
2. Jones SS, Adams JL, Schneider EC, Ringel JS, and McGlynn EA, Electronic health record adoption and quality improvement in US hospitals. American Journal of Managed Care, 2010. 16: SP64-SP72.
3. Kazley AS and Diana ML, Hospital computerized provider order entry adoption and quality: an examination of the United States. Health Care Management Review, 2011. 36: 86-94.
4. Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, and Powe NR, Clinical information technologies and inpatient outcomes: a multiple hospital study. Archives of Internal Medicine, 2009. 169: 108-114.
5. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al., Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 2005. 293: 1223-1238.
6. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al., Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 2006. 144: 742-752.
7. Goldzweig CL, Towfigh A, Maglione M, and Shekelle PG, Costs and benefits of health information technology: new trends from the literature. Health Affairs, 2009. 28: w282-w293.
8. O'Connor PJ, Sperl-Hillen JM, Rush WA, Johnson PE, Amundson GH, Asche SE, et al., Impact of electronic health record clinical decision support on diabetes care: a randomized trial. Annals of Family Medicine, 2011. 9: 12-21.
9. Palen TE, Price DW, Snyder AJ, and Shetterly SM, Computerized alert reduced D-dimer testing in the elderly. American Journal of Managed Care, 2010. 16: e267-e275.