Information Retrieval (Search) in Health and Biomedicine Still "Springs" Eternal

One of my earliest visions of computers in medicine was the ability to type in a question and get an answer. In 1980s, while everyone in informatics was trying to build expert systems, I followed a different dream, of being able to find clinical information seamlessly. In that decade, however, I never could have imagined being able to pull up something called a Web browser, typing in words, and getting back "pages." Especially as I can do now, with something that fits in my pocket, also makes phone calls, and is connected to something I had not yet heard of in the 1980s (before I started my informatics training) called the Internet.

This fascination guided my early research interests in the area of information retrieval. I write about it now because every spring I teach my course on this topic in the OHSU graduate program, BMI 514/614. (Hence the title of this posting.) My interest in this area resulted in dozens of scientific papers and a textbook, currently in its third edition [1]. Despite the marvel I have for today's modern systems, I always have to ask myself, Why didn't I think of the idea of ranking the output (Web pages) by how many other pages pointed to them? Had I thought of that before a couple Stanford graduate students named Brin and Page, my life might be considerably different. Or at least my wealth!

I suppose one is getting up in the years when you marvel at how things are now relative to how you remember them. I certainly recall "searching" when I was in medical school in the 1980s, which involved thumbing through the giant Index Medicus books on long shelves in the library. You would "link" to the full text by walking to a different part of the library where the journals were. If your needs were really critical, you could call on a librarian for help, who would take your request to a special computer that accessed a database somewhere (which happened to be MEDLINE, from the National Library of Medicine).

I actually did my first on-line searching in the 1980s. I was able to access PaperChase, and later Elhill, through dial-up networks, though at a price. For an even heftier price, you could get access to the full text … at least "text" in monospaced font and no figures or images. The world did advance, and by 1998 you could search Pubmed for free. (Al Gore, who actually deserves more credit in this area than his critics deny him, did the first "free" search.)

Now, of course, searching is ubiquitous. You can't even not do it, since most browsers will throw you into a search engine when you type in an invalid Web address (URL) into your browser. And the world not only searches, but searches for health information. The two major periodic surveys of health information searching show that 80% of Internet users have searched for health information for themselves, their family, or their friends [2, 3].

Of course, like many areas of informatics, while use of systems is ubiquitous, not all of the problems of systems are solved. Indeed, a few years ago I wrote a short piece on this topic [4]. As wonderful as today's search systems are, we still have many areas for improvement. In that paper, I identified four areas where grand challenges remained:

• Content - getting diverse users to the right information for the right task
• Indexing - developing better metadata to get searchers to that proper content
• Linkage - allowing navigation across multiple resources, even those of different publishing entities
• Access - making access as open as possible but still being protective of intellectual property

Just as I could not fathom the World Wide Web in the 1980s, I wonder as I write this in 2011 what the world of search and on-line knowledge access will be a decade or two from now.

References

Hersh, W. (2009). Information Retrieval: A Health and Biomedical Perspective (3rd Edition). New York, NY. Springer.
Fox, S. (2011). Health topics. Washington, DC, Pew Internet & American Life Project. http://www.pewinternet.org/~/media//Files/Reports/2011/PIP_HealthTopics.pdf.
Taylor, H. (2010). "Cyberchondriacs" on the Rise? Those who go online for healthcare information continues to increase. Rochester, NY, Harris Interactive. http://www.harrisinteractive.com/vault/HI-Harris-Poll-Cyberchondriacs-2010-08-04.pdf.
4. Hersh, W. (2008). Ubiquitous but unfinished: grand challenges for information retrieval. Health Information and Libraries Journal, 25(Suppl 1): 90-93.

What is the Evidence Base for Informatics, Health IT, and Related Areas? Some Recent Analyses

The first part of 2011 has brought a number of publications, and subsequent discussion, about the "evidence base" for the efficacy of biomedical and health informatics interventions, including electronic health records. These publications and conversations come against a backdrop of a very poisoned political environment in the United States, where everything about healthcare, including informatics, has become unfortunately very politicized. In this posting, however, I will stick to the science.

The first high-profile study of the year was the on-line posting of the Archives of Internal Medicine paper by Romano and Stafford [1], which I discussed in an earlier posting. The official publication of the paper, as well as letters about it, will be published in May, 2011.

Probably the next most high-profile study was the publication of an update of a systematic review of studies of outcomes from health information technology interventions by Buntin and colleagues [2]. This was actually the second update of an original systematic review that was published in 2006 by Chaudhry and associates [3], the first update of which was published by Goldzweig and colleagues in 2009 [4].

Systematic reviews are comprehensive reviews of all research evidence on a given area or question [5]. When studies are homogeneous enough (e.g., all studies assessing the treatment of hypertension to reduce cardiovascular disease), a mathematical technique known as meta-analysis can be performed to combine results across studies to achieve larger a sample size and more statistical power. But most areas, certainly so in informatics, have research questions too heterogeneous to enable use of meta-analysis. Nonetheless, studies can be categorized to look at general questions asked, such as efficacy of decision support to reduce medical error or access to data in a more timely manner to reduce cost of care.

The three successive systematic reviews [2-4] using relatively similar methodology have summarized outcomes of studies of health information technology (HIT) over particular time periods:

• Chaudhry, 2006 – studies from 1995-2004 [3]
• Goldzweig, 2009 – studies from 2004-2007 [4]
• Buntin, 2011 – studies from 2007-2010 [2]

As with most systematic reviews, these captured a broad net of literature and reviewed it for quality of methodology and its results.

Chaudhry et al. identified 257 studies, with the most benefit shown for:

• Adherence to guideline-based care
• Enhanced surveillance and monitoring
• Decreased medical errors

An interesting caveat of the results that the authors noted was that 25% of the identified studies came from four institutions (Partners Healthcare, Veteran's Administration, Indiana University/Regenstrief Institute, and Vanderbilt University) and there were few studies of commercial systems, raising concerns about generalizability.

In their update, Goldzweig et al. found 179 new studies. They noted comparable results to the study of Chaudhry et al., but also found an increased number of studies of patient-focused applications that ran external to EHR, e.g., Web-based care management. They note a small increase in the number of studies of commercial, off-the-shelf systems, though 20% of studies still came from the four leading institutions. They also found there was still a paucity of cost-benefit analyses.

In the new systematic review, Buntin et al. identified 154 new studies with 278 individual outcome measures. While acknowledging wide divergence of study quality and methodologies, not to mention outcomes studied, they noted that 96 (62%) of studies had positive improvement in one or more aspects of care, with 142 (92%) showing positive or mixed positive-negative outcomes. They found that the studies used quantitative and qualitative approaches, with those using statistical hypothesis testing more likely to have positive outcomes. They slightly redefined “health IT leader” institutions, but noted that a large number (28) still came from these institutions, but did decreased somewhat to 18% of the studies. Somewhat reassuring  was that the “leader” studies did not differ in methods or results from the other studies.

Buntin et al. grouped the outcomes into seven categories, noting document improvement in all of them:

• Access to care
• Preventive care
• Care process
• Patient satisfaction
• Provider satisfaction
• Effectiveness of care
• Efficiency of care

Another bit of evidence from early 2011 was a review of all eHealth systematic reviews took exception to direction and quality of evidence [6]. The authors note that many studies of eHealth, including clinical applications (i.e., health IT), had poor methodology, raising concern over validity of the results. The results echo those of a systematic review I led about telemedicine studies several years ago [7]. One concern about this new review is that its methodology of being a review of reviews might magnify poor evidence. But someone needs to reconcile this review with the one of Buntin et al. [2].

It should be noted that another line of thought has been critical of the experimental approach to evaluation of health IT. Two recent commentaries note that these approaches cannot capture the whole picture of a health IT intervention, especially ones that occur in real-world implementations in complex settings, like states or even whole countries [8, 9]. I acknowledge these criticisms, though would argue back that we should not view these approaches as either-or. There is hopefully plenty of room for all types of disciplined evaluation of informatics, with clinical trials and similar experiments

References

1. Romano, M. and Stafford, R. (2011). Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Archives of Internal Medicine, Epub ahead of print.
2. Buntin, M., Burke, M., et al. (2011). The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Affairs, 30: 464-471.
3. Goldzweig, C., Towfigh, A., et al. (2009). Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28: w282-w293.
4. Chaudhry, B., Wang, J., et al. (2006). Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144: 742-752.
5. Anonymous (2011). Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC, Institute of Medicine.
6. Black, A., Car, J., et al. (2011). The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Medicine, 8(1): e1000387.
7. Hersh, W., Hickam, D., et al. (2006). Diagnosis, access, and outcomes: update of a systematic review on telemedicine services. Journal of Telemedicine & Telecare, 12(Supp 2): 3-31.
8. Greenhalgh, T. and Russell, J. (2010). Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Medicine, 7(11): e1000360.
9. Patrick, J. (2011). The validity of personal experiences in evaluating HIT. Applied Clinical Informatics, 1: 462-465.

Looking Back, Moving Forward

This week marks a year ago that I woke up (on the morning of Friday, April 2nd, to be precise) to find emails in my inbox telling me that Oregon Health & Science University (OHSU) had received our two awards from the Office of the National Coordinator for Health IT (ONC) Workforce Development Program. As most readers know, those programs are going well, and I am planning to provide my occasional updates of our efforts in the programs here in the coming weeks ahead. A succinct news report of the programs, for which I was interviewed, is available from the California Health Care Foundation (CHCF).

Another anniversary of sorts is for this blog, which has now been in existence a little over two years (since March 2, 2009, to be precise). I have enjoyed having this forum to share my thoughts about topics of interest and passion to me. I have tried to create thoughtful pieces that explore various issues, and not just brief streams of consciousness.

The year 2011 also is another anniversary year, which is the 15th year of informatics degree programs being offered by OHSU. In 1996, we opened the doors to our initial Master of Science degree. Of course, we have added a number of other degrees since then, such as our Master of Biomedical Informatics (non-thesis, professional master's), PhD, and Graduate Certificate. To celebrate the 15-year anniversary, as well as the first two groups of graduates from our ONC funding, we are planning to hold a celebration in September. The event will be open to the public and is scheduled to take place on September 9-10, 2011. All alumni, students, faculty, and friends of the program will be invited, with alumni being able to present about what they are currently doing, along with a number of other keynote speakers. (Save the date!)

Of course, we are not resting on our laurels, and are quite busy with our current work, the totality of which would be much longer than anyone would want to read. I am happy to announce that the 10x10 ("ten by ten") program continues going strong, with  a number of new offerings planned to start in the next few months. One of the offerings is a general AMIA offering but the rest demonstrate the partnerships that we have built for specific offerings. As with all 10x10 courses, the offerings include the basic on-line portion of the course and an in-person session often associated with a professional meeting.

They include:

• Regular AMIA offering aimed at all audiences, starting April 27, 2011 with in-person session at any AMIA national meeting in the next year. (Next meeting in Washington, DC, with in-person session on October 23, 2011)
• Offering focused on dietitians and the area of nutrition informatics, in partnership with the American Dietetic Association (ADA), starting April 13, 2011, with the in-person session at the ADA meeting in San Diego, CA on September 24, 2011
• Offering in Singapore in partnership with Gateway Consulting, starting May 2, 2011, with the in-person session in Singapore (!) on September 14, 2011
• Offering focused on emergency physicians, in partnership with the American College of Emergency Physicians (ACEP), starting June 29, 2011, with the in-person at the ACEP meeting in San Francisco, CA on October 14, 2011