First Year of the OHSU University-Based Training (UBT) Program

It has been a little over a year since Oregon Health & Science University (OHSU) was awarded two grants from the Office of the National Coordinator for Health IT (ONC) Workforce Development Program. Activity on these projects has been a major part of the work in our department, and certainly of my time, over this period. In this posting, I will report on our University-Based Training (UBT) Program. In a later posting, I will report on our work on the other funded project, the Curriculum Development Centers/National Training & Dissemination Center Program.

OHSU was one of nine universities (or consortia thereof) awarded a UBT grant. We have met all of our goals and timelines so far for the project. The gist of our funded proposal was to enroll students into our Graduate Certificate and Master of Biomedical Informatics (MBI) programs, with additional course requirements based on the specific ONC workforce roles. As OHSU is on an academic quarter system, Graduate Certificate (classified by ONC as Type 1) students are expected to complete the program in an accelerated part-time status in four quarters (one year) while MBI (classified by OC as Type 2) students are expected to complete the Master's program in six quarters (one and a half years) as full-time students. We were awarded $3.08 million to fund 135 Type 1 and 13 Type 2 students over three years through 2013. The "ad" on this page links to the Web page describing the program.

We accepted 12 Type 1 students to start in the summer quarter of 2010, with 11 of those students expected to graduate in June, 2011, along with two additional students who completed the program in an accelerated manner. We will also have one Type 2 student graduating in June, 2011. These 14 graduates will be eligible (and encouraged!) to attend the OHSU June 6, 2011 Commencement.

We have an additional 74 Type 1 students in the pipeline who started the program in the fall (34), winter (26), and spring (14) quarters. These students, along with eight Master's students, are for the most part on track to graduate on time.

The students we have accepted have a great deal of geographic and occupational diversity. Similar to our distance learning program in general, our UBT students reside all across the United States. (We actually have distance learning students living in 40 states as well as six countries.) Our UBT students reside in 20 different states, with some over-representation in our region, probably reflecting proportions of applicants. Those states with more than one student in our UBT program include:

• Oregon - 39 (49%)
• Washington - 7 (9%)
• California - 4 (5%)
• New York - 4 (5%)
• Texas - 3 (4%)
• Maryland - 3 (4%)
• Tennessee - 2 (3%)
• Utah - 2 (3%)
• Virginia - 2 (3%)
• Minnesota - 2 (3%)
• Wisconsin - 2 (3%)

Our students also come from many diverse career backgrounds. While the majority come from healthcare fields, a decent-sized minority do not, and some have highly technical backgrounds who are coming to learn how information technology is applied in healthcare settings. The backgrounds with more than one representative include:

• Medicine (Physician) - 16 (20%)
• Nursing - 13 (16%)
• Business/Management - 10 (13%)
• Liberal Arts/Humanities - 6 (8%)
• Computer Science - 6 (8%)
• Public Health - 4 (5%)
• Biochemistry/Biology/Chemistry - 4 (5%)
• Finance/Accounting - 2 (3%)
• Health Information Mgmt - 2 (3%)
• Healthcare Management/Administration - 2 (3%)

Our students also have a variety of highest degrees, with over half having a graduate-level degree already. The distribution of highest degrees is as follows:

• Bachelors - 34 (44.1%)
• Masters - 24 (31.1%)
• MD - 16 (20.8%)
• PhD - 3 (3.9%)
• Other healthcare doctorate - 1 (1.3%)

One of the challenges we have faced is the competitive admissions process. We have had many more qualified applicants than we have funded positions for, so we have not been to fund some highly qualified applicants. The rate of acceptance has been 34 out of 162 (21.0%) for the fall, 26 out of 102 (25.5%) for the winter, and 14 out of 73 (19.2%) for the spring.

Moving forward, we are on track to have an additional 34 Type 1 graduates at the end of the summer quarter in early September, 2011. On September 9-10, OHSU plans to hold an informatics program reunion event, celebrating the 15 year anniversary of our first informatics degree program and the first graduates of our UBT program. Additional students will graduate later this year and into 2012, including our initial cohort of MBI students.

All told, we have committed 78 of our 135 (57.8%) Type 1 slots and eight of our 13 (61.5%) Type 2 slots. We are taking the summer quarter off for new admissions and will be admitting Type 1 and Type 2 students starting again in the fall quarter. We will award the rest of our funded slots during the 2011-2012 academic year, aiming to have everyone complete the program by the end of grant in April, 2013. During this time, our existing program is still operational, and those not awarded UBT funding can still enroll as self-funded students.

We have also implemented practicum (for Graduate Certificate students) and internship (for Master's students) programs . These programs are being administered by an Internship Coordinator whom we have hired. Students are required to find their own practicum or internship, although we help them however we can. The hosting organizations so far include health care organizations, regional extension centers, and vendors.

Another hire is our career counselor, who will help students identify and apply for jobs. We also hope this individual will collaborate with the internship coordinator as well as lay the foundation for continued relationships with employers beyond the end of the UBT funding.

All told, we are pleased with what we have accomplished in the ONC UBT program. We hope this will lead to a sustainable increased interest in biomedical informatics education and careers beyond the end of the grant itself.

Professional Science Masters: The Direction for Masters-Level Professional Degrees in Informatics?

This week, the Department of Medical Informatics & Clinical Epidemiology (DMICE) of Oregon Health & Science University (OHSU) is hosting a regional workshop focused on Professional Science Masters (PSM) degrees and programs. While attendees will come from across the Pacific Northwest, the Oregon University System (OUS) is moving forward with development of a statewide program. We are interested in exploring whether our Master of Biomedical Informatics (MBI) might fit the bill to transform into a PSM. For more information on what a PSM is, see their Web site.

PSM programs are professional science degrees with three additional attributes:

1. "Plus" courses that provide the student skills for working in industry settings, such as business and management, writing and communications, and others
2. A rigorous internship program that replaces the traditional master's thesis or capstone
3. Guidance by an external advisory committee from industry that oversee the curriculum and/or participate in the internship program

DMICE offers graduate-level programs in the field of biomedical informatics. Although we are not formally a PSM, our existing programs, especially our MBI degree, have many of the attributes required of a PSM, namely the "plus" courses, an internship program, and an external advisory committee. We changed an MBI program requirement last year that allows a structured internship to be acceptable as the program capstone.

We were actually exploring the PSM option when the large amount of funding from American Recovery & Reinvestment Act (ARRA) for investment in health information technology came along and sidetracked these efforts. Of course, our Office of the National Coordinator for Health IT (ONC) University-Based Training (UBT) grant has many conceptual overlaps with the PSM concept, with its goal of producing informatics professionals who will develop, implement, and lead electronic health record (EHR) adoption in healthcare settings.

Of course, our informatics program is focused on more than EHR adoption, even though that is the largest need. But there are plenty of other critical needs for informatics in health and biomedicine, including in genomics, clinical and translational research, public health, consumer health, and even other clinical applications, such as telemedicine. As the UBT program reaches a steady state, and with it winding down in 2013, we are now reconsidering again the transformation of the program to an official PSM. This week's workshop will help inform our next steps.

Overview of the OHSU Biomedical Informatics Program

People sometimes ask me for a big picture overview of all the programs available in the Biomedical Informatics Graduate Program in the Department of Medical Informatics & Clinical Epidemiology (DMICE) at Oregon Health & Science University (OHSU). I provide that in this posting.

Biomedical informatics is the field that uses information and related technologies to advance individual health, healthcare, public health, and biomedical research. Students enter with a variety of backgrounds and upon graduation take jobs in a diverse array of settings, including healthcare organizations, industry, research labs, and public health agencies. The OHSU program has offerings along many dimensions.

One dimension is the degree/certificate type:

• Doctor of Philosophy (PhD) in Biomedical Informatics
• Master of Science (MS) in Biomedical Informatics
• Master of Biomedical Informatics (MBI)
• Graduate Certificate (GC) in Biomedical Informatics

A second dimension is the program track:

• Clinical Informatics (CI) - focus on health care, individual health, and public health
• Bioinformatics and Computational Biology (BCB) - focus on computational aspects of genomics and molecular biology, especially their relation to human health
• Health Information Management (HIM) - focus on Registered Health Information Administrator (RHIA) certification

A third dimension is whether the program is on-campus (oc) or on-line (ol), although the two can be co-mingled, especially by local students in the Portland area. The GC program can be done completely on-line, while the MBI program done on-line requires the student to take two on-campus "short" (one week) courses.

The following table shows the degree/certificate and track dimensions, with each cell indicating whether or not the program is offered on-campus or on-line.

Track Degree	CI	BCB	HIM
PhD	oc	oc
MS	oc	oc	oc
MBI	oc/ol	oc	oc/ol
GC	oc/ol		oc/ol

Where does the 10x10 ("ten by ten") program fit into this? The 10x10 curriculum is essentially equivalent to the introductory course (BMI 510) in the CI and HIM tracks.

More information is available on our program Web site: http://www.ohsu.edu/informatics/

Information Retrieval (Search) in Health and Biomedicine Still "Springs" Eternal

One of my earliest visions of computers in medicine was the ability to type in a question and get an answer. In 1980s, while everyone in informatics was trying to build expert systems, I followed a different dream, of being able to find clinical information seamlessly. In that decade, however, I never could have imagined being able to pull up something called a Web browser, typing in words, and getting back "pages." Especially as I can do now, with something that fits in my pocket, also makes phone calls, and is connected to something I had not yet heard of in the 1980s (before I started my informatics training) called the Internet.

This fascination guided my early research interests in the area of information retrieval. I write about it now because every spring I teach my course on this topic in the OHSU graduate program, BMI 514/614. (Hence the title of this posting.) My interest in this area resulted in dozens of scientific papers and a textbook, currently in its third edition [1]. Despite the marvel I have for today's modern systems, I always have to ask myself, Why didn't I think of the idea of ranking the output (Web pages) by how many other pages pointed to them? Had I thought of that before a couple Stanford graduate students named Brin and Page, my life might be considerably different. Or at least my wealth!

I suppose one is getting up in the years when you marvel at how things are now relative to how you remember them. I certainly recall "searching" when I was in medical school in the 1980s, which involved thumbing through the giant Index Medicus books on long shelves in the library. You would "link" to the full text by walking to a different part of the library where the journals were. If your needs were really critical, you could call on a librarian for help, who would take your request to a special computer that accessed a database somewhere (which happened to be MEDLINE, from the National Library of Medicine).

I actually did my first on-line searching in the 1980s. I was able to access PaperChase, and later Elhill, through dial-up networks, though at a price. For an even heftier price, you could get access to the full text … at least "text" in monospaced font and no figures or images. The world did advance, and by 1998 you could search Pubmed for free. (Al Gore, who actually deserves more credit in this area than his critics deny him, did the first "free" search.)

Now, of course, searching is ubiquitous. You can't even not do it, since most browsers will throw you into a search engine when you type in an invalid Web address (URL) into your browser. And the world not only searches, but searches for health information. The two major periodic surveys of health information searching show that 80% of Internet users have searched for health information for themselves, their family, or their friends [2, 3].

Of course, like many areas of informatics, while use of systems is ubiquitous, not all of the problems of systems are solved. Indeed, a few years ago I wrote a short piece on this topic [4]. As wonderful as today's search systems are, we still have many areas for improvement. In that paper, I identified four areas where grand challenges remained:

• Content - getting diverse users to the right information for the right task
• Indexing - developing better metadata to get searchers to that proper content
• Linkage - allowing navigation across multiple resources, even those of different publishing entities
• Access - making access as open as possible but still being protective of intellectual property

Just as I could not fathom the World Wide Web in the 1980s, I wonder as I write this in 2011 what the world of search and on-line knowledge access will be a decade or two from now.

References

Hersh, W. (2009). Information Retrieval: A Health and Biomedical Perspective (3rd Edition). New York, NY. Springer.
Fox, S. (2011). Health topics. Washington, DC, Pew Internet & American Life Project. http://www.pewinternet.org/~/media//Files/Reports/2011/PIP_HealthTopics.pdf.
Taylor, H. (2010). "Cyberchondriacs" on the Rise? Those who go online for healthcare information continues to increase. Rochester, NY, Harris Interactive. http://www.harrisinteractive.com/vault/HI-Harris-Poll-Cyberchondriacs-2010-08-04.pdf.
4. Hersh, W. (2008). Ubiquitous but unfinished: grand challenges for information retrieval. Health Information and Libraries Journal, 25(Suppl 1): 90-93.

What is the Evidence Base for Informatics, Health IT, and Related Areas? Some Recent Analyses

The first part of 2011 has brought a number of publications, and subsequent discussion, about the "evidence base" for the efficacy of biomedical and health informatics interventions, including electronic health records. These publications and conversations come against a backdrop of a very poisoned political environment in the United States, where everything about healthcare, including informatics, has become unfortunately very politicized. In this posting, however, I will stick to the science.

The first high-profile study of the year was the on-line posting of the Archives of Internal Medicine paper by Romano and Stafford [1], which I discussed in an earlier posting. The official publication of the paper, as well as letters about it, will be published in May, 2011.

Probably the next most high-profile study was the publication of an update of a systematic review of studies of outcomes from health information technology interventions by Buntin and colleagues [2]. This was actually the second update of an original systematic review that was published in 2006 by Chaudhry and associates [3], the first update of which was published by Goldzweig and colleagues in 2009 [4].

Systematic reviews are comprehensive reviews of all research evidence on a given area or question [5]. When studies are homogeneous enough (e.g., all studies assessing the treatment of hypertension to reduce cardiovascular disease), a mathematical technique known as meta-analysis can be performed to combine results across studies to achieve larger a sample size and more statistical power. But most areas, certainly so in informatics, have research questions too heterogeneous to enable use of meta-analysis. Nonetheless, studies can be categorized to look at general questions asked, such as efficacy of decision support to reduce medical error or access to data in a more timely manner to reduce cost of care.

The three successive systematic reviews [2-4] using relatively similar methodology have summarized outcomes of studies of health information technology (HIT) over particular time periods:

• Chaudhry, 2006 – studies from 1995-2004 [3]
• Goldzweig, 2009 – studies from 2004-2007 [4]
• Buntin, 2011 – studies from 2007-2010 [2]

As with most systematic reviews, these captured a broad net of literature and reviewed it for quality of methodology and its results.

Chaudhry et al. identified 257 studies, with the most benefit shown for:

• Adherence to guideline-based care
• Enhanced surveillance and monitoring
• Decreased medical errors

An interesting caveat of the results that the authors noted was that 25% of the identified studies came from four institutions (Partners Healthcare, Veteran's Administration, Indiana University/Regenstrief Institute, and Vanderbilt University) and there were few studies of commercial systems, raising concerns about generalizability.

In their update, Goldzweig et al. found 179 new studies. They noted comparable results to the study of Chaudhry et al., but also found an increased number of studies of patient-focused applications that ran external to EHR, e.g., Web-based care management. They note a small increase in the number of studies of commercial, off-the-shelf systems, though 20% of studies still came from the four leading institutions. They also found there was still a paucity of cost-benefit analyses.

In the new systematic review, Buntin et al. identified 154 new studies with 278 individual outcome measures. While acknowledging wide divergence of study quality and methodologies, not to mention outcomes studied, they noted that 96 (62%) of studies had positive improvement in one or more aspects of care, with 142 (92%) showing positive or mixed positive-negative outcomes. They found that the studies used quantitative and qualitative approaches, with those using statistical hypothesis testing more likely to have positive outcomes. They slightly redefined “health IT leader” institutions, but noted that a large number (28) still came from these institutions, but did decreased somewhat to 18% of the studies. Somewhat reassuring  was that the “leader” studies did not differ in methods or results from the other studies.

Buntin et al. grouped the outcomes into seven categories, noting document improvement in all of them:

• Access to care
• Preventive care
• Care process
• Patient satisfaction
• Provider satisfaction
• Effectiveness of care
• Efficiency of care

Another bit of evidence from early 2011 was a review of all eHealth systematic reviews took exception to direction and quality of evidence [6]. The authors note that many studies of eHealth, including clinical applications (i.e., health IT), had poor methodology, raising concern over validity of the results. The results echo those of a systematic review I led about telemedicine studies several years ago [7]. One concern about this new review is that its methodology of being a review of reviews might magnify poor evidence. But someone needs to reconcile this review with the one of Buntin et al. [2].

It should be noted that another line of thought has been critical of the experimental approach to evaluation of health IT. Two recent commentaries note that these approaches cannot capture the whole picture of a health IT intervention, especially ones that occur in real-world implementations in complex settings, like states or even whole countries [8, 9]. I acknowledge these criticisms, though would argue back that we should not view these approaches as either-or. There is hopefully plenty of room for all types of disciplined evaluation of informatics, with clinical trials and similar experiments

References

1. Romano, M. and Stafford, R. (2011). Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Archives of Internal Medicine, Epub ahead of print.
2. Buntin, M., Burke, M., et al. (2011). The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Affairs, 30: 464-471.
3. Goldzweig, C., Towfigh, A., et al. (2009). Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28: w282-w293.
4. Chaudhry, B., Wang, J., et al. (2006). Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144: 742-752.
5. Anonymous (2011). Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC, Institute of Medicine.
6. Black, A., Car, J., et al. (2011). The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Medicine, 8(1): e1000387.
7. Hersh, W., Hickam, D., et al. (2006). Diagnosis, access, and outcomes: update of a systematic review on telemedicine services. Journal of Telemedicine & Telecare, 12(Supp 2): 3-31.
8. Greenhalgh, T. and Russell, J. (2010). Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Medicine, 7(11): e1000360.
9. Patrick, J. (2011). The validity of personal experiences in evaluating HIT. Applied Clinical Informatics, 1: 462-465.

Looking Back, Moving Forward

This week marks a year ago that I woke up (on the morning of Friday, April 2nd, to be precise) to find emails in my inbox telling me that Oregon Health & Science University (OHSU) had received our two awards from the Office of the National Coordinator for Health IT (ONC) Workforce Development Program. As most readers know, those programs are going well, and I am planning to provide my occasional updates of our efforts in the programs here in the coming weeks ahead. A succinct news report of the programs, for which I was interviewed, is available from the California Health Care Foundation (CHCF).

Another anniversary of sorts is for this blog, which has now been in existence a little over two years (since March 2, 2009, to be precise). I have enjoyed having this forum to share my thoughts about topics of interest and passion to me. I have tried to create thoughtful pieces that explore various issues, and not just brief streams of consciousness.

The year 2011 also is another anniversary year, which is the 15th year of informatics degree programs being offered by OHSU. In 1996, we opened the doors to our initial Master of Science degree. Of course, we have added a number of other degrees since then, such as our Master of Biomedical Informatics (non-thesis, professional master's), PhD, and Graduate Certificate. To celebrate the 15-year anniversary, as well as the first two groups of graduates from our ONC funding, we are planning to hold a celebration in September. The event will be open to the public and is scheduled to take place on September 9-10, 2011. All alumni, students, faculty, and friends of the program will be invited, with alumni being able to present about what they are currently doing, along with a number of other keynote speakers. (Save the date!)

Of course, we are not resting on our laurels, and are quite busy with our current work, the totality of which would be much longer than anyone would want to read. I am happy to announce that the 10x10 ("ten by ten") program continues going strong, with  a number of new offerings planned to start in the next few months. One of the offerings is a general AMIA offering but the rest demonstrate the partnerships that we have built for specific offerings. As with all 10x10 courses, the offerings include the basic on-line portion of the course and an in-person session often associated with a professional meeting.

They include:

• Regular AMIA offering aimed at all audiences, starting April 27, 2011 with in-person session at any AMIA national meeting in the next year. (Next meeting in Washington, DC, with in-person session on October 23, 2011)
• Offering focused on dietitians and the area of nutrition informatics, in partnership with the American Dietetic Association (ADA), starting April 13, 2011, with the in-person session at the ADA meeting in San Diego, CA on September 24, 2011
• Offering in Singapore in partnership with Gateway Consulting, starting May 2, 2011, with the in-person session in Singapore (!) on September 14, 2011
• Offering focused on emergency physicians, in partnership with the American College of Emergency Physicians (ACEP), starting June 29, 2011, with the in-person at the ACEP meeting in San Francisco, CA on October 14, 2011

Honorable Mention

Who should be the next National Coordinator for Health Information Technology (HIT), i.e., the Director of the Office of the National Coordinator for HIT (ONC)? My name appeared recently on a list of 24 individuals nominated as a potential replacement for the current National Coordinator, Dr. David Blumenthal, who is leaving the post next month to return to academia at Harvard University.

Well, the results of the voting are in, and I was flattered to finish fifth, capturing 5.6% of the 736 votes cast. The winner of the poll was Jessica Kahn, Technical Director for HIT for Medicaid at the Centers for Medicare and Medicaid Services. Following in a close second was Dr. Marc Chasin. Vice President and Chief Medical Informatics Officer at St. Luke's Health System in Boise, Idaho. I am delighted to report that Dr. Chasin is a former student of mine, having taken the 10x10 ("ten by ten") course.

I have to admit that I am ordinarily unswayed by magazine or Web polls that are completely unscientific and really just popularity contests. Still, I was flattered to be part of this. Perhaps, as they say in show business, any publicity is good publicity.

This all said, I don't think I will be leaving Oregon Health & Science University any time soon. I have waited my whole career for the situation I and the department I lead are currently in, seeing the maturation of our field and the resources now available to support education and research endeavors within it.

I also do not envy the person who actually replaces Dr. Blumenthal. Clearly the HITECH Act has, to use the terminology of Gartner Hype Cycle, hit its peak of expectations. Some of the programs are down in the trough of disillusionment, although I am confident that most if not all of them will eventually level off in the plateau of productivity. Given the current political state of Washington, DC, where scoring political points seems to have overtaken governing and producing value from government programs, the next National Coordinator is likely to spend a good deal of time in non-productive Congressional hearings. It's not that I don't think government bureaucrats, like everyone else, need to be held accountable, but it is unlikely the primary purpose of those hearings will be to report on the value to healthcare and economy that HITECH has wrought.

Being the optimist that I am, I am not dwelling too much on the current poisoned atmosphere in Washington, DC. I will certainly defend to anyone the productive investment that has been made the federal government in HIT. In our programs funded by educational grants, skills and leadership have been imparted on a new cadre of individuals, and the curricular materials we are producing will have a lasting impact on the primary goal of biomedical and health informatics, which is to improve human health, healthcare, biomedical research, and public health with information.

Natural Language Processing: A Dream That Won't Die … and Shouldn't

One of the longest-standing dreams of informatics, dating back to the early (i.e, 1960s) era of artificial intelligence, is the use of natural language processing (NLP) to extract data about patients from clinical narrative data (e.g., progress notes, discharge summaries, etc.) in the electronic health record (EHR). The notion that you can take the narrative language of clinicians and turn it into concrete facts that can be used for clinical decision support, clinical research, quality measurement, surveillance, etc. is immensely appealing.

Alas, that dream, at least in a generalizable way, is still a dream. You can count a number of my published papers over the years as a few among the many valiant efforts. Unfortunately, the variability (or some might say mangling, especially by physicians) of language, along with the hidden context and meaning "between the lines," makes NLP a very difficult task to program in a computer.

Some, however, have managed to succeed in focused ways. For example, generalizable decision support also never succeeded but it has been found that focused decision support works quite well and is used in EHRs daily. Likewise, there have a number of focused areas where NLP has provided useful data for clinical processes.

It is in this context that I am pleased to report on another contribution to the literature of clinical NLP, which is a paper that appeared in a recent issue of the Journal of the American Medical Informatics Association (JAMIA) and is lead-authored by a former student, Mary Stanfill [1]. I am a co-author. It is always a thrill to see a student publish a peer-reviewed paper, especially one that started as a term paper in one of my courses, advanced to a capstone project in a master's degree, and ultimately ended up in one of the leading journals in our field.

This paper also makes a valuable contribution of being a systematic review of all studies that report results of "automated coding and classification." The analysis shows that there have been many efforts performed using many methods in a variety of clinical domains, with a wide range of results. Of course, this gets to a gripe I have had with clinical NLP and related text mining researchers over the years, which is that evaluation studies have not advanced much beyond measuring the accuracy (e.g., recall, precision, sensitivity, specificity, etc.) of how well systems identify concepts in the text [2]. I would prefer to see the next step in systems being evaluated, such as how well NLP can impact the tasks it might be used for, such as quality measurement programs or facilitating clinical research studies. This would be akin to the "task-oriented" studies of information retrieval systems I performed years ago, which focused on how well searchers completed tasks using retrieval systems rather than just measuring how many relevant articles they retrieved [3].

The good news is that systems using NLP are starting to be deployed in operational clinical settings or clinical and translational research programs, and there is an ever-increasing amount of real data in electronic form for them to use. In addition to a growing number of individual studies, there are also large-scale projects of which NLP is a significant part. There include:

• Informatics for Integrating Biology and the Bedside (i2b2) - a long-standing project to facilitate the use of clinical data for genomic and clinical research. One of its activities includes a yearly challenge evaluation that allows research to compare systems and results on a common task. The i2b2 challenge has looked at automatic de-identification [4], identification of smoking status [5], recognition of obesity and co-morbidities [6], and extraction of medication information [7].
• Electronic Medical Records and Genomics (eMERGE) Network - a multi-center project focused on the use of data in EHRs to facilitate the study of how genetic variability contributes to health and disease [8]. One of the foci includes the use of NLP for extracting data from clinical narratives and integrating it with other data in the clinical record. One accomplishment of this research to date has been the ability to replicate four of seven known gene-disease associations [9].
• SHARP 4 - one of the four collaborative research centers being funded under the HITECH Program to facilitate meaningful use of EHRs, with a focus on secondary use of EHR data.

Another development is the launching this year of a medical records track in the Text Retrieval Conference (TREC) annual information retrieval challenge evaluation. The track will use de-identified records developed by Wendy Chapman and colleagues.

It is also impossible to discuss this topic without acknowledging the discussion around the IBM Watson question-answering system, which recently proved its mettle in a television game show Jeopardy match [10]. IBM has announced some research partnerships that will apply Watson to medical data. This is an interesting research area, but we will need to see real research results to back up the hype [11].

While there are still challenges for clinical NLP, I believe we are seeing a convergence of new methods coupled with growing needs to make use of the increasing volume of clinical data as well as our desire to facilitate re-use of that data for many purposes, such as clinical decision support, quality measurement and improvement, clinical research, and public health reporting and surveillance. While there may be generalizable approaches yet to be discovered, I suspect that evolution will be much like clinical decision support, which has been more successful when engineered to specific domain areas. But as we have also seen with clinical decision support, the ability to perform those specific tasks successfully will be highly valuable to healthcare.

References

1. Stanfill, M., Williams, M., et al. (2010). A systematic literature review of automated clinical coding and classification systems. Journal of the American Medical Informatics Association, 17: 646-651.
2. Hersh, W. (2005). Evaluation of biomedical text mining systems: lessons learned from information retrieval. Briefings in Bioinformatics, 6: 344-356.
3. Hersh, W., Crabtree, M., et al. (2002). Factors associated with success for searching MEDLINE and applying evidence to answer clinical questions. Journal of the American Medical Informatics Association, 9: 283-293.
4. Uzuner, O., Luo, Y., et al. (2007). Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association, 14: 550-563.
5. Uzuner, O., Goldstein, I., et al. (2008). Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association, 15: 14-24.
6. Uzuner, O. (2009). Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association, 16: 561-570.
7. Uzuner, O., Solti, I., et al. (2010). Extracting medication information from clinical text. Journal of the American Medical Informatics Association, 17: 514-518.
8. McCarty, C., Chisholm, R., et al. (2010). The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Genomics, 4(1): 13. http://www.biomedcentral.com/1755-8794/4/13.
9. Denny, J., Ritchie, M., et al. (2010). PheWAS: Demonstrating the feasibility of a phenome-wide scan to discover gene-disease associations. Bioinformatics, 26: 1205-1210.
10. Ferrucci, D., Brown, E., et al. (2010). Building Watson: an overview of the DeepQA Project. AI Magazine, 31(3): 59-79. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303.
11. Anonymous (2011). IBM to Collaborate with Nuance to Apply IBM’s "Watson" Analytics Technology to Healthcare. Armonk, NY, IBM. http://www-03.ibm.com/press/us/en/pressrelease/33726.wss.

PCAST Report: What's the Big Deal?

As anyone who works in informatics knows, there is unremitting stream of reports, white papers, blog entries, and other writings from various government agencies, non-profit organizations, consultants, research organizations, and others involved in health information technology (HIT). Some of these reports promote various points of view, including policy directions, while others present interesting ideas to read. (Some do neither!)

One recent report has garnered more attention than any in the last several months (perhaps since the release of the meaningful use rules). This is of course the recent report from the President's Council of Advisors on Science and Technology (PCAST) entitled, Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward, which was released in December, 2010. This report, called the “PCAST report” by many, has gained high visibility, perhaps reflecting its origin from the White House. It has led the Office of National Coordinator for HIT (ONC) to ask its HIT Policy Committee to create a workgroup tasked with collecting and analyzing public comments and making recommendations relative to current and future ONC activities.

The report states its case by noting that current HIT systems do not meet their potential, mainly due to the lack of interoperability that results from proprietary data stores (mostly in proprietary systems) that blunt the free flow of data. This is hardly new. The report goes on to advocate what it views as the critical solution to the problem, which is the development of a “universal exchange language” (abbreviated by some though not in the report as “UEL”) based on the notion of data elements being reduced to their atomic core. It advocates that each of these core elements have metadata (“data about data”) tagging that includes the element and its value along with an identity of the patient, a patient-controlled privacy designation, and other provenance information about the element.

The report has certainly generated a great deal of discussion, with most of the major organizations involved in HIT having weighed in during the ONC comment period. The report certainly piqued my interest, since I have always held the view that data is the most critical aspect of everything we do with HIT. I agree with Dr. Blumenthal that data is the “lifeblood” of medicine, and my informatics nirvana consists of data freely flowing between systems and the appropriately authorized people to use it. In my dream world, one could be in one EHR and “Save as…” the data for loading into another EHR or some other application. The data would be so interoperable that any application would recognize documents (e.g., discharge summaries or progress notes), measurements (e.g., vital signs or lab values, and structure data (e.g., prescriptions).

So where does PCAST fit into all of this? In this posting, I will review the PCAST Report, summarize the commentary and criticisms, and give my own further analysis to get others thinking (as a good educator should!).

The report begins with the usual accolades for how HIT has the potential to transform healthcare. In addition to the usual improving clinician access to patient data and decision support, involving patients in their care, and enabling public health and clinical research, the report also notes that HIT can create new markets and jobs for technology as well as support healthcare reforms, including economic changes in the system. It lays out nine use cases that benefit patients, clinicians, public health, and clinical research.

However, the report notes, HIT has fallen short of its promise for four reasons. First, most current systems are proprietary applications not easily adapted into clinical workflow, with proprietary data formats not easily exchangeable. Data is not easily disaggregated or re-aggregated. Second, most healthcare organizations focus on the internal value of EHRs and have no incentive for secondary or external uses of their data for patients, other healthcare organizations, public health, or research. A third reason is that patients have concerns about the privacy and security of information of their data. Finally, the report notes that HIT has been largely focused toward administrative functions and not on improving healthcare.

The UEL will require a common infrastructure for locating and assembling data elements of a patient’s records via secure data element access services (DEAS). Data would remain local, and DEAS would be distributed, intercommunicating, and interoperable. A single appropriately authorized query would be able to locate and assemble a patient’s record across multiple DEAS.

The essential core of the report is Chapters 4-5 (pp. 39-52), which deal with the core of the technology and privacy. Chapter 4 asserts that the approach of applications as “services” does not scale up. (Though the authors seem to violate the notion of separating the application and the data.) It is argued that a better approach for healthcare data is the UEL, which is coded in (eXtensible Mark-up Language) XML and tagged with three metadata elements (in addition to the data and its value):

• Identifying information about patient – including information enabling location of the data (not necessarily a universal identifier)
• Privacy protection information – who may access the data, for what purposes, and either in an identified or de-identified state
• Provenance of data – date, time, equipment used to collect data, personnel who collected it, etc.

The UEL would aim for semantic interoperability. While adherence to specific controlled terminology sets would not be required, it would be strongly encouraged. This would allow over time for data to truly be represented in a universal way.

DEAS activities would include “crawling, indexing, security, identity, authentication, authorization, and privacy.” Queries would be issued against all DEAS on the Internet, and results could be re-constituted into a complete picture of patient. Governments, healthcare organizations, and others would operate the DEAS. In some way, this process would act like Web search engines, although they would need to have very high recall and precision to insure all the appropriate data was retrieved, while incorrect data was not. In conclusion, the chapter claims the UEL is extensible, extractable by middleware, and will lead to innovative uses and applications.

Chapter 5 lays out the privacy and security aspects of the UEL and DEAS. In essence, each and every data element would have a patient-controlled privacy attribute, allowing access to the element and its use in identified or de-identified scenarios. All data would be encrypted, which would not only protect security, but also insert a mechanism to audit access.

A number of leading HIT organizations took the opportunity of the ONC comment period to state their positions. While all applauded the raising of awareness of issues related to data and its interoperability, they also raised a number of criticisms. Many advocated that the PCAST Report serve as a broader vision rather than holding concrete solutions.

The comments about the report can be summarized as follows:

1. The constellation of current standards, as imperfect as they are, meet many of the data-related goals laid out by the report.
2. Many of the ideas of the report, while interesting and worthy of further research, are untested. Having them be the drivers of Stage 2 of meaningful use would be a substantial change in direction and put the larger HITECH Program at risk.
3. Clinical data has context, and reducing its entirety to atomic elements has the potential to lose that context. Re-constituting it may not be possible if that context is lost.
4. Much of the context in clinical data requires that records be more document-centric or at least structured in groups of elements. Disaggregating documents could lose the context they provide.
5. While everyone agrees that structured data is most desirable, some data in healthcare is too nuanced, and unstructured text is required to describe it.
6. While industry-wide standards are important, no industry with data as complex as healthcare has tried an approach like this.
7. The notion of setting privacy at the individual element is highly problematic. Allowing the patient to choose which elements can be seen or not seen by clinicians, researchers, or others could introduce many unintended consequences. It would be preferable for the patient to specify general privacy policies that are then referenced by the data elements.
8. Patients’ views of privacy may change over time, as diseases change and their own disease course changes.
9. Search engines are imperfect. The DEAS would need to operate at high levels of recall and precision that are unprecedented for Internet-based search mechanisms.
10. While the report correctly notes that current HIE efforts are struggling, it ignores that major reasons for this, which have more to do with the lack of a business model for HIE than anything about the technology.

A number of specific comments from these organizations are poignant:

• American Hospital Association (AHA) - The report should set a broader vision rather than focus on concrete solutions. Setting privacy at element level could fracture the patient record. Tagging each piece of data could be costly and inefficient. DEAS are likely to face same challenges as HIEs, with lack of a business model. ONC should change policy direction for Stage 2 of meaningful use only with great caution.
• Radiological Society of North America (RSNA)/American College of Radiology (ACR) - Echoed many of the same comments and noted we need a uniform method to manage patient identity.
• Integrating the Healthcare Enterprise (IHE) USA - Noted that the current IHE profiles cover most of the functionality required for the 9 use cases.
• Healthcare Information and Management Systems Society (HIMSS) -Privacy is contextual and changing, especially as diseases change and information about them becomes less sensitive. Encryption of the data elements provides security and an audit trail but can adversely impact workflow. The objectives of the report might not be possible without a universal patient identifier. By atomizing data, we run risk of data becoming dissociated and not being able to detect errors, so any grouping in the source data should be maintained. Metadata tagging should be virtual and not physical. Tags should be referenced and not attached, since some (e.g., privacy) might change over time. Data elements separated from documents and records potentially robs them of their context.
• HIMSS Electronic Health Record Association (EHRA) -It is better to tag data on document or record level. The privacy approach is potentially unworkable. A large-scale effort of this approach is untested.
• AMIA - Chapter 4 provides general ideas but no details nor references. There is no evidence that this approach will lead to improved care. The report was for the most part silent about other federal agencies, especially the National Library of Medicine, which has great expertise in some aspects of the proposed approach, especially related to terminology development and usage. The report underestimates the complexity of modeling the domain of medicine. It ignores past failed efforts along similar lines, such as the caBIG caDSR project. The DEAS may not be scalable or practical. ONC should not deviate from already tight timeline of Stage 2 of meaningful use. We can learn lessons from the slow adoption of HL7 Version 3, which is not suited to efficient description of task information models. There is too much focus on healthcare and not enough on health.

Other organizations that weighed included:

• HL7
• Federation of American Hospitals (FAH)
• Center for Democracy in Technology
• Project HealthDesign
• Markle Foundation
• Clinical Groupware Collaborative
• Faster Cures
• Society for Participatory Medicine
  

And of course, a number of bloggers had things to say. As always, John Halamka provided great early summaries of the report and the deliberations of the ONC HIT Policy Committee:
http://geekdoctor.blogspot.com/2010/12/spirit-of-pcast.html
http://geekdoctor.blogspot.com/2011/01/primer-on-xml-rdf-json-and-metadata.html
http://geekdoctor.blogspot.com/2011/01/example-for-pcast.html
http://geekdoctor.blogspot.com/2011/01/general-principles-of-universal.html

In a widely cited posting, Wes Rishel noted some critical points: Information flow for patient care occurs at the level of documents. Taking elements out of larger context can lose context. PCAST data elements are actually molecules, not atoms. There are plenty of molecule definitions, these should be used. Another well-known blogger, Keith Boone, added that a good deal of what the report hopes to accomplish can be done with existing standards.

What will be the impact of the PCAST report? We will find out for sure in April when the ONC releases its analysis and plans for incorporating the report’s ideas and proposals. If nothing else, the report has led to increased discussion about the importance of data interoperability, which even its critics applaud. My hope is that there is at least an acceleration toward the vision of interoperable data that most in informatics share.

Some Supplemental Information from the PCAST Report

The nine use cases were lumped into three categories based upon to whom they provided value.

Value to patients:

• Patient on warfarin wanting to know if it is safe to take an NSAID drug for an injury.
• Woman with lung mass newly discovered in a community hospital referred to a large academic medical center.

Value to clinicians:

• Internist developing primary care medical home.
• Small practice with clinicians sharing records and communicating with patients via email.
• Cardiology clinics in a part of country collaborating to improve care for patients with recent myocardial infarction.
• Family physician embedding alerts in practice to improve preventive care.

Value to research and public health:

• Physician caring for a patient enrolled in a national clinical trial.
• FDA carrying out post-marketing surveillance of adverse reactions to drugs.
• Communities or states measuring improvement toward health goals.

The final published conclusions of the report were:

1. HHS and ONC have laid a foundation for progress under meaningful use and HITECH.
2. Achievement of goals for healthcare involves accelerated progress toward robust health information exchange.
3. Effort should now focus on development of a universal exchange language that enables data to be shared and re-assembled across institutions, subject to strong privacy safeguards based on patient privacy preferences.
4. Creating these requirements is technically feasible.
5. ONC should move rapidly to develop these capabilities for stages 2 and 3 of meaningful use.
6. CMS should modernize and restructure its IT platforms to serve as a driver for progress in health IT.

Follow-up from HIMSS

This year's Healthcare Information Management and Systems Society (HIMSS) Annual Conference in Orlando, Florida was interesting and enjoyable as always. Clearly the focus was on healthcare organizations and vendors achieving meaningful use of electronic health records (EHRs) . However, there was also plenty of other buzz on topics such as health information exchange, the impact of the departure of Dr. Blumenthal, and changes going on in Washington, DC.

There was also, of course, much talk about health IT workforce issues and the ONC Workforce Development Program. I was video-interviewed on aspects of the topic by:

• Joseph Conn of Modern Healthcare, who produced a six and a half minute video on workforce issues and educational programs
• Mary Stevens of CMIO Magazine, who produced a three and a half minute video on education and career opportunities for physicians, especially those aspire to be CMIOs.

Accolades for the Informatics Professor and His Blog

I have had the opportunity to publish and speak in some high-profile venues lately, and my blog and my book have received some honors. For the sake of the reader, however, I will roll all of my "bragging" into just this single post.

Let me start with publishing, where I have been invited to make some commentaries on various aspects of informatics. One of these was a contribution to a series of commentaries on the HITECH program in the prestigious journal Nature. (Subscription required for HTML and PDF versions.)

Another was an invitation to write an editorial accompanying a study on the development of registries in chronic kidney disease in the journal, Clinical Journal of the American Society for Nephrology (subscription required).

I have also be invited to talk on panels at a couple of high-profile meetings. One talk was on a panel at the AcademyHealth National Health Policy Conference (February 7-8, Washington, DC). The panel was focused on workforce issues, with speakers addressing different aspects of the healthcare workforce issue. Naturally I was invited to speak on informatics issues, describing the ONC Health IT Workforce Program as well as informatics competencies for modern healthcare professionals (slides).

Another talk is on a panel at this week's HIMSS Annual Conference on the ONC Health IT Workforce Program. I will speak about the Health IT Curriculum Development Centers Program (slides). OHSU serves not only as one of the five Curriculum Development Centers but also the National Training & Dissemination Center tasked with disseminating the curricular materials via a Web site and training community college faculty in their use.

I am also pleased to report that the Informatics Professor blog has been recognized by three sites that rate blogs:

• Top Professors of Math & Science Blogs
• Forty-Six Health Care Blogs for Professionals
• 50 Resources for Students Attending Online Health Information Schools

Finally, I am honored to have my book reviewed by one of my favorite blogs, Keith Boone's Healthcare Standards blog. It turns out that Keith did some work in search earlier in his career and has some poignant things to say about my book (and my blog). I look forward to his book on Clinical Document Architecture due out this spring.

Welcome to Oregon, President Obama and Mr. Otellini

Today I will have the opportunity to attend President Obama's visit to Intel Corp. in Portland, Oregon. While I will just be among the crowd, I would love the opportunity to talk with both of them, and will post my comments of what I would say in this blog entry.

President Obama, I want to thank you for the opportunity afforded by the HITECH Act in your stimulus legislation. I wish we could have you over to visit Oregon Health & Science University (OHSU), where we have put the $5.8 million awarded to us under the HITECH Act to good use. It is an investment in human capital that is building the workforce of future professionals who will lead the "meaningful use" of electronic health records (EHRs) to improve the quality and safety of healthcare. On time and on budget, we are not only one of nine universities training these professionals from around the country, but also one of five universities developing curricular materials for shorter-term community college programs. In the latter program, we also serve as the National Training & Development Center, training and supporting community college faculty in use of the materials and disseminating them via a secure Web site. We have also enhanced the scope of the program by building a curriculum around the VistA EHR system from the Veteran's Administration. This system is another exemplary federal project that has drastically improved healthcare delivery by the VA, and now will provide additional value in training future health information technology professionals in using and configuring EHRs.

To Intel Corp., namely Mr. Paul Otellini, CEO, I would love the opportunity to show off the work being done by a local university, OHSU, in your company's own backyard here in Oregon. I would relish the opportunity to work with your company more to increase economic development and create high-skill, high-paying jobs in Oregon. I have written in this blog, a local tech blog, and even our local newspaper, the Oregonian, about the potential that exists in Oregon for synergy among companies, an innovative healthcare system, and a world class academic program in biomedical informatics at OHSU to create an economic cluster around health information technology. Investing in students, research, and companies could pay off well for our region.

Postscript: I had an enjoyable time visiting Intel and hearing President Obama speak. I managed to get a front row seat and was able to shake the President's hand afterwards. Of course I took pictures and posted some of them on Facebook. I still would love the opportunity to address either the President or Mr. Otellini about the issues raised above.

HITECH: Improving Healthcare Through Data and Action

Every now and then, I am asked to give an overview of the Health Information Technology for Economic and Clinical Health (HITECH) Act of the American Recovery and Reinvestment Act (ARRA, also known as the “economic stimulus bill”). The centerpiece of HITECH is a plan to vastly expand the adoption and “meaningful use” of electronic health records (EHRs) [1], based on a growing body of research demonstrating that EHRs, especially when combined with clinical decision support (CDS), can improve the quality, safety, and coordination of healthcare [2, 3]. Similar to other areas related to technology and/or healthcare, the US has become a laggard in the adoption of EHRs, falling behind most other developed countries [4].

HITECH provides up to $27 billion for eligible professionals and hospitals to receive incentives for achieving the meaningful use of EHRs [5]. Meaningful use is a critical concept. The goal of HITECH is not just to put computers into physician offices and on hospital wards, but rather to use them toward five goals for the US healthcare system: improve quality, safety and efficiency; engage patients in their care; increase coordination of care; improve the health status of the population; and ensure privacy and security. As such, every criterion in meaningful use (e.g., drug-drug interaction checking) must tie back to a healthcare goal (e.g., improve quality, safety and efficiency).

Government funds for HITECH incentives will be distributed through the public Medicare and Medicaid reimbursement systems. Depending on choice of funding through Medicare or Medicaid, eligible professionals can receive $44,000-$63,000, while eligible hospitals can receive $2-9 million between 2011 and 2018. The main purpose of these incentive funds is to cover the costs of investment in EHR systems. It is anticipated that further costs will become part of the "costs of doing business" for healthcare.

The HITECH legislation recognizes that incentives alone will not be enough to achieve all the goals of meaningful use. As such, HITECH allocates an additional $2 billion for various human and organizational infrastructure elements to attain its mandates. A critical portion of this infrastructure is the ability to achieve health information exchange (HIE), which is the secure flow of data to wherever it is needed for patient care, including across traditional business and other boundaries in the healthcare system [6]. About $547 million is allocated to states for HIE development.

Another critical piece of the infrastructure is the provision of technical support to achieve meaningful use. This is done with the allocation of about $677 million to 62 regional extension centers that are providing a variety of forms of assistance, mainly to small primary care practices [7].

An additional portion of the required infrastructure is a competent professional workforce to develop, implement, and train users of EHR and related systems. It has been estimated that the HITECH agenda will require an additional 50,000 professionals trained in fields such as biomedical informatics and health information management [8]. About $118 million has been allocated for both short-term training programs in community colleges as well as longer programs mostly at the graduate level in universities. My institution, Oregon Health & Science Univeristy, is playing a major role in this program.

The HITECH legislation also recognizes that additional research and development is required. As such, $60 million has been allocated to establish four collaborative research centers focusing on the topics of security and health information technology, patient-centered cognitive support, health care application and network design, and secondary use of EHR information. A related funding initiative is the Beacon Communities Program, which has funded about $250 million for 17 advanced demonstration projects “shine the light” forward.

Just as meaningful use connotes that EHR adoption is not just about installing computer technology in clinical settings, there are related initiatives in the United States that will synergize with the substantial HITECH investment. One initiative from the Institute of Medicine aims to develop the “learning health care system” that learns from the growing volume of captured data what does and does not work in healthcare [9]. This is closely related to the growing push for “comparative effectiveness research” that aims to compare tests, treatments, and other medical activities in head-to-head studies carried out in real-world settings [10]. This infrastructure will also likely contribute to the growing push for translational research, as exemplified by funding for the Clinical & Translational Science Award (CTSA) program of the National Institutes of Health [11].

Taken collectively, all these programs from HITECH to ACA, the learning healthcare system, and CTSA provide a vision of a new healthcare system that learns from its successes and changes based on its mistakes. This vision uses data as the critical enabler of coordinating, measuring, and researching care. HITECH is indeed a grand experiment, and it is likely be that some elements of this experiment will succeed whereas others fail. But in the end, the healthcare system should benefit this unprecedented investment in information systems, human capital, and goals for improving health.

References

1. Blumenthal D, Launching HITECH. New England Journal of Medicine, 2010. 362: 382-385.
2. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al., Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 2005. 293: 1223-1238.
3. Goldzweig CL, Towfigh A, Maglione M, and Shekelle PG, Costs and benefits of health information technology: new trends from the literature. Health Affairs, 2009. 28: w282-w293.
4. Schoen C, Osborn R, Doty MM, Squires D, Peugh J, and Applebaum S, A survey of primary care physicians in eleven countries, 2009: perspectives on care, costs, and experiences. Health Affairs, 2009. 28: w1171-1183.
5. Blumenthal D and Tavenner M, The “meaningful use” regulation for electronic health records. New England Journal of Medicine, 2010. 363: 501-504.
6. Vest JR and Gamm LD, Health information exchange: persistent challenges and new strategies. Journal of the American Medical Informatics Association, 2010. 17: 288-294.
7. Maxson E, Jain S, Kendall M, Mostashari F, and Blumenthal D, The regional extension center program: helping physicians meaningfully use health information technology. Annals of Internal Medicine, 2010. 153: 666-670.
8. Hersh W, The health information technology workforce: estimations of demands and a framework for requirements. Applied Clinical Informatics, 2010. 1: 197-212.
9. Eden J, Wheatley B, McNeil B, and Sox H, eds. Knowing What Works in Health Care: A Roadmap for the Nation. 2008, National Academies Press: Washington, DC.
10. Murray RK and McElwee NE, Comparative effectiveness research: critically intertwined with health care reform and the future of biomedical innovation. Archives of Internal Medicine, 2010. 170: 596-599.
11. Zerhouni EA, Translational research: moving discovery to practice. Clinical Pharmacology and Therapeutics, 2007. 81: 126-128.

Electronic Health Records Do Not Impact the Quality of Healthcare?

The headlines have blared the news this week that a new study published in Archives of Internal Medicine showed that electronic health records (EHRs) in the ambulatory setting do not appear to lead to higher quality patient care [1]. This in turn has led many leading news organizations to have stories with headlines such as, Stanford researchers find EHRs don't boost care quality.

For those of us who work in informatics, this is a pretty serious finding. As responsible scientists and citizens, we cannot ignore negative results about the work we do. However, we also have an obligation to place this work in the larger context of all research on the relationship between health information technology (HIT) and quality of medical care.

Like almost all science that gets reported in the general media, there is more to this study than what is described in the headlines and news reports. The study was published in a prestigious medical journal by two Stanford researchers. The implementation of the research methods they used appears to be sound. There is no reason to believe that the results obtained do not derive from the methods used.

However, there are serious limitations to this type of study and to the data resources used to answer the researchers' question, which was whether ambulatory EHRs that include clinical decision support (CDS) lead to improved quality of medical care delivered. While I do believe this study has a place in the evidence base of HIT, it suffers from limitations that are inherent in studies like this that that are observational, correlational, and retrospective. This study used a data source collected for other purposes, the National Ambulatory Medical Care Survey, and compared physicians who were identified users of CDS with those who were not to see if there were differences in the quality of care they provided based on 20 process quality measures. The results found there were no differences between the groups, i.e., those using EHRs and CDS did not deliver higher quality care than those not using them.

Before delving into the details, it is also worth noting that this study is not the first to apply this methodology. Within the last couple months, two other studies have used a similar approach to assess associations between quality measures and hospital EHR adoption [2] and computerized provider order entry [3], giving mixed results, i.e., some measures showing benefit and others not. In addition, Archives of Internal Medicine published another study using this sort of approach in 2009 showing that hospital notes, test results, order entry, and decision support were variably associated with improved patient outcomes and reduced costs [4]. (It was surprising to see the latter not referenced in the article.) If we were to take all of these studies as definitive, then we might conclude that EHRs usage in hospitals improves quality of care, even if EHR usage does not in ambulatory settings.

But whether the results are favorable or not, it is important to understand some serious limitations in these types of studies and this one in particular. A first limitation is that the study looks at correlation, which does not mean causality. This was an observational and not an experimental study. The data used for the study was not collected for the purposes of assessing the quality of care by EHRs. As with any correlational study, there may be confounders that cause the correlation or lack of it. As we know from evidence-based medicine, the best study design to assess causality is an experimental randomized controlled trial. Indeed, such studies have been done and many have found that EHRs do lead to improvements in quality of care. There have been several systematic reviews of such studies noting that while some of the studies suffer from methodologic limitations, others are well-designed and do demonstrate positive value for various aspects of EHR and CDS [5-7]. There is a continuing stream of such studies and two have been published in the last couple months. One showed that an EHR with targeted CDS led to increased improvements in glucose control in diabetics [8], while another found that a real-time alert cut inappropriate use of D-dimer testing by 70% [9]. Not all such studies demonstrate positive results, but enough to do to show that there is value in the well-informed use of HIT.

A second limitation of this study is the quality measures used. Quality measures are of two general types, process and outcome. Process measures look at what was done, such as ordering a certain test or prescribing a specific treatment. Outcome measures look at the actual clinical outcomes of the patient, e.g., whether there was a reduction of mortality, complications, or cost. It is a fair criticism of the current state of the healthcare quality movement that most measures used (including those in the meaningful use criteria) are process measures that may or may not result in improved patient outcomes.

A third limitation of this study is that we do not know whether the physicians using EHRs and CDS had decision support in place to impact the specific quality measures that the researchers studied. While the quality measures are important process measures for physicians to adhere to, they may not be amenable to CDS generally or the CDS used in these systems.

A fourth limitation is that the study assesses episodes of care and not longitudinal care over time. This may not portray an accurate picture of a physician's practice.

A fifth limitation is that the data analyzed was collected in 2005-2007. While EHRs and CDS were available at that time, they were less widely used and less mature at that time.

Finally, we have no idea how well trained these physicians were at using the CDS that they had. We know that success with HIT is based on many factors that go well beyond the technology itself, such as proper implementation and training. Well-designed research must address these factors too.

There are also some other limitations to the study that are discussed in an accompanying editorial that unfortunately few people, especially those who get news of the study from news reports as opposed to the journal itself, will read. The editorial writers appropriately point out that other studies, including experimental ones, have shown value for HIT interventions.

The results of this study are a legitimate addition to the evidence base of informatics and cannot be dismissed out of hand. However, these findings must take their place in the proper context of all research on HIT. If nothing else, this study highlights the need for more and better research to truly identify where HIT helps, has no impact, or outright harms patients.

References
1. Romano MJ and Stafford RS, Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Archives of Internal Medicine, 2011: Epub ahead of print.
2. Jones SS, Adams JL, Schneider EC, Ringel JS, and McGlynn EA, Electronic health record adoption and quality improvement in US hospitals. American Journal of Managed Care, 2010. 16: SP64-SP72.
3. Kazley AS and Diana ML, Hospital computerized provider order entry adoption and quality: an examination of the United States. Health Care Management Review, 2011. 36: 86-94.
4. Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, and Powe NR, Clinical information technologies and inpatient outcomes: a multiple hospital study. Archives of Internal Medicine, 2009. 169: 108-114.
5. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al., Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 2005. 293: 1223-1238.
6. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al., Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 2006. 144: 742-752.
7. Goldzweig CL, Towfigh A, Maglione M, and Shekelle PG, Costs and benefits of health information technology: new trends from the literature. Health Affairs, 2009. 28: w282-w293.
8. O'Connor PJ, Sperl-Hillen JM, Rush WA, Johnson PE, Amundson GH, Asche SE, et al., Impact of electronic health record clinical decision support on diabetes care: a randomized trial. Annals of Family Medicine, 2011. 9: 12-21.
9. Palen TE, Price DW, Snyder AJ, and Shetterly SM, Computerized alert reduced D-dimer testing in the elderly. American Journal of Managed Care, 2010. 16: e267-e275.

Reflections at the End of Another Amazing Year for Informatics

Last year, in wrapping up the first year of the Informatics Professor blog, I marveled at how amazing the year of 2009 had been. I noted that the year started with both uncertainty and hope; the former fueled by the recession and the precarious financial state of Oregon Health & Science University (OHSU) due to that recession and the latter driven by the excitement of the election of President Barack Obama and (at least for me) the hope for real change. By the end of 2009, it was clear that profound change had indeed occurred, if not generally then at least in the biomedical and health informatics field.

The hope and change, of course, were driven by the HITECH program with the president's economic stimulus package. At the end of 2009, the path forward was clear: health information technology would be driven by the concept of "meaningful use," and the part nearest and dearest to my heart, education and training, would be driven by the ONC Workforce Development Program, which itself was driven by Section 3016 of the HITECH Act that I played a role in influencing.

I spent the latter days of 2009 and early part of 2010 writing proposals, in particular for the curriculum development program and the university-based training program. With the Funding Opportunity Announcements (FOAs) for these and other programs, such as Beacon, SHARP, and regional extension centers, released in December and due in January, many in the informatics field lamented that ONC stood for the "Office of No Christmas." I spent a good part of my winter break last year working on these proposals. The only enjoyable aspect of the process was that they allowed us to envision how we could implement the educational programs we always dreamed of if we ever had the money, which now it looked like we did.

The most harrowing part of the year was the time between the submission of the proposals and receiving word about funding. As well-positioned as we were to receive these competitively awarded proposals, there was an undercurrent of fear that perhaps we forgot to address some required aspect of the program or that some reviewer felt we had taken the wrong approach. In all honesty, it would have been quite an embarrassment to not be selected for funding, since OHSU's program laid the groundwork for some of the thinking that had emerged surrounding health IT workforce development.

All the agony came to an end on Friday, April 2nd, when I awoke in the morning to find out that both OHSU proposals had been funded. For the curriculum development project, we were not only funded as one of the five curriculum development centers, but also chosen as the lead National Training and Dissemination Center (NTDC). For the university-based training program, we were one of nine programs selected for funding tuition assistance in our graduate program. A common quip in academia is that the downside to getting grants funded is that you then have to do the work. However, this was literally a dream come true. Between both grants, we were funded for $5.8 million to do what we always envisioned we could do if we had the funding. While the short-term emphasis of the funding (due to their being stimulus funds) required us to make some decisions we might otherwise not make, it was still a great position in which to be.

Also on the second to last day of 2009, the preliminary meaningful use rules were released. These were followed by a 60-day comment period, modification of the rules, and the release of the final rules on July 13th. I happened to be in a hotel room in Singapore (10 pm local time, 10 am Eastern time) when listening to their unveiling. While everyone had qualms with this criteria or that criteria, I believe that the majority of people were content with the approach to meaningful use taken by ONC.

With our own projects, we hit the ground running. Out of the gate, the curriculum development project required the most work up front. After a two and a half day workshop in Washington, DC the second week of the grant, we began our long quest that would result in the first version of the curriculum being developed and handed off to the community colleges by the end of October. Being the NTDC, OHSU also had to organize a training event for community college faculty in August and launch a Web site for dissemination of the materials around that time, both of which we did. We even added an aspect to the project of creating an educational version of the VA VistA electronic health record system.

The university-based training grant project was a little slower to get started, but not by much. With funding for 135 Graduate Certificate and 13 master's degree students over three years, our plan was to use the funding mainly as a form of tuition assistance for new students entering the field. We started providing support for students in the summer academic quarter and really ramped up in the fall. The main regret is that we have received two to three times as many qualified applicants as we having funding to accept. A decent proportion of those individuals have enrolled as self-funded students.

While a good proportion of my year was spent around these ONC initiatives, there were other achievements as well. Due to ONC and other funding, the Department of Medical Informatics & Clinical Epidemiology catapulted to second among the 25 departments at OHSU in external funding. We have many other initiatives in comparative effectiveness research, bioinformatics, and related areas. The big challenge for the department in 2011 and beyond is how to consolidate and build upon the success of the stimulus-era funding. I am confident we will find ways to do this, as the need for our disciplines to advance healthcare, personal health, and biomedical research will not diminish even as the federal budget tightens.

The coming year will also be an interesting one for the informatics world. How many eligible professionals and eligible hospitals will achieve meaningful use? What unforeseen bumps in the road will emerge? How will healthcare reform impact the use of health information technology? What will happen to healthcare reform itself? One thing is certain: we will live through exciting times!

I have now been writing this blog for almost two years. I have been pleased to have this type of forum to share my views on various aspects of my work. I am also pleased that others have noticed, not only the 129 people who follow the blog, but also winning awards like being on the list for the 2010 Top Math & Science Professor Blogs Award.

I plan to keep running the blog pretty much like I have been, with a fewer number of more substantive posts than the stream of consciousness approach used by many other blogs. I do hope to branch out a little bit more this coming year beyond workforce and education, as I occasionally did this year.

The Comfort of Connectivity

My family, friends, and colleagues believe I spend way too much time keeping up with my email and related work activities, even when I am on vacation, as I am now. They are probably right, as I type this while on vacation in lovely Oaxaca, Mexico.

Maybe it is because I remember the days when email and Internet connectivity from afar were hit or miss. Now, however, I have to admit that I marvel at the ease of accessing Wi-Fi and even my Verizon smartphone (phone, texting, and Internet on my Droid) from this lovely city that is not exactly at the forefront of technology. I am staying in a mid-range apartment, which has Wi-Fi, as does the Instituto Cultural de Oaxaca, where I am studying Spanish for a couple weeks. (The Droid is a wonderful helper for learning Spanish, as I am using two apps for Spanish-English dictionaries.) Since everything on my Droid works here, I am even able to set up a 3G mobile hot spot in a pinch!

Many eating, coffee, and other establishments have Wi-Fi as well, even the Parque El Llano a couple blocks from our apartment. My Droid has even worked in most of the small villages outside of Oaxaca. Perhaps even more amazing was that some homes in these poor villages actually have broadband Internet.

I truly am trying to take a vacation and only responding to critical emails. I have to admit there is a certain comfort to know that my connectivity is there, even if I am trying to minimize its use. I am reading my emails if for no other reason to not have thousands awaiting my return from this two and a half week vacation.

Of course, my physical and virtual lives are so merged that it would be very difficult to not use my computer and access the Internet, even when on vacation. I certainly enjoy taking pictures with my digital camera and sharing them. I also do a good deal of my news reading these days on-line. And of course there are my many friends and others on Facebook with whom I enjoy interacting. In addition, figuring out the details of visiting tourist sites, restaurants, and other places is greatly facilitated when one has Internet access. So it would be truly difficult if not impossible to completely unplug.

There is literally no place on the planet where the Internet is not accessible these days. In the past few years, I have connected from places such as Zimbabwe and Cuba. While ubiquitous global connectivity has some drawbacks, I firmly believe it is positive overall, and the ease of communication and sharing can foster better relations among peoples of the world.

10x10 ("Ten by Ten") at the End of 2010 and Beyond

With the end of 2010 approaching, I am asked with increasing frequency whether we met the goals set out by the 10x10 ("ten by ten") program, which was launched in 2005 with the goal of training 10,000 healthcare professionals in informatics by the year 2010. Now that 2010 is coming to an end, how did we do?

I can say that the program has been an unqualified success. The OHSU 10x10 offerings trained nearly 1000 (999, to be precise) people, with another eight universities training an additional 258 more, for a total of 1257 from 2005-2010. Many of those of completing the program have enhanced their current careers. From the OHSU courses, about 15% pursued additional training in the field. While our numbers did not add up to 10,000, there was clearly value for those who completed the course. The program also helped expand educational capacity in the field generally and highlighted the need that led to legislation such as Section 3016 of the American Recovery and Reinvestment Act (ARRA) and the resulting ONC Workforce Development Program.

The 10x10 courses are offered on-line, with an in-person session at the end that brings participants together face to face. The amount of material in each course is roughly comparable to an introductory three-credit graduate-level course, as shown in the syllabus from the OHSU course. In a demonstration that the Internet knows no boundaries, the course has attracted participants from all corners of the globe, such as Argentina, Hong Kong, Singapore, Israel, Pakistan, South Korea, Saudi Arabia, China, India, and Nigeria. The enthusiasm from Latin America led a group from Hospital Italiano of Buenos Aires to translate the course into Spanish and offer it across Latin America. About 500 individuals have completed this version of the course from a number of Spanish-speaking countries. Another version of the OHSU course has been offered in Singapore four times, with the in-person session held in Singapore.

We absolutely plan to continue the 10x10 program beyond the end of 2010. Two more OHSU offerings started in late 2010, along with a few more from other universities. There are no plans whatsoever to end the program, whose need continues to be demonstrated as increasing numbers of healthcare professionals and hospitals seek to achieve "meaningful use" of electronic health records. Of course, biomedical informatics is about more than meaningful use and EHRs, as demonstrated in the course syllabus.

AMIA has already changed the tag line of the program from "10,000 Trained by 2010" to "Training Next-Generation Informatics Leaders." Maybe we should just say that 10x10 now the program that aims to train 10,000 individuals in biomedical and health informatics without giving a specific deadline. Clearly the need remains.

The end of 2010 is also a time to reflect on how we arrived here. In 2005, Dr. Charles Safran, who was then President of the American Medical Informatics Association (AMIA), began taking an interest in the informatics capacity of healthcare organizations. In a letter to the editor of JAMA, he stated that each hospital in the US should have at least one physician and one nurse trained in informatics. Meanwhile, AMIA was looking to beef up its e-learning offering, but found new development of content would be prohibitively expensive. At the same time, I had already been offering the introductory course in the OHSU Biomedical Informatics Graduate Program on-line for some time. It was apparent that we could repackage the course relatively easily. Building on Charlie's call, I coined the name 10x10, aiming to train 10,000 people within five years, by 2010.

I have thoroughly enjoyed developing and teaching the 10x10 course. It has been personally gratifying to meet so many people who took the course and found it of value. I am delighted that some colleagues from Argentina translated the course to Spanish, as noted above. The course name even made its way into legislation in a bill that passed the US House of Representatives (though not the US Senate), the 10,000 Trained by 2010 Act introduced by Congressman David Wu (D-OR). A demo version is available for those who want to take a look.

Some have asked why the Chair of a department would enjoy teaching the introductory course so much. I take great satisfaction in providing people their first introduction to the field of biomedical and health informatics. I enjoy the give and take with students, including those who challenge me. The 10x10 course and my other educational accomplishments make it clear that these activities are my passion and calling in life.