Sunday, August 26, 2012

Data Entry: An Under-Discussed Grand Challenge for Informatics

Everyone, including this blog writer, has been touting the virtues of the vast troves of data already or soon to be available in the electronic health record (EHR), which will usher in the learning healthcare system [1, 2]. There is sometimes unbridled enthusiasm that the data captured in clinical systems, perhaps combined with research data such as gene sequencing, will effortlessly provide us knowledge of what works in healthcare and how new treatments can be developed [3, 4]. The data is unstructured? No problem, just apply natural language processing [5].

I honestly share in this enthusiasm, but I also realize that it needs to be tempered, or at least given a dose of reality. In particular, we must remember that our great data analytics and algorithms will only get us so far. If we have poor underlying data, the analyses may end up misleading us. We must be careful for problems of data incompleteness and incorrectness.

There are all sorts of reasons for inadequate data in EHR systems. Probably the main one is that those who enter data, i.e., physicians and other clinicians, are usually doing so for reasons other than data analysis. I have often said that clinical documentation can be what stands between a busy clinician and going home for dinner, i.e., he or she has to finish charting before ending the work day.

I also know of many clinicians whose enthusiasm for entering correct and complete data is tempered by their view of the entry of it as a data blackhole. That is, they enter data in but never derive out its benefits. I like to think that most clinicians would relish the opportunity to look at aggregate views of their patients in their practices and/or be able to identify patients who are outliers in one measure or another. Yet a common complaint I hear from clinicians is that data capture priorities are more driven by the hospital or clinic trying to maximize their reimbursement than to aid clinicians in providing better patient care.

Another challenge for clinicians is the time required for electronic data entry. There is no question that the 20th century means of clinical documentation, mostly consisting of scribbling illegible notes on paper, was much easier and faster than typing and/or clicking. While I think that few clinicians want to go back to hand-written notes, there is an appeal of their ease of use, at least for the person doing the entry.

Related to the time for electronic data entry is the "tension" between structured data, which makes aggregation and analysis easier, and "flexible" (or narrative) data, which allows the clinician to tell the story of the patient [6]. Many clinicians report that excess structuring of data (i.e., pointing and clicking) loses the story of the patient, although those who process the data know that structured data is easier to analyze.

An additional challenge for electronic data entry for clinicians is the shift of the focus from the patient to the computer. This was exemplified in a cartoon published earlier this year in JAMA that showed a 7-year-old's sketch of an exam room with the physician hunched over the computer, his back turned away from the patient and her family [7] (the sketch viewable at

An excellent example of the promise but limitations of current data entry systems was recently documented by Parsons et al. [8], who found in a wide sample of primary care EHRs in New York City that the accuracy of data for measuring breast cancer screening quality measures was highly variable due to differing practices in documentation, workflow, and related factors. While some physicians had the quality of their care measured accurately, for many others it was underestimated due to data limitations and not the care they provided.

I cannot claim to have easy answers to this grand challenge, but two related aspects of it sit in front of us:

  1. We need to find better and faster ways for clinicians to enter data into the EHR that allow data whose quality is good enough to be re-used for other purposes, such as research, quality measurement and improvement, and public health.
  2. We must reward clinicians for their efforts in entering high-quality data. We must allow them to see aggregate views of patients in their practices and be able to identify outliers. We must also engage them in research, quality improvement, and other system uses of their data.

In short, the concept of "garbage in, garbage out" still remains a problem for computers and information technology nearly a half-century after it was coined. In healthcare, we must give clinicians the best tools and incentives for them to participate in the learning healthcare system. For informatics, the problem of data entry is a grand challenge every bit as important as how to make use of its growing quantity, since the knowledge derived from that data will only be as good as the quality of what is input.


1. Friedman, C., Wong, A., et al. (2010). Achieving a nationwide learning health system. Science Translational Medicine, 2(57): 57cm29.
2. Greene, S., Reid, R., et al. (2012). Implementing the learning health system: from concept to action. Annals of Internal Medicine, 157: 207-210.
3. McCarty, C., Chisholm, R., et al. (2010). The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Genomics, 4(1): 13.
4. Rea, S., Pathak, J., et al. (2012). Building a robust, scalable and standards-driven infrastructure for secondary use of EHR data: The SHARPn project. Journal of Biomedical Informatics, 45: 763-771.
5. Nadkarni, P., Ohno-Machado, L., et al. (2011). Natural language processing: an introduction. Journal of the American Medical Informatics Association, 18: 544-551.
6. Rosenbloom, S., Denny, J., et al. (2011). Data from clinical notes: a perspective on the tension between structure and flexible documentation. Journal of the American Medical Informatics Association, 18: 181-186.
7. Toll, E. (2012). The cost of technology. Journal of the American Medical Association, 307: 2497-2498.
8. Parsons, A., McCullough, C., et al. (2012). Validity of electronic health record-derived quality measurement for performance monitoring. Journal of the American Medical Informatics Association, 19: 604-609.

Sunday, August 19, 2012

The Internationalization of 10x10

One of the most gratifying aspects of my work in informatics has been its international acceptance. I also enjoy my interactions with international colleagues, both professionally and personally. In addition, I obtain great satisfaction interacting internationally with students, whether they study under myself or others. All told, I enjoy making contributions that are known and valued around the entire world.

As I have written before, I have come to learn that many of the problems faced by informatics are global in nature, i.e., not unique to the United States. All who work in healthcare, public health, and research face challenges in collecting, organizing, and making best use of data and information. All types of information systems present challenges to workflow, usability, and value, among other things.

In recent years, one of my major collaborations has been with colleagues in South America, particularly in Argentina. One of these activities has been for them to translate my well-known 10x10 ("ten by ten") course into Spanish. They have offered the course all across Latin America, with nearly 1000 having completed it. The Spanish 10x10 course uses the same basic approach as my original course, consisting of a series of online units with an in-person session where students meet, learn more together, and/or present projects. While the original version of the Spanish course was a close translation of the English one, the content has now evolved to take on a more Latin American perspective. The core informatics issues are still global, but there are some regional differences, e.g., no delving into the details of HIPAA!

I am pleased to report that the collaboration with Argentina and its translation of the 10x10 course into Spanish has now come full circle, in that the Spanish course has made its way back to the United States via Puerto Rico. As Puerto Rico is fully eligible for funding through the HITECH programs, it has made efforts to help its eligible professionals and hospitals achieve meaningful use of electronic health records. One aspect of this has been the establishment of a regional extension center (REC), the Ponce School of Medicine REC (PSMREC). One of the activities of the PSMREC has been to bring informatics education to clinicians, and it has done this by engaging my Spanish-speaking colleagues and their version of 10x10 to create a new instance of the course, Certificado en Informática Médica para Puerto Rico. The course organizers invited me to participate in their in-person session in San Juan to kick off the course. I spent an enjoyable day with faculty and students, giving talks on the HITECH program as well as the secondary use of clinical data. I was also impressed with the backgrounds of the students taking the course, representing leaders from the REC, academic institutions, healthcare organizations, and others. It is always gratifying as an educator to help those you teach make a difference, which I am certain will happen in this course and what follows in Puerto Rico.

Another active collaboration involving the 10x10 course is in Singapore, with my colleague Dr. KC Lun of Gateway Consulting and the National University of Singapore. This offering of 10x10 uses my English version of the course, with my traveling to Singapore for the end-of-course in-person session. A total of 126 people have completed six offerings of the course over the last several years, with a seventh slated to start this month. I have enjoyed this offering of the course, both through the various people I have met in the Singapore healthcare community as well as the chance to learn about the Singapore healthcare system, which delivers high-quality care at about one-half the per-capita cost of the US.

I also remain active internationally in a number of other ways. I have served for six years as Chair of the Health and Medical Informatics Education Working Group of the International Medical Informatics Association (IMIA). I am a member of the Editorial Board of the International Journal of Medical Informatics. I serve on a number of institutional advisory committees in Europe, Asia, and Africa.

I cannot deny that being able to travel to many of these places has been one of the best perks of my job. Although the real beauty of this travel is that I do not just visit these places as a detached tourist. Rather, I enjoy being welcomed into the work and even home environments of my colleagues. I value seeing firsthand the informatics-related work that they do. I look forward to continuing my work with all of them.

Saturday, August 4, 2012

Accolades for the Informatics Professor: Update

As readers of this blog know, I periodically point to other places on the Web where the Informatics Professor has been mentioned or honored. One recent accolade is an article in the magazine, For the Record, describing the recent update to the ONC Health IT Curriculum. I also wrote another article myself for HIMSS Clinical Informatics Insights, a perspective piece about the entire ONC Health IT Workforce Program.

Another accolade is being asked to serve on the (get ready for a mouthful!) Workforce Subgroup of the Certification and Adoption Workgroup of the ONC Health IT Policy Committee. The Health IT Policy Committee sets policy for ONC initiatives, including implementation of meaningful use. The Certification and Adoption Workgroup is tasked with making recommendations "on issues related to the adoption of certified electronic health records that support meaningful use, including issues related to certification, health information extension centers and workforce training." The Workforce Subgroup on which I will be serving is tasked with making "recommendations to the Health IT Policy Committee on ways to provide health IT education to all health care workers."

An additional accolade I have received is being elected as a Fellow of the American College of Medical Informatimusicology (ACMImimi). This honor is bestowed upon those who "see the perfect harmony found in combining healthcare, information technology and music" and manifest it by performing music in a public setting, in my case a recent rendition of some Neil Young songs at the National Library of Medicine Trainees meeting in Madison, Wisconsin on June 26, 2012.

ACMImimi is run by Ross Martin, MD, MHA, an accomplished informatician in many ways, but perhaps most distinctive by his making informatics concepts, especially the HITECH program, accessible via music. I have steered students in recent years to some of his most entertaining and educational pieces:
HITECH: An Interoperetta in Three Acts
The Meaningful Yoose Rap

Thursday, August 2, 2012

MOOCs Coming to Informatics?

A few months ago, I wrote a post about the new development of massive open online courses (MOOCs) and what disruptive innovation they might have on higher education. While noting that academic biomedical and health informatics sat at the intersection of the two industries having the least amount of disruptive innovation - healthcare and higher education - I did note that MOOCs could have an impact, if they ever came to our field.

Well now they have, and it turns out that one of my projects is playing a major role in their development. The Health Informatics Forum, an international blogging and social network site for informatics, has started to turn the entire ONC Health IT Curriculum into a MOOC. They recently posted Unit 1 of Component 1 on their site, with announcement of plans to add a new component every four weeks.

How successful will this effort be? There will certainly be value in providing learning materials to the entire world. But there are some caveats. First, as those of us in the ONC Health IT Curriculum project have noted, the materials are designed more for educators than learners. While they provide a rich amount of learning substrate, like all good education they require more, including a teacher, a structured learning process, and ideally fellow learners. In addition, any professional educational experience also requires a connection to the real world through practical opportunities, such as internships. Furthermore, in any rapidly changing field, such as health IT and informatics, the curricular materials must be regularly updated and otherwise improved. It will be interesting to see how sites like The Health Informatics Forum address this latter challenge, particularly as the field evolves. (For example, the Stage 2 meaningful use rules as well as new HIPAA regulations are due out in the next few weeks. As of now, neither of these are covered in the ONC curriculum.)

One final caveat is that the total quantity of these materials represent about 20 college-level courses. This means that any one person will require a great deal of time and effort to work through all of them. By the same token, there are a number of advanced informatics topics that are not covered by the ONC curriculum, such as secondary use of data, natural language processing, and analytics, to name a few. Still, I will be eager to see how this all works out, and hope to lend my expertise to increase its likelihood of success.

It turns out that The Health Informatics Forum is not the only organization that has utilized the totality of the ONC materials as a large learning experience. Two other organizations have done this as well, one of which charges a fee, which is allowable under the Creative Commons license under which the materials have been released:
The discussion around MOOCs also continues to flourish in the press. The New York Times has run a series of articles, mostly focused on the two efforts led by Stanford (Coursera) and Harvard/MIT (edX) but now expanding to include other universities in their partnerships. This has included articles about the expansion of Coursera as well as the the early experiences of Coursera and edX and one call for caution:
It is still too early to tell how these efforts will fare, and what their impact will be on higher education. As one who has been teaching online for 13 years, I can say that learning is very possible, and often desirable, especially when the learner is separated from the learning experience by distance or time. We have many students in our distance learning program at Oregon Health & Science University (OHSU) who desire our education but live far from Oregon and/or work during the time that on-campus classes are offered. In fact, we have a number of "local distance" who live in the area but value the convenience of the online classes.

But our courses at OHSU are anything but MOOCs. They feature direct interaction from our faculty. Furthermore, students can participate in and get credit from structured practicum and internship experiences. This leads one to wonder whether MOOCs might become a means to deliver higher education rather than complete experience in and of themselves. Nonetheless, it will be interesting to see the outcome of this natural experiment in education.