Further Evidence That Health IT Job Growth Has Been Underestimated, and Some Ramifications

New data from an analysis of online job postings confirms that employment growth in health information technology (HIT) has even further exceeded projections, driven by funding from the Health Information Technology for Economic and Clinical Health (HITECH) Act. In addition, two other reports show that the job market for those working with electronic health record (EHR) and related systems continues to be strong for employees and challenging for employers.

In the new analysis, Schwartz and colleagues used a comprehensive database of 84 million online job postings,  extracted out those related to HIT, and built a model aiming to determine the influence of HITECH [1]. The authors limited their focus to jobs that would be defined in the realm of clinical informatics. Although this was important for their goal of assessing the influence of HITECH, from the perspective of a biomedical informatics graduate program director like myself, this excluded other important informatics jobs, such as those in imaging informatics, bioinformatics, clinical research informatics, and other areas where a graduates of our program are employed.

The analysis classified jobs into two broad categories, HIT core jobs and HIT clinical user jobs. The former category included those developing, implementing, supporting, and selling EHRs, while the latter included clinicians, receptionists, technicians, and other personnel making heavy use of EHRs in their jobs. Wearing my informatics educational program director's hat, I was most interested in the authors' results for the HIT core job listings, as these individuals would be most likely to be employed in (and to seek education preparing them for) informatics careers.

Schwartz et al. counted a total of HIT-related 434,282 job postings between 2007-2011, with 226,356 HIT core jobs and 207,926 HIT-related clinical user jobs. Yes, not 41,000 [2], not even 51,000 [3], but over 226,000!

For both categories of HIT jobs combined, the authors categorized employer type and provided a percentage of all HIT jobs for each. The largest employer category was IT vendor, which included IT service providers, consulting firms, sales firms, and IT staffing firms hiring developers and posted 42% of all HIT-related listings. Another 39% were posted by healthcare provider organizations. The remaining 19% were either ambiguous as to the employer type (15%) or another type of employer (4%).

For all HIT jobs, they also categorized and tallied job responsibilities, which could be assigned to more than one category for a given posting. The most frequent job responsibility was implementation support, with 43% of jobs including responsibilities such as system installation, customization, building, debugging, purchasing, or workflow redesign. The next highest category was user training (27%), followed by system development (22%). This was followed by technical support, with 21% of jobs including the maintaining of continued technical functionality or providing customer support. Other responsibilities included IT strategy (long-term IT planning and system optimization in the clinical setting - 13%), sales (11%), and research (quantitative hypothesis testing using health IT systems - 6%).

Their model estimated that about 48% of the job growth was due to HITECH, with the remainder due to growth that would have continued at historical trends prior to HITECH. Some other interesting findings for the time period between 2007-2011 included HIT jobs growing from 0.75% to nearly 2.5% of all healthcare job postings (consistent with prior findings of healthcare organizations hiring one IT employee per 48-60 non-IT employees [2]) and an approximately four-fold increase in the number of jobs posted.

It will remain to be seen how strong the job market remains as the HITECH incentives wind down, although HIT will continue to be a cost of doing business in healthcare, especially as the system moves to payment models that require better management and use of data, such as accountable care organizations and primary care/patient-centered medical homes [4].

Two other reports show that healthcare organizations are responding to the need for hiring and maintaining more HIT talent. A report by Towers Watson that surveyed 100 healthcare organizations [5] found that 67% of reported problems in attracting experienced IT employees, with an even higher proportion (73%) finding problems hiring Epic-certified employees. These organizations also reported problems in retaining experienced IT employees (38%) and Epic-certified employees (52%) as well. Many fewer organizations reported problems in attracting (14%) or retaining (9%) new graduates with IT skills, indicating a strong incentive for those early in their careers to get up to speed quickly.

The Towers Watson report also discerned differences between employees and employers in the drivers of attracting and retaining IT personnel. For attracting personnel, employees listed job security and salary as their primary concerns, whereas employers believed that challenging work and organizational reputation were key drivers. There was more concordance on drivers of IT employee retention, with both employees and employers ranking salary and opportunities for career advancement highest.

The second report was the first of what will be an annual survey of the HIT workforce by HIMSS Analytics [6]. The survey was completed by 224 individuals who were HIMSS corporate members as well as hospital and health system IT executives. About three-quarters of respondents were employed by healthcare provider organizations, including standalone hospitals (45%), hospitals as part of an integrated delivery system (17%), and corporate offices of a delivery system (13%). The remaining respondents worked vendor organizations, including companies (20%) and consulting firms (4%).

Over 80% of healthcare provider organizations reported adding IT FTE in the past year, with half hiring 1-5 FTE and the rest hiring more. Only 8% reporting laying off staff. The most common areas for hiring were clinical application support (51% of all healthcare provider organizations hiring), help desk (51%), IT management (29%), financial application support (28%), system design and implementation (24%), IT security (22%), project management (21%), clinical informatics/clinical champion (19%), system integration (19%), user training (15%). About three-quarters of provider organizations outsourced rather than hired some of the above types of personnel. Essentially similar percentages were seen in hiring plans for the coming year.

A similar picture was seen for vendors and consultants for the past year, with over 90% adding FTE, and nearly 60% hiring more than 20 FTE. Most common areas for hiring included sale and marketing (88%), field support staff (84%), support staff (68%), and executive team (58%). Nearly a third reported laying off some staff. Similar to provider organizations, comparable percentages were noted in hiring plans for the coming year.

Various types of certification were deemed important, more so vendors than healthcare provider organizations. The most highly rated certifications were security professional, network/architecture support, database administrator, project manager, and informatics professional.

Strategies for retaining qualified staff were also important for providers and vendors, including offering professional development activities (60% for providers, 64% for vendors), paid tuition (48%, 41%), membership payment in professional associations (35%, 41%), and telecommuting (29%, 52%).

About 80% of providers and 57% of vendors reported lack of fully qualified staff as a barrier to achieving organizational IT goals. The most common reason for lack of staff was lack of qualified staff in their local region (43%, 56%). Both types of organizations reported hires being attracted to other organizations by more lucrative offers (25%, 19%). About 31% of provider organizations reported putting an IT initiative on hold due to inadequate staffing, with another 19% contemplating doing so.

Overall, these new data show that the HIT job market is robust, and provides great opportunities for a career. Of course, like essentially all professional areas these days, the job market will change. This will be driven in HIT by the tapering of the incentive funding from HITECH and the shift from implementing to optimizing (i.e,, making use of the data) systems. This will require that the skill sets of professionals, and the curricular content of educational programs, adapt to (or ideally anticipate) the likely changes.

References

1. Schwartz, A, Magoulas, R, et al. (2013). Tracking labor demand with online job postings: the case of health IT workers and the HITECH Act. Industrial Relations: A Journal of Economy and Society. 52: 941–968.
2. Hersh, W (2010). The health information technology workforce: estimations of demands and a framework for requirements. Applied Clinical Informatics. 1: 197-212.
3. Conn, J (2010). 50,000 new health IT workers might be needed. Modern Healthcare, May 25, 2010. http://www.modernhealthcare.com/apps/pbcs.dll/article?AID=/20100525/NEWS/100529949/.
4. Anonymous (2011). Support for Accountable Care: Recommended Health IT Infrastructure. Washington, DC, eHealth Initiative. http://www.ehidc.org/resource-center/reports/view_document/103-reports-support-for-accountable-care-recommended-health-it-infrastructure-accountable-care.
5. Anonymous (2013). Closing the IT Talent Gap in Health Care - The Towers Watson 2013 Health Care IT Survey Report, Towers Watson. http://www.towerswatson.com/en/Insights/IC-Types/Survey-Research-Results/2013/03/Closing-the-IT-Talent-Gap-in-Health-Care.
6. Anonymous (2013). 2013 HIMSS Workforce Survey. Chicago, IL, HIMSS Analytics. http://www.himssanalytics.com/research/AssetDetail.aspx?pubid=82097&tid=128.

Accolades for the Informatics Professor - Summer 2013 Edition

Periodically I like to share some of the accolades I receive in my work, and this posting is another installment.

First, I was featured in a piece in Health Data Management concerning my accomplishments generally and related to the HITECH program specifically.

Next, this blog was named among the best health information technology (HIT) blogs from an HIT recruiter site.

Finally, I had the opportunity to present in an interesting format at the Ignite Health meeting in Portland, OR on September 19, 2013. The format for each presenter was to have five minutes to talk and each slide advancing automatically after 15 seconds, for a total of 20 slides. It was fun by challenging, and I also had to go after a speaker who had a wonderfully choreographed presentation with beautiful graphical slides. It was quite challenging for an academic used to a longer time to talk and less graphical slides, but I made the best of it. I chose to focus my presentation and its slides on the new Informatics Discovery Lab in our department. There is also an index to the overall program.

Postscript: The day after posting, I was mentioned in this nice article on the clinical informatics board exam.

ONC Funding for the OHSU Informatics Program: Coming to an End

It almost seems like yesterday when I woke up on the morning of April 2, 2010 to find out that OHSU had been awarded $5.8 million for two health information technology (HIT) workforce development grants by the office of the National Coordinator for HIT (ONC), one to serve as the lead among five universities in developing the national HIT curriculum for community college programs and the other for funding students in the University-Based Training (UBT) program aiming to accelerate the growth of the HIT workforce.

Now it is hard to believe that next week, at the end of this month, the second of our two grants will end, and our program will no longer have any direct funding from ONC. (The curriculum development grant ended in March. And technically while we have through the end of December to continue graduating students, the funding portion of the grant is ending on September 30, 2013.)

Overall, I am proud of the work we accomplished with this funding and the contributions we made to the field. While the curricular materials are sitting in a static mode on our Web site, they are still freely available and provide a foundation for anyone wishing to develop an educational program in HIT. In addition, the materials have found many uses beyond the community college programs, including in our own graduate program at OHSU. Before long, however, the materials will require updating before making use of them, but in their current state, they still do prevent someone from having to start from scratch. I hold faint hope that some mechanism to update and extend them will emerge.

I am likewise proud of what we accomplished in the UBT program. We not only launched many HIT careers, but also added some functionality to our own informatics educational program that we plan to maintain, namely a practicum/internship program and a career development service. Some of our best stories are published here, here, and here.

It is somewhat fitting that this time period coincides with a new book released in the Springer Health Informatics series about informatics education [1]. Edited by colleague Dr. Eta Berner, the book features a number of chapters on a variety of topics, including two written by myself, one on the ONC workforce program [2] and the other on the 10x10 program [3]. We also recently published a journal paper evaluating the ONC HIT curriculum materials with its primary users, who were community college faculty [4].

References

1. Berner, E, Ed. (2014). Informatics Education in Healthcare: Lessons Learned. London, England, Springer.
2. Hersh, WR (2014). Informatics for the Health Information Technology Workforce. Informatics Education in Healthcare: Lessons Learned. E. Berner. London, England, Springer: 93-107.
3. Hersh, WR (2014). Online Continuing Education in Informatics: The AMIA 10 × 10 Experience. Informatics Education in Healthcare: Lessons Learned. E. Berner. London, England, Springer: 109-120.
4. Mohan, V, Abbott, P, et al. (2013). Design and evaluation of the ONC health information technology curriculum. Journal of the American Medical Informatics Association: Epub ahead of print.

A 20th Century Model of Education for a 21st Century Profession? OHSU Response to ACGME Draft Requirements for Clinical Informatics Fellowship Programs

As noted last month, the Accreditation Council for Graduate Medical Education (ACGME) released draft requirements for clinical informatics fellowship programs in late July, with a 45-day comment period that ended last week. A group from Oregon Health & Science University (OHSU), including myself, some other informatics faculty, our Senior Associate Dean for Education, and our Associate Dean for Graduate Medical Education submitted a response to ACGME.

The bottom line, as the title of this post says, is that the proposed ACGME approach really applies an increasingly outdated 20th century model of medical training to the vibrant 21st century subspecialty of clinical informatics. There is no question that clinical informatics training, like any other training, requires knowledge, skills, and experience. But the standard time-based, in situ approach to training likely will not build the capacity needed or provide a pathway that many who seek to join this profession can take.

One irony of this sort of approach is that OHSU was just awarded one of 11 grants from the American Medical Association (AMA) in their Accelerating Change in Medical Education initiative. Three schools, including OHSU, have as one of their aims for the grant to move medical education from a time-based to competency-based approach. There is no reason why every medical student has to spend exactly four years in school. Some may have backgrounds that allow them to accelerate their pace, which will be helpful as the need for physicians grows due to aging baby boomers and healthcare reform.

In a posting last year, I expressed concern about a time-based, in situ approach, which not only may limit the growth of capacity in the subspecialty, but also lock out a pathway to the profession for those who cannot disrupt work, families, or other aspects of their lives to uproot their lives to pursue a site-based clinical informatics subspecialty. OHSU has trained many physicians and others who have gone on to successful informatics careers using a mostly distance-based approach.

There are other problems that our response noted as well. A key one is the limitation of programs being administratively linked to the six specialties of Anesthesiology, Emergency Medicine, Medical Genetics, Pathology, Pediatrics, or Preventive Medicine. While this does not mean that physicians of any specialty will not be allowed to participate in a fellowship, we expressed concerns programs may be beholden to the affiliated specialty, either philosophically or fiscally, who may impose demands that could compromise the clinical informatics training experience. In addition, it may be difficult for trainees of specialties outside the affiliated one to pursue clinical work in their own specialty within the fellowship in a given institution that has a fellowship linked to a specific specialty.

We also expressed concern that clinical informatics fellows might not be able to practice their specialty as attending-level physicians and bill for their work. Being able to bill for practice in their primary specialty will be important not only for fellows’ maintaining clinical skills in their primary specialty but also for financial viability of the fellowship program.

We will eagerly await the ACGME response to ourselves and others who replied to their draft. In the meantime, planning will move forward for a clinical informatics subspecialty fellowship at OHSU. We also hope to work with other programs who seek help in providing educational content in their programs.

Do Physicians Spend Too Much Time With Computers?

A recent study of work hours of medical interns in the new era of duty hour regulations produced an interesting side finding, which is that modern medical interns spend about 40% of their time at a computer [1]. To some, this prompted concern that computers were drawing medical trainees away from patients and their care.

A finding like this certainly warrants attention. However, I wonder whether many expressing concern are asking the wrong question. The proper question is not whether this is too much time at a computer, but rather if this amount of time compromises the interns' care of their patients or of their learning experience.

Implicit among those who raise the question of too much time with computers is the assumption that computers are taking physicians away from patients. It is instructive, however, to consider historic data of how much time physicians spend in direct vs. indirect care of patients. It turns out that physicians have historically spent most of their working time in activities other than in the presence of their patients.

Time studies of hospital [2-6] and emergency [7] physicians show physicians spend about 15-38% of their time in direct patient care versus 50-67% of their time in indirect patient care, divided among reviewing results, performing documentation, and engaging in communication. Likewise, studies of outpatient physicians find that 14-39% of work takes place outside the exam room [8-9]. In addition, work related to patients when they are not even present at the hospital or office consumes 15-23% of the physician work day [9-11].

Therefore, this new study does not necessarily indicate the computers are drawing physicians away from patients. It is difficult to compare the proportions of these interns' time devoted to direct and indirect care with those of other physicians who have completed their training. However, it is worthy to note that the four-fold ratio of indirect to direct care is not too far off what was document for practicing physicians in the other studies. Interns have different time demands anyways, not only working longer hours but also devoting more time to educational activities.

Furthermore, we also have to consider the premise that there good reason for spending more time in front of computers, as some evidence supports the notion that there may be value to patients as well as clinician education in having access to knowledge in unprecedented ways that previous generations of physicians did not have [12-13]. There is no question that we must pay more attention to physician workflow with computers so that they are not unduly wasting time, especially time that could be better spent with patients. But we must also consider the benefits of computers, and try to determine the most appropriate amount of time for physicians to spend using them during their working hours.

References

1. Block, L, Habicht, R, et al. (2013). In the wake of the 2003 and 2011 duty hours regulations, how do internal medicine interns spend their time? Journal of General Internal Medicine. 28: 1042-1047.
2. Ammenwerth, E and Spötl, HP (2009). The time needed for clinical documentation versus direct patient care. A work-sampling analysis of physicians' activities. Methods of Information in Medicine. 48: 84-91.
3. Tipping, MD, Forth, VE, et al. (2010). Systematic review of time studies evaluating physicians in the hospital setting. Journal of Hospital Medicine. 5: 353-359.
4. Tipping, MD, Forth, VE, et al. (2010). Where did the day go?--a time-motion study of hospitalists. Journal of Hospital Medicine. 5: 323-328.
5. Kim, CS, Lovejoy, W, et al. (2010). Hospitalist time usage and cyclicality: opportunities to improve efficiency. Journal of Hospital Medicine. 5: 329-334.
6. Yousefi, V (2011). How Canadian hospitalists spend their time - a work-sampling study within a hospital medicine program in Ontario. Journal of Clinical Outcomes Management. 18: 159-164.
7. Chisholm, CD, Weaver, CS, et al. (2011). A task analysis of emergency physician activities in academic and community settings. Annals of Emergency Medicine. 18: 117-122.
8. Gilchrist, V, McCord, G, et al. (2005). Physician activities during time out of the examination room. Annals of Family Medicine. 3: 494-499.
9. Gottschalk, A and Flocke, SA (2005). Time spent in face-to-face patient care and work outside the examination room. Annals of Family Medicine. 3: 488-493.
10, Farber, J, Siu, A, et al. (2007). How much time do physicians spend providing care outside of office visits? Annals of Internal Medicine. 147: 693-698.
11. Chen, MA, Hollenberg, JP, et al. (2010). Patient care outside of office visits: a primary care physician time study. Journal of General Internal Medicine. 26: 58-63.
12. Buntin, MB, Burke, MF, et al. (2011). The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Affairs. 30: 464-471.
13. McCoy, AB, Wright, A, et al. (2013). State of the art in clinical informatics: evidence and examples. Yearbook of Medical Informatics. 8: 13-19.

ACGME Releases Draft Clinical Informatics Fellowship Program Requirements For Public Comment

As with all medical subspecialties, the new clinical informatics (CI) subspecialty will need to develop fellowship training programs for those seeking to enter the field. After the first five years of the subspecialty (which starts in 2013), during which the training requirements to be eligible to sit for the certification exam can be met by the "practice pathway" or a “non-traditional fellowship" (i.e., "grandfathering"), starting in 2018 the only way to become certified will be through a fellowship accredited by the Accreditation Council for Graduate Medical Education (ACGME).

On July 29, 2013, the ACGME released a draft program requirements document and opened up a 45-day comment period for public feedback (with comments due September 11, 2013). This posting provides a summary of the 26-page document. In a subsequent post, I will provide the comments that the Oregon Health & Science University (OHSU) biomedical informatics program submits to ACGME.

As with most training requirements documents, there is boilerplate (required of all specialties, in bold text style) and specialty-specific text (plain text style).

All CI programs will need to be administratively integrated with one of six specialties: Anesthesiology, Emergency Medicine, Medical Genetics, Pathology, Pediatrics, or Preventive Medicine. This does not mean that a program needs to be focused in one of these specialties; it only means that it must be administered by one of them. CI programs will not have their own residency review committees (RRCs), but instead will be reviewed by RRCs from these six specialties. Physicians from other specialties can enroll in any of these programs.

Programs will be required to be of 24 months duration, with the fellow having completed the program within 48 months.

There must be a single program director who is board-certified in CI or a subspecialty acceptable to the RRC. There are substantial administrative responsibilities for the director. He or she must also have five years experience working in CI. There must be two additional faculty members, with the three faculty collectively devoting at least two FTE to administration, supervision, and teaching. There must also be a program coordinator to provide administrative support.

In addition to resources for education, the program must have a "clinical information system" that contains health and wellness data, includes clinical decision support, and is accessible in all (inpatient and outpatient) healthcare settings.

The program must of course have an educational program that has clear competency-based goals that are distributed to faculty and fellows at least annually. There must be regularly scheduled didactic sessions. The document is not specific as to the content of the educational program, but of course the content must as a minimum prepare the student to pass the CI subspecialty board exam.

The educational competencies that the program must follow are based on the six ACGME core competencies, with some additional learning objectives specific to CI. The ACGME competencies and some of CI extensions include:

1. Patient care and procedural skills – leverage information and communication technology across the dimensions of healthcare to improve processes and outcomes
2. Medical knowledge – demonstrate knowledge of informatics theory and practice
3. Practice-based learning and improvement – develop skills and habits for self-evaluation and life-long learning
4. Interpersonal and communication skills – communicate effectively, including serving as a liaison between information technology professionals, administrators, and clinicians
5. Professionalism – demonstrate all aspects of professionalism, including the ability to recognize the causes and consequences of security breaches and to show sensitivity to the impact of information system changes
6. Systems-based practice – in addition to understanding the operations of the healthcare system, be able to recognize and disclose the role of oneself and systems in medical error as well as identify and improve the impact of systems on clinical care

Programs must be evaluated at the level of the fellow, the faculty, and the whole program, to be done via:

• Clinical Competency Committee of three program faculty to evaluate fellows semi-annually
• Faculty evaluations to be done at least annually
• Program Evaluation Committee of two faculty and one fellow for ongoing evaluation of program

In addition, the document states standard requirements for duty hours, supervision, moonlighting, mandatory time free of duty, and maximum duty period length. Clinical work may be performed in the fellow’s primary specialty.

(Thanks to Ben Munger of University of Arizona for reviewing this summary and providing feedback. All errors, however, are my responsibility.)

We Can Learn About the Difficulties of Healthcare Reform from the Health Problems of Former Presidents

One of mantras of those who oppose healthcare reform is that it will deny people needed care. Programs that require measurement of healthcare quality or aim to discourage overuse of care are viewed by some as efforts to deny Americans their rightful access to healthcare. Consumers want "choice" to get the care they believe they need, and much of the healthcare system is well-configured with the financial incentive to meet that need.

This week, former President George W. Bush underwent a coronary stent procedure to open up a 70% blockage in one of his coronary arteries. The details of his symptoms are unclear but, as noted by an article in Forbes magazine [1], if President Bush was not having cardiac symptoms (e.g., chest pain), then there is no scientific evidence that the stenting procedure he underwent will prevent a future heart attack or prolong his life compared to just using optimal medical therapy (e.g., treatment of hypertension, hyperlipidemia, etc.) [2].

President Bush is not the only former President to have had possibly suboptimal healthcare for heart disease, which is still the top killer of Americans. President Bill Clinton also had heart problems, although he was acutely symptomatic and required urgent treatment. President Clinton was in suburban New York at the time and went to the nearest emergency department. This hospital had a referral arrangement with Columbia-Presbyterian Medical Center (CPMC) in New York City, where the former President was transferred. It turns out that CPMC, as great an academic medical center as it is, had poor performance on a number of quality measures in the New York State Cardiac Surgery Reporting System, a system whose data has been shown to be associated with beneficial clinical outcomes [3]. Even worse, the surgeon who operated on the former President had a worse-than-average rate of complications.

President Clinton did suffer a complication, and we cannot know for sure whether the complication was a result of the poorer quality care provided by his hospital or surgeon. But as noted in an article in Slate [4], this does raise questions as to the limits of consumer-driven healthcare. If two former Presidents, who presumably have more access to resources and information than anyone else on the planet, cannot make optimal healthcare decisions, can we expect the average consumer to do so? Of course, cardiac disease is one of those conditions for which we have more studies and more quality data than almost any other, and it gets worse from there.

It is unfortunate that a combination of politics and financial self-interest have created a climate of equating any attempt to rein in unnecessary healthcare as "denying" someone care. Focus groups of consumers show there is widespread skepticism, based on misunderstanding, of terms like "quality guidelines" and "evidence-based care" [5,6]. Efforts to have Medicare reimburse physicians for consultation about end-of-life care become "death panels" [7], the Politifact lie of the year for 2009. Efforts to be more appropriately evidence-based about the use of mammography in younger women were viewed as evidence of government malfeasance, when it reality more effective mammography would save the government money if they it led to improved treatment outcomes for breast cancer [8].

I certainly support patient engagement in healthcare decisions. I applaud the "Choosing Wisely" initiative of leading medical societies to highlight care that it is ineffective or outright dangerous [9]. But I remain, like many, frustrated that our political landscape and healthcare financing system impede a forthright discussion of the facts.

References

1. Husten, L (2013). Did George W. Bush Really Need A Stent? Forbes, August 6, 2013. http://www.forbes.com/sites/larryhusten/2013/08/06/questions-about-president-george-w-bushs-stent/
2. Stergiopoulos, K and Brown, DL (2012). Initial coronary stent implantation with medical therapy vs. medical therapy alone for stable coronary artery disease: meta-analysis of randomized controlled trials. Archives of Internal Medicine. 172: 312-319.
3. Jha, AK and Epstein, AM (2006). The predictive accuracy of the New York State coronary artery bypass surgery report-card system. Health Affairs. 25: 844-855.
4. Sanghavi, D (2009). Talk to the Invisible Hand - The promises and perils of treating patients more like consumers. Slate, September 28, 2009. http://www.slate.com/articles/news_and_politics/prescriptions/2009/09/talk_to_the_invisible_hand.html
5. Carman, KL, Maurer, M, et al. (2010). Evidence that consumers are skeptical about evidence-based health care. Health Affairs. 29: 1400-1406.
6. Ross, M, Igus, T, et al. (2009). From our lips to whose ears? Consumer reaction to our current health care dialect. Permanente Journal. 13(1): 8-16.
7. Nyhan, B, Reifler, J, et al. (2013). The hazards of correcting myths about health care reform. Medical Care. 51: 127-132.
8. Quanstrum, KH and Hayward, RA (2010). Lessons from the mammography wars. New England Journal of Medicine. 363: 1076-1079.
9. Cassel, CK and Guest, JA (2012). Choosing wisely: helping physicians and patients make smart decisions about their care. Journal of the American Medical Association. 307: 1801-1802.

Data Scientists Must Also Be Research Methodology Scientists

I had the chance last week to attend a conference in Singapore, Big Data and Analytics in Health Care. It was an interesting blend of academics, operational health information technology professionals, and data scientists from companies in the emerging analytics market. I was also in Singapore for the end in-person session of the 10x10 ("ten by ten") introductory informatics course we offer there.

The talks were all interesting, but I was struck by the difference in the content and tone of the academic and clinical operations speakers compared to those from analytics companies and who called themselves "data scientists." Whereas the academic and clinical operational types were cautious in their methods and results, the data scientists implied their techniques would revolutionize healthcare and threw around terms like "big data" and "analytics" at every turn. One of the latter types showed a "model" of the pathways leading to good (conservative) and bad (surgery) outcomes in back pain, with the intermediate nodes representing actions along the path, such as medication use, physical therapy, and chiropractic care. It was not clear to me how this model could be used to improve care, and I am not sure the speaker really understood that correlations do not prove causality. A second such speaker showed some interesting correlations between words and phrases that occur in clinical narratives of patients with diabetes and aspects of their care. I understand machine learning and how it might be used to "learn" things about patients with diabetes, but I did not see any evidence that this work would lead to any kind of improved patient outcomes.

Another concern I have about proponents of clinical data analytics is their presumption that their algorithms can somehow take all of the growing amount of operational electronic health record (EHR) data and automatically turn it into medical knowledge, as if they could turn a crank with data going in and knowledge emerging. I do have great enthusiasm for some of what can be done with this data, but I also have concerns about the quality and completeness of this data as well as the causality issues that arise without controlling observations in experimental ways.

I had the opportunity to speak at the conference as well, and gave a talk pulling together my cautious enthusiasm for using operational clinical data for research and other analytical purposes. This was the first public talk I have given on this topic since publication of a paper with ten other colleagues on caveats for the use of operational electronic health record data in comparative effectiveness research in the journal Medical Care [1]. The paper was commissioned by AcademyHealth and is part of a special supplement of the journal devoted to electronic data methods.

Our paper notes that while there are many opportunities for using clinical data for research and analytics, we also must remember the limitations of such data. In particular, EHR and other clinical data may be:

• Inaccurate - data entry is not always a top priority for clinicians, and they may take shortcuts, such as copy-and-paste
• Incomplete - patients do not get all of their care in one setting
• Transformed in ways that undermine meaning - coding for billing is the best known example of this
• Unrecoverable for research - data may be in clinical narratives or other less accessible places
• Of unknown provenance - we need to know where data comes from and how likely it is to be accurate
• Of inappropriate granularity - data too coarse for research purposes
• Incompatible with research protocols - patients are not always diagnosed and treated consistently with best practices

Despite these caveats, I am optimistic that there will be uses for this data, especially if we can generate it in a standards-based way and otherwise improve its quality. Hopefully clinicians, researchers, patients, public health authorities, quality improvement leaders, and other who might benefit from the data will have incentive to improve it by more meticulous entry as well as use of standards-based, such as those proscribed by Stage 2 of the meaningful use program [2]. For many clinicians especially these days, the EHR can be a data sink hole into which they enter data, spending a great deal of time but getting little in return.

The bottom line is that while data scientists may be able to generate interesting and important results with their methods, they must also understand basic principles of research science, such as inferential statistics, clinical significance, and cause and effect. In addition, they must demonstrate their methods lead to improvements in health and/or healthcare, and are not just generating interesting associations. In other words, they must show evidence that their methods add value, just as medical care and informatics are required to do.

References

1. Hersh, WR, Weiner, MG, et al. (2013). Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical Care. 51(Suppl 3): S30-S37, http://journals.lww.com/lww-medicalcare/Fulltext/2013/08001/Caveats_for_the_Use_of_Operational_Electronic.7.aspx.
2. Metzger, J and Rhoads, J (2012). Summary of Key Provisions in Final Rule for Stage 2 HITECH Meaningful Use. Falls Church, VA, Computer Sciences Corp. http://assets1.csc.com/health_services/downloads/CSC_Key_Provisions_of_Final_Rule_for_Stage_2.pdf.

OHSU Informatics Graduation 2013: UBT Winding Down, Other Growth Steady, and Future Opportunities Abound

One of my favorite activities each year at Oregon Health & Science University (OHSU) is graduation. It is always a joy to march in the faculty procession and see our graduates who attend receive their degrees and certificates. Since we first started having graduates in 1998, I have only missed one commencement ceremony. Below is a picture of the happy group that attended this year's ceremony.

With this year's graduates, our biomedical informatics graduate program has now awarded 537 degrees and certificates to 509 people. We have 181 people who are still enrolled in our various programs. A total of 1558 people have been enrolled in our program at one time or another dating back to its launch in 1996. The figure below shows the distribution in the program tracks - bioinformatics and computational biology (BCB), clinical informatics (CI), and health information management (HIM) - for each degree or certificate.

This graduation also marks the end of studies for almost all students funded by the University-Based Training (UBT) program, the initiative of the Office of the National Coordinator for Health IT (ONC) to vastly expand the health IT workforce. Funded under the Health Information Technology for Economic and Clinical Health (HITECH) Act, this program began in April, 2010 and in essence served as a scholarship program to pay tuition for students selected for funding.

The figure below shows the flowchart for applicants through graduates of the OHSU UBT program. While we will fall slightly short of our goal to graduate 148 students (135 Graduate Certificate and 13 Master's), we have launched many new informatics careers. While we accepted more people than our goal (176), the attrition rate turned out to be higher than we anticipated (24.4%, although about one-third continued as non-UBT-funded students). In addition, a number of those to which we were not able to award funding still enrolled in the program (65). In the meantime, a substantial number of people who were not candidates for UBT funding or otherwise did not apply still enrolled and/or graduated during the time period of the UBT grant (272 enrolled, 136 graduated).

The UBT funding will formally end on September 30, 2013. After that time, we will have three additional months to allow the remaining UBT-funded students to complete the program. Given how much of professional effort has revolved around two ONC initiatives, the UBT program and the Health IT Curriculum Development Project, it is amazing to think of my life devoid of them.

The good news is that there is not only plenty of continued demand for informatics education, but new opportunities are emerging. Interest and enrollment in our program has not abated. As for the opportunities, one will be training programs for the new clinical informatics subspecialty for physicians. In addition, there will likely be additional opportunities when new certifications emerge from the Advanced Interprofessional Informatics Certification Task Force of the American Medical Informatics Association (AMIA) as well as others from groups such as the Commission on the Accreditation of Health Informatics and Information Management (CAHIIM). There are in addition opportunities for developing informatics education for medical students, enabled by a recent grant awarded to OHSU by the American Medical Association (AMA).

Changes in the Meaning of Birthdays As We Age

Today is my 55th birthday. I am halfway through my fifties. Overall, I cannot complain about my life. I have a wonderful family, great friends and colleagues, and a rewarding career. I also have my health, which I work at to maintain via healthy living, although I acknowledge that some health problems occur that are unrelated to diet and exercise, i.e., over which we have very little control.

Earlier in my life, my birthday represented a next step in my life, such as to achieve a new age or take a new job (one day after the usual July 1st starting date of the new year in academic medicine). Now, however, birthdays are more introspective, giving me pause to think and reflect about where my life has gone and what lies ahead.

In my rise in my career, I have mostly been an overachiever, accomplishing more than many people relative to my age. I reached a significant leadership role in launching the forerunner of my academic department, and then becoming Chair when we became a true department. My colleagues used to kid me at one point when I was the youngest member of the Board of Directors elected by the American Medical Informatics Association. My overachievement in this part of my life made up for my being a relative underachiever earlier in my life, especially in elementary and early high school.

As I have reflected on my birthdays in recent years, one thing is clear to me: While I still have a number of pathways open to me in life, my finite time left closes some options. I certainly have plenty of time to make adjustments in my career, but there are few careers that I could undertake completely de novo at this time.

Not that I have any desire to drastically change my career. I have been fortunate to be a part of something special, from the days of informatics being an obscure academic discipline to the modern days of HITECH, smartphones, and the ubiquitous Internet. My students laugh at me when I sometimes awe at the new technology and reminisce about the limits of what we had to deal with in the old days. However, I will keep looking forward in the informatics field and in life, and wonder about what amazements I will come upon at future birthdays.

Would You Get a $485 MRI Exam?

Last month, I was traveling to my daughter's college commencement in Corvallis, OR, and saw an interesting billboard along the way, advertising a $485 magnetic resonance imaging (MRI) exam. We are so unaccustomed to seeing healthcare businesses try to compete on price that this billboard took me by surprise.

And then I thought to myself, would I use this service if I needed an MRI? Fortunately, I have health insurance through my employer, so am able to get exams like an MRI without having to shop based on price. Of course, I am not sure I would want shop for something that might be serious for my health on the basis of price alone, even though I have to admit that I have no idea (nor does anyone, really) whether the quality of this $485 MRI or the one costing fourfold more that most other local healthcare institutions offer is better or worse. Still, I feel some comfort in knowing I do not have to choose based on cost, especially on cost alone.

This dilemma goes to the heart of our current conundrum of healthcare. We know that healthcare costs too much, especially in the United States. Yet I suspect that few of us, especially when a serious diagnosis is being contemplated, would want any care less than the best possible care. This is especially the case when one needs care acutely; the last thing we do when suffering from severe abdominal or chest pain is to go shopping for the best price.

As I have written before and even before that, while there may be things we can do to reduce the cost of healthcare without sacrificing quality, unleashing the free market, which works so well in some industries like computers and agriculture, is unlikely to be the solution to our healthcare cost problems. While our healthcare system can do a much better job providing information about quality and safety, I just do not see most people choosing bargain-basement prices for healthcare when confronted with serious illness.

The Limits to De-Identification of Clinical Data

There is a wealth of opportunity for putting digital clinical data to use for better understanding health and disease as well as improving healthcare delivery, consistent with the vision from the Institute of Medicine of the "learning health system" [1]. Yet as we have seen from the recent news around the US government monitoring of phone call metadata and Internet data, there are serious concerns about the mis-use of digital data that could be a deal-breaker for health-related data if we do not address privacy and security head on.

Concerns about privacy and security of health data are quite valid. Barely a day goes by before we hear about another data breach in a healthcare organization, with those large enough going on the "wall of shame" of the US Department of Health and Human Services Office of Civil Rights (OCR). These concerns are demonstrated well in the famous but eerie ACLU pizza video. Another recent study shows it is quite easy to discern attributes, including those that may be health-related, about people based on what they share on their Facebook wall [2].

If we want to get the health-related benefits of clinical data, we must address privacy and security issues. We need not only strict regulation of what can and cannot be done with data, but also an ethos around its responsible use. However, if we address those issues appropriately, then perhaps we should be, as pointed out by Zak Kohane recently, demanding "more surveillance" of medical records for health-related purposes.

When it comes to protecting data, we need to be realistic about what does and does not work. One solution commonly proposed is "de-identification" of data, i.e., removal of elements that identify individuals. There is certainly a role for the use of de-identified data in many types of analysis of clinical data. There are, however, limits to the use of de-identified data.

The problems of de-identified are two-fold. First, as famously shown by Latanya Sweeney over a decade ago, data de-identified one database can be combined with data in other sources to re-identify people, including the Governor of Massachusetts [3]. She recently demonstrated this again with a study of people who volunteered their data for the Personal Genome Project [4]. Cimino has shown groups of lab test results (e.g., chemistry panels) can allow re-identification of people [5]. The bottom line is that we are awash in data than can allow re-identification of people, and it will only be exacerbated by the ultimate personal identifier that will soon be available, namely the variants in our own genome.

But perhaps the more important limitation is that data that is truly de-identified, i.e., to the point it cannot be re-identified, may lead to incompleteness in the ability for its use in a comprehensive manner. This is mainly because people get healthcare at different places [5,6]. While we may be able to re-identify data within an organization, it is typically difficult when data goes into multi-organization repositories. Again, this de-identified data may be perfectly fine for some purposes, it does not give us the longitudinal data to which we might want to ask more complex questions.

While recent events may give us a jaundiced view of the uses of data, hopefully that view will moderate as we start to see the benefits. Many of those benefits may emanate from healthcare, making it imperative that we both address privacy and security concerns seriously but also put such data to beneficial use.

References

1. Smith M, Saunders R, Stuckhardt L, and McGinnis JM, Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. 2012, Washington, DC: National Academies Press.
2. Kosinski M, Stillwell D, and Graepel T, Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 2013. 110: 5802-5805.
3. Sweeney L, k-anonymity: a model for protecting privacy. International Journal on Uncertainty, Fuzziness and Knowledge-based Systems, 2002. 10: 557-570.
4. Sweeney L, Abu A, and Winn J, Identifying participants in the personal genome project by name. Social Science Research Network, 2013. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2257732.
5. Cimino JJ, The false security of blind dates: chrononymization’s lack of impact on data privacy of laboratory data. Applied Clinical Informatics, 2012. 3(4): 392-403.

What is a Thinking Informatician to Think of IBM's Watson?

One of the computer applications that has received the most attention in healthcare is Watson, the IBM system that achieved fame by beating humans at the television game show, Jeopardy!. Sometimes it seems there is such hype around Watson that people do not realize what the system actually does. Watson is a type of computer application known as a "question-answering system." It works similarly to a search engine, but instead of retrieving "documents" (e.g., articles, Web pages, images, etc.), it outputs "answers" (or at least short snippets of text that are likely to contain answers to questions posed to it).

As one who has done research in information retrieval (IR, also sometimes called "search") for over two decades, I am interested in how Watson works and how well it performs on the tasks for which it is used. As someone also interested in IR applied to health and biomedicine, I am even more curious about its healthcare applications. Since winning at Jeopardy!, Watson has "graduated medical school"  and "started its medical career". The latter reference touts Watson as an alternative to the "meaningful use" program providing incentives for electronic health record (EHR) adoption, but I see Watson as a very different application, and one potentially benefitting from the growing quantity of clinical data, especially the standards-based data we will hopefully see in Stage 2 of the program. (I also have skepticism for some of these proposed uses of Watson, such as its "crunching" through EHR data to "learn" medicine. Those advocating Watson performing this task need to understand the limits to observational studies in medicine.)

One concern I have had about Watson is that the publicity around it has been mostly news articles and press releases. As an evidence-based informatician, I would like to see more scientific analysis, i.e., what does Watson do to improve healthcare and how successful is it at doing so? I was therefore pleased to come across a journal article evaluating Watson [1]. In this first evaluation in the medical domain, Watson was trained using several resources from internal medicine, such as ACP Medicine, PIER, Merck Manual, and MKSAP. Watson was applied, and further trained with 5000 questions, in Doctor's Dilemma, a competition somewhat like Jeopardy! that is run by American College of Physicians and in which medical trainees participate each year. A sample question from the paper is, Familial adenomatous polyposis is caused by mutations of this gene, with the answer being, APC Gene. (Googling the text of the question gives the correct answer at the top of its ranking to this and the two other sample questions provided in the paper).

Watson was evaluated on an additional 188 unseen questions [1]. The primary outcome measure was recall (number of correct answers) at 10 results shown, and performance varied from 0.49 for the baseline system to 0.77 for the fully adapted and trained system. In other words, looking at the top ten answers for these 188 questions, 77% of those Watson provided were correct.

We can debate whether or not this is good performance for a computer system, or a computer system being touted to provide knowledge to expert users. But a more disappointing aspect of the study is its limitations that I would have brought up had I been asked to peer-review the paper.

The first question I had was, how does Watson's performance compare with other systems, including IR systems such as Google or Pubmed? As noted above, for the three example questions provided in the paper, Google gave the answer in the snippet of text from the top-ranking Web page each time. It would be interesting to know how other online systems would compare with Watson's performance on the questions used in this study.

Another problem with the paper is that none of the 188 questions were provided, not even as an appendix. In all of the evaluation studies I have performed (e.g., [2-4]), I have always provided some or all of the questions used in the study so the reader could better assess the results.

A final concern was that Watson was not evaluated in the context of a real user. While systems usually need to be evaluated from the "system perspective" before being assessed with users, it would have been informative to see whether Watson provided novel information or altered decision-making in real-world clinical scenarios.

Nonetheless, I am encouraged that a study like this was done, and I hope that more comprehensive studies will be undertaken in the near future. I do maintain enthusiasm for systems like Watson and am confident they will find a role in medicine. But we need to be careful about hype and we must employ robust evaluation methods to test our claims as well as determine how they are best used.

References

1. Ferrucci, D, Levas, A, et al. (2012). Watson: Beyond Jeopardy! Artificial Intelligence. Epub ahead of print.
2. Hersh, WR, Pentecost, J, et al. (1996). A task-oriented approach to information retrieval evaluation. Journal of the American Society for Information Science. 47: 50-56.
3. Hersh, W, Turpin, A, et al. (2001). Challenging conventional assumptions of automated information retrieval with real users:  Boolean searching and batch retrieval evaluations. Information Processing and Management. 37: 383-402.
4. Hersh, WR, Crabtree, MK, et al. (2002). Factors associated with success for searching MEDLINE and applying evidence to answer clinical questions. Journal of the American Medical Informatics Association. 9: 283-293.

Can We Have an Informed Discussion About Healthcare Reform?

One of the responsibilities that I enjoy a great deal and take most seriously as an educator is getting people to think about and to look at issues from all perspectives. Yes I do have my own opinions about many things, but I genuinely try to get people to form their own opinions based on an informed analysis of the facts. I am also willing to change my opinions when the facts no longer support them.

In this vein, I find the national debate in the US about healthcare reform very frustrating. While I admittedly support the Affordable Care Act (aka, Obamacare), I also know it is imperfect. If any healthcare policy wonk were designing healthcare reform from scratch, they would likely not come up with Obamacare. Yet I also recognize that healthcare reform is a political process, and that political outcomes, based on compromise and tradeoffs, never completely satisfy anyone. In addition, we cannot forget why we need healthcare reform in the first place, which is because our current healthcare system is wasteful, harmful, and not sustainable. Doing nothing is not an option, and I believe that Obamacare is preferable to maintaining the status quo.

One of the biggest ironies about Obamacare is that while most Americans oppose the overall law, they support most of the provisions in it, particularly the requirements denying lifetime limits on coverage or on preexisting conditions. They also see changes to their own health insurance plans, changes that would have come with Obamacare or not and are usually at the behest of their employers facing continued premium increases, and blame them on Obamacare. (And clearly those who have fundamental disagreements with Obamacare, or just want to see the President fail no matter what, exploit this to their advantage.)

Probably the main reason why I find the healthcare reform debate so frustrating is that most Americans do not understand many of the core issues around healthcare delivery and finance. In particular, they do not understand the difference between health insurance and healthcare expenditures. Very few Americans, only the very wealthy, can afford to pay for all healthcare costs. Instead, we all pay for healthcare insurance. Furthermore, free markets do not really work in most areas of healthcare, and it is debatable whether we should even try to make them work, as I noted in this blog during the height of the debate over healthcare reform legislation.

This ignorance is best exemplified by postings such as one on an anti-Obamacare site. The quote at the top of the women's healthcare portion of the site (reproduced as a picture below in case the site changes) lays bare how badly people misunderstand health insurance (private or public): “I had a hysterectomy, I have no need for maternity coverage, but I have to now pay for it. I have to pay not only my own premium but I have to subsidize everybody else.” (Kudos to JD Kleinke for pointing out this site in one of his blog postings. I also agree with another posting of his that Obamacare is more conservative, i.e., less liberal, than Medicare.)

The person quoted on this site obviously misunderstands the concept of health insurance. How many people not needing a hysterectomy subsidized this woman's hysterectomy? She obviously does not understand that the whole idea behind insurance is that we "subsidize" each other's needed care, so that when we need it ourselves, it is available for us. If we start lopping off this condition or that procedure from health insurance, then we soon lose the whole concept of insurance. (This is one of the reasons why most Obamacare insurance exchange plans will be more expensive than cut-rate plans that offer meager coverage and can be terminated at any time.) Carrying this woman's logic to an extreme, does she now no longer support paying for women who need a hysterectomy, since she no longer needs one?

Another manifestation of this thinking concerns Medicare. The famous quote "keep your government hands off my Medicare" (last couple paragraphs of this Washington Post article) best demonstrates how little many people truly understand about Medicare. Less blatantly, however, many elderly people who think nothing of demanding anything and everything from Medicare are the same people who are opposed to other forms of government-run health insurance, especially Medicaid for the poor. Yet these seniors do not realize that they are getting severalfold more benefits from Medicare than the contributions they have made over their lives (Fried, J (2008). Democrats and Republicans - Rhetoric and Reality: Comparing the Voters in Statistics and Anecdotes. New York, NY, Algora Publishing.).

But I do agree with those who argue that we cannot provide unfettered access to any and all types of care to everyone, seniors or otherwise. We do need to make some decisions as a society about what constitutes adequate healthcare coverage, and who should pay what. There are some areas where competition and free markets work in healthcare, and those should be encouraged. But the notion that we can buy less costly insurance policies, covering only this or that, really does not make sense.

I am willing to explore all the possible options for healthcare reform. Some conservative ideas make sense. But before we can have those discussions, a good proportion of the population needs to understand some basic realities about healthcare and its financing, and be willing to have an honest discussion about them.

Basic Statistics Should Be a Core Competency of Every Citizen of the World

A medical educator recently argued in her blog that medical school admissions requirements should minimize requirements in math and science topics, especially areas like calculus and physics. There is no question that medicine, and even informatics for that matter, require knowledge and competency in many areas beyond math and science.

However, the problem with the math we teach to potential healthcare professionals and informaticians, indeed to everyone in society, is that we teach the wrong math. I took three semesters of calculus in college and can say that I have almost never used any of it. On the other hand, I had almost no education in statistics, a type of math I use not only in my work, but also in my function as an informed citizen. Indeed, most healthcare professionals, whether clinicians or researchers, use statistics daily. Likewise, as thoughtful citizens in society, we also encounter statistics daily in the news and other aspects of our lives.

For these reasons, I believe that statistics should be a core competency of every citizen in the modern world.

It is not even the mathematics in statistics that are most important, but rather the concepts and the thinking they engender. Every citizen in the world should understand the basic concepts of inferential statistics and be able to answer such questions as:

• What does statistical significance mean? How is it different from a clinical (not necessarily in the medical context) significance?
• What is the difference between absolute and relative risk? What is the meaning of large relative risk differences in the setting of small absolute risk?
• In health-related topics, how do we discern and compare different types of health risks?
• Also in health, what do sensitivity and specificity of diagnostic tests mean, and how does prevalence impact the risk of disease in the face of positive or negative diagnostic tests?

One of the most articulate advocates of this view is John Allen Paulos, whose books Innumeracy and A Mathematician Reads the Newspaper inform us why basic numeracy and statistical competency are so important. These kinds of engaging writings, and basic education about statistics, should be a part of every high school education, not to mention in the education of clinicians and informaticians.

Universal EHR? No. Universal Data Access? Yes.

A recent blog posting calls for a "universal EMR" for the entire healthcare system. The author provides an example and correctly laments how lack of access to the complete data about a patient impedes optimal clinical care. I would add that quality improvement, clinical research, and public health are impeded by this situation as well.

However, I do not agree that a "universal EMR" is the best way to solve this problem. Instead, I would advocate that we need universal access to underlying clinical data, from which many different types of electronic health records (EHRs), personal health records (PHRs), and other applications can emerge.

What we really need for optimal use of health information is not an application but a platform. This notion has been advanced by many, perhaps most eloquently by Drs. Kenneth Mandl and Isaac Kohane of Boston Children's Hospital [1,2]. Their work is being manifested in the SMART platform that is being funded by an ONC SHARP Award.

Mandl and Kohane point to the iPhone as an example of building a platform on top of a common data store. I see this in action every day on my iPhone, when different applications make use of various data stores built into the phone, such as its GPS data. (Android and other phones offer similar functionality.) Not only Google Maps uses this data, but also my LA Fitness app that tells me where the nearest club is located when I am in a different city and hoping to find a gym.

A common data store, on top of which a thousand flowers (or apps) can bloom, is the ideal situation to the health information system "ecosystem." This will allow new ideas and innovations to flourish, while insuring that interoperable data will be accessible by all apps that have appropriate and authorized access. It will insure competition and a healthy marketplace to bring out the best in health information technology.

References

1. Mandl, KD and Kohane, IS (2009). No small change for the health information economy. New England Journal of Medicine. 360: 1278-1281.
2. Mandl, KD and Kohane, IS (2012). Escaping the EHR trap--the future of health IT. New England Journal of Medicine. 366: 2240-2242.

The Workforce Group of the ONC Health IT Policy Committee Makes Its Recommendations

For the last nine months, I have had the opportunity to be part of a workgroup of the ONC Health IT Policy Committee focusing on the health IT workforce issues. This week, Larry Wolf, co-chair of the workgroup made a presentation of the group's recommendations to a meeting of the full Health IT Policy Committee.

The recommendations of the workgroup can be summarized as follows:

1. ONC should summarize and publicize the results of the several workforce development programs it has funded.
2. ONC should summarize and widely disseminate the core competencies for members of the workforce that it has identified.
3. ONC should publicize the resources and best practices that they and other organizations have made available.
4. There is an emerging need for soft and hard skills related to team-based care, population health and patient engagement. ONC should recommend new program development and funding to address these needs.
5. ONC should learn from what is happening with the current workforce. It should do this by recommending funding of studies on the impact of health IT on the workforce, such as turnover, enrollment in healthcare vocations (schools), and new jobs, such as nurse informaticists.
6. The current Standard Occupational Classification (SOC) does not address health IT. ONC should host an SOC input process from the health IT community.

While I agree with all of the recommendations that our group made, I would have added two additional recommendations, which actually build on the fourth and fifth recommendations in the list. The first of these emanates in part from the slide presentation, which might be read by some to imply that informatics and IT jobs in healthcare are "technician" jobs. While healthcare organizations certainly need well-trained health IT technicians, this ignores the larger role that informatics will play as advanced health IT is adopted and used as an integral part of efforts such as quality measurement and improvement, accountable and coordinated care, and biomedical advances such as personalized medicine. Health IT is not merely a support function, but a critical component of our armamentarium to achieve the triple aim of improved health, better care, and lower cost.

My second additional recommendation builds on the recommendation for learning about the current workforce. In light of the larger role for health IT described in the previous paragraph, we need a much more comprehensive understanding than just impact on the current workforce and new jobs. We need to better understand not only of current workforce practices but also how to develop and educate the best workforce going forward into the new era of accountable and coordinated care and new advances such as personalized medicine.

I look forward to the continued efforts of the workgroup and our academic program at Oregon Health & Science University is certainly incorporating this forward-looking view as we revise and augment the curricula of our programs.

PCORI Clinical Data Research Network Funding Opportunity Casts Broad Vision for Clinical Data Use for Research

A couple weeks ago, the Patient-Centered Outcomes Research Institute (PCORI), an organization funded under the Affordable Care Act (ACA), released a funding opportunity to establish a series of Clinical Data Research Networks (CDRNs). The goal for these CDRNs is to develop a data infrastructure that will allow them to provide the data infrastructure for comparative effectiveness research (CER) to be done both within and outside their networks. (A separate announcement for complementary Patient-Powered Research Networks was also made and is included in the PDF announcing funding for the CDRNs.)

Naturally my institution is evaluating whether we have the resources and commitment to apply ourselves, and this gave me a reason to review the funding opportunity announcement in detail. I generated the following summary for my local colleagues, but also can state that no matter what we do, the funding opportunity presents a comprehensive vision for what CDRNs in general should look like, and indeed advances our moving toward the Institute of Medicine (IOM) vision of the learning health system [1].

A total of $56M will be awarded for eight or so $7M projects. Applicants can ask for a higher budget than $7M, but that must be approved by PCORI before a proposal at a higher funding level is submitted. The proposal process will entail two steps, the first of which is the submission of a letter of intent by June 19, 2013. PCORI will then invite some but not all of those submitting a letter of intent to submit full proposals, which will be due by September 27, 2013.

Each CDRN will be required to engage two or more different health systems and have one million or more patients enrolled among them. All systems will be required to have an electronic health record (EHR) system in place and the ability to standardize data among them. In essence, the system must adhere to the data standards specified by Stage 2 of the federal meaningful use program. This includes adoption of standards such as Consolidated Clinical Document Architecture (CCDA) for patient summaries, SNOMED CT for problem lists, RxNorm for electronic prescriptions, ICD-10-CM for diagnoses, ICD-10-PCS for procedures, CMS PQRI 2009 Registry XML Specification for quality reporting, and HL7 Version 2.5.1 for reporting of public health laboratory data and immunization administration. (A succinct overview of these standards in provided in a summary of all of the Stage 2 meaningful use requirements by Metzger and Rhoads [2].)

Another requirement is the ability of the CDRN to identify and recruit cohorts of patients with defined conditions. In particular, three patient cohorts must be able to be identified:

• A disorder of applicant’s choosing that includes 10,000 identified patients
• One or more rare diseases with prevalence less than 1 per 1500 persons in the US
• Overweight or obese patients, identified for presence of diabetes or pre-diabetes

Among the other requirements for the CDRN are:

• Ability to capture complete information on these patients over time, which could be a challenge due to patients getting care in multiple locations that has been shown to be widespread [3, 4]
• Process for patient as well as clinician engagement in governance as well as setting of research priorities.
• Commitment and active involvement of leadership of all participating organizations to play an active role in the governance of the CDRN and its meeting all key objectives
• Willingness to serve as a national data infrastructure resource for the conduct of CER by researchers both within and outside of the network - the award itself does not fund any research but the CDRN must engage CER researchers, including those outside network
• Demonstration of capacity to connect with patients for collecting data and recruiting them into clinical trials
• Capacity to support large CER randomized trials, with the embedding of research activity in functioning healthcare systems while not disrupting their usual healthcare business functions - this will require support of these activities from the respective administrative and executive leadership of participating organizations
• Integrating human subjects oversight, institutional review board (IRB) activities, and informed consent procedures across the network
• Policies to maintain data security, patient privacy, and confidentiality, along with organizational privacy
• Providing access to biological specimens for research purposes
• Robust governance, with the ability to identify and act upon unanticipated problems or issues
• Description of the efficient use of human and other resources to accomplish the project

One additional capacity I would have liked to see required would be participation in informatics and related research looking at the organization and function of the CDRNs and their data collection and usage issues. The development of CDRNs is necessary to advance the learning health system, but it is something for which there are still many unknowns. A research agenda to explore the issues of how to collect and use for research the operational clinical data on a network of multiple healthcare delivery systems and one million patients is critical to make sure our knowledge of how to do this effectively is learned and documented.

References

1. Smith, M, Saunders, R, et al. (2012). Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC, National Academies Press.

2. Metzger, J and Rhoads, J (2012). Summary of Key Provisions in Final Rule for Stage 2 HITECH Meaningful Use. Falls Church, VA, Computer Sciences Corp.

3. Bourgeois, FC, Olson, KL, et al. (2010). Patients treated at multiple acute health care facilities: quantifying information fragmentation. Archives of Internal Medicine. 170: 1989-1995.

4. Finnell, JT, Overhage, JM, et al. (2011). All health care is not local: an evaluation of the distribution of emergency department care delivered in Indiana. AMIA Annual Symposium Proceedings, Washington, DC. 409-416.

The Road to Big Data Passes Through Informatics

I have written a number of postings over the last year about various aspects of electronic health record (EHR) data, from the transition of the work of informatics from implementation to analytics to the problems that still prevent us from making optimal use of data, such as the difficulties of data entry. One of my themes has been that knowledge will not just fall out of the data; we will need to improve the quality and completeness of data to learn from it. The requirements for getting better data include widespread adherence to data standards, engaging and motivating those who enter data to improving it, making it easier for those individuals to enter quality data, and evolving our healthcare system to valuing this data. If we are not able to meet these challenges with our current data, it is unlikely we will be able to do so when we have "big data," i.e., that which is orders of magnitude larger and more complex beyond what we have now. No field has devoted more thought, research, or evaluation to the challenges of clinical and health data than informatics. Thus, whether it is tackling issues of how to implement systems in complex clinical settings; meeting the needs of clinicians, patients, and others; or how to maximize the quality of data, the road to making the best use of (big or non-big) data must pass through informatics.

An example of the fact that knowledge will not just fall out of the data comes from some research activity I have been involved in over the last couple years, which is the Text Retrieval Conference (TREC) Medical Records Track [1]. As those familiar with the field of information retrieval (IR) know, TREC is an annual "challenge evaluation," sponsored by the National Institute for Standards and Technology (NIST) [2]. Challenge evaluations bring research groups with common interests and use cases together to apply their systems to a common task or set of tasks, using a common data set, and comparing results using agreed-upon metrics (ideally in a scholarly and not an overly competitive forum). TREC operates on a yearly cycle, consisting of 5-7 "tracks" that each represent a specific focus of IR research. TREC began with the straightforward tasks of "ad hoc" retrieval (user entering queries into a search engine seeking relevant documents) and "routing" (user seeking relevant documents from a new stream of documents based knowledge of previous relevant documents). In subsequent years, TREC evolved to its current state of diverse tracks representing newer problems in IR, such as Web search, video searching, question-answering, cross-language retrieval, and user studies. (Some of these tracks have spawned their own challenge evaluations, especially in the area of cross-language evaluation, an important issue in Europe and Asia.) Virtually all tracks have focused on generic content, typically newswire or Web content, with very few being "domain specific," although I have been involved in two domain-specific tracks in the areas of genomics literature [3] and medical records [1].

In TREC and IR jargon, test collections consist of an adequately large and realistic collection of content, such as documents, medical records, Web pages, etc. [2]. Test collections also include a set of topics, usually at least 25-50 for statistical reliability [4], that are instances of the task being studied. A final component is human relevance judgments or assessments over the content items, indicating which are relevant and should be retrieved for each topic. Success is usually measured by some sort of aggregate statistic that combines the base measures of recall (proportion of relevant content items in the test collection retrieved) and precision (proportion of relevant content items in the search retrieved). (For those familiar with medical diagnostic test characteristics, these correspond to sensitivity and positive predictive value. The reciprocal of precision is also sometimes called number needed to retrieve, since it measures how many overall documents must be read or viewed for each relevant one retrieved.)

The use case for the track TREC Medical Records Track was identifying patients from a collection of medical records who might be candidates for clinical studies. This is a real-world task for which automated retrieval systems could greatly aid in ability to carry out clinical research, quality measurement and improvement, or other "secondary uses" of clinical data [4]. The metric used to measure systems employed was inferred normalized distributed cumulative gain (infNDCG), which takes into account some other factors, such as incomplete judgment of all documents retrieval by all research groups.

The data for the track was a corpus of de-identified medical records developed by the University of Pittsburgh Medical Center. Records containing data, text, and ICD-9 codes are grouped by "visits" or patient encounters with the health system. (Due to the de-identification process, it is impossible to know whether one or more visits might emanate from the same patient.) There were 93,551 documents mapped into 17,264 visits.

I was involved in a number of aspects of organizing this track. I contributed in both guiding the task (or use case) as well as leading some of track infrastructure activities, namely development of search topics and relevance assessments. This work has been aided greatly by students with medical and other expertise in the OHSU Biomedical Informatics Graduate Program.

The results of the TREC Medical Records Track provide a good example of why the road to big data passes through informatics, or in other words, why there is still considerable work to be done from an informatics standpoint before knowledge simply falls out of data. While the performance of systems in the track has been good from an IR standpoint, they also show these systems and approaches have a considerable ways to go before we can just turn the data analytics crank and have medical knowledge emanate. The magnitude of how far we need to go comes from the precision at various levels of retrieval (e.g., precision at 10 retrieved, 50 retrieved, 100 retrieved, etc.), demonstrating how many nonrelevant visits are retrieved. In the case of typical ad hoc IR, we can probably quickly dispense with documents are relatively easy to identify as not relevant. But this may be a more difficult task for complex patients and complex records.

A failure analysis over the data from the 2011 track carried out at OHSU demonstrated why there are still many challenges that need to be overcome [5]. This analysis found a number of reasons why visits frequently retrieved were not relevant:

• Notes contain very similar term confused with topic
• Topic symptom/condition/procedure done in the past
• Most, but not all, criteria present
• All criteria present but not in the time/sequence specified by the topic description
• Topic terms mentioned as future possibility
• Topic terms not present--can't determine why record was captured
• Irrelevant reference in record to topic terms
• Topic terms denied or ruled out

The analysis also found reasons why visits rarely retrieval were actually relevant:

• Topic terms present in record but overlooked in search
• Visit notes used a synonym for topic terms
• Topic terms not named and must be derived
• Topic terms present in diagnosis list but not visit notes

A number of research groups used a variety of techniques, such as synonym and query expansion, machine learning algorithms, and matching against ICD-9 codes, but still had results that were not better than manually constructed queries (which also require a form of informatics expertise in knowing how to query the clinical domain). The results data also show this is a challenging task, as the performance of different systems varied widely on different topics.

From my perspective, these results show that successful use of big data will not come just from smart algorithms and fast computer hardware. It will also require the informatics expertise to design and implement EHRs, high-quality and complete clinical data, and a proper understanding of the clinical/health domain to make most effective use of the data. As such, achieving the value of big data passes through informatics.

References

1. Voorhees, E and Hersh, W (2012). Overview of the TREC 2012 Medical Records Track. The Twenty-First Text REtrieval Conference Proceedings (TREC 2012), Gaithersburg, MD. National Institute for Standards and Technology.

2. Voorhees, EM and Harman, DK, Eds. (2005). TREC: Experiment and Evaluation in Information Retrieval. Cambridge, MA, MIT Press.

3. Hersh, W and Voorhees, E (2009). TREC genomics special issue overview. Information Retrieval. 12: 1-15.

4. Buckley, C and Voorhees, E (2000). Evaluating evaluation measure stability. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 33-40.

5. Edinger, T, Cohen, AM, et al. (2012). Barriers to retrieving patient information from electronic health record data: failure analysis from the TREC Medical Records Track. AMIA 2012 Annual Symposium, Chicago, IL, 180-188.

Biomedical and Health Informatics vs. Data Science, mHealth, etc. - New Disciplines or New Terminology?

When I entered the field of informatics in the 1980s, a great deal of the research was driven by "artificial intelligence" (AI). Many people were trying to build "rule-based expert systems," while those interested in knowledge representation were constructing "semantic networks." We rarely hear these terms in quotes these days, perhaps with the exception of AI that one hears occasionally. It is not, however, that no one is trying to build systems that guide decision-making and represent knowledge in complex ways, but we just different terminology now, such as clinical decision support and ontologies.

Fast forward to the present, and we see the introduction of new terms, most prominently right now data science [1] and mHealth [2]. Many who are doing work in these areas talk of them as the primary focus of their work. I question, however, whether these are truly new disciplines, or just concentrations (at least for those working in health-related areas) within biomedical and health informatics [3]?

I am most concerned about mHealth, when I see new people coming forward with brilliant ideas and truly innovative technologies, yet not incorporating the experiences from decades of work in informatics. I do not deny that some aspects of using mobile connected devices for health are truly novel, yet what I consider to be the basic principles of informatics still apply, namely things like scalability, interoperability, usability, and so forth. I just see nothing novel enough about mHealth to not call it part of informatics.

The same holds, in my opinion, for data science. There are certainly "computationalist" techniques of which many who work in informatics are not skilled. "Big data" applications will require specialized knowledge. But informatics is a broad field, and no one can master everything. There are other aspects of informatics, such as (I am repeating myself from the previous paragraph here) scalability, interoperability, usability, and so forth that must be married from the results of data science to make the latter's output truly usable. One case in point is the growing number of analyses that predict undesired outcomes, such as hospital readmissions [4]. I am as intellectually interested in these applications as much as anyone, but until it is shown these analyses can be actionable, they will mostly remain interesting theoretical exercises.

I am excited for mobile health applications and advanced uses of data techniques to improve health, healthcare, and research. I hope that those pursuing them do not lose sight of the larger picture of providing end-to-end value for the use of data, information, and knowledge in health-related endeavors, i.e., the goal of biomedical and health informatics [3].

References

1. Davenport, TH and Patil, DJ (2012). Data Scientist: The Sexiest Job of the 21st Century. Harvard Business Review, October, 2012. http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/.
2. Krohn, R and Metcalf, D (2012). mHealth: From Smartphones to Smart Systems. Chicago, IL, Healthcare Information Management Systems Society.
3. Hersh, W (2009). A stimulus to define informatics and health information technology. BMC Medical Informatics & Decision Making. 9: 24. http://www.biomedcentral.com/1472-6947/9/24/.
4. Gildersleeve, R and Cooper, P (2013). Development of an automated, real time surveillance tool for predicting readmissions at a community hospital. Applied Clinical Informatics. 4: 153-169.