Friday, July 1, 2022

Reflections on Publishing a Negative Study

Although scientists are supposed to be disinterested observers of the results of their work, the reality is that none of us really want to see a negative result from our research. This is especially the case in fields like biomedical and health informatics, where we aim to develop tools that can be used to help patients and/or clinicians with their health or for delivery of care.  Nonetheless, it is critically important to publish such outcomes, especially if the research was methodologically sound. This is especially the case in currently hyped technologies such as machine learning.

One of the challenges for negative studies is the myriad of potential reasons why they failed, which sometimes makes the true reason hard to pinpoint. Was it poor methods, inadequate data, or mis-application of the technology? Or more than one of those? But especially in the case of studies that involve human subjects who may be exposing themselves to risk, we must always remember the words of noted physician-scientist Iain Chalmers, who stated that failure to publish studies with patients is a form of scientific misconduct.(1)

A couple years ago, my colleague Aaron Cohen and I received some funding from a pharmaceutical company that had developed a highly effective new treatment for a rare disease, acute intermittent porphyria (AIP).(2) The new treatment, although expensive, significantly reduces the severe and sometimes disabling manifestations of AIP. As such, tools to identify patients who might have the disease could significantly improve their lives. Our approach used data from their electronic health record (EHR).

We were thrilled when our machine learning model was found to identify a group of patients with the classic presentation of AIP, yet the diagnosis had never been considered.(3) AIP is known to be one of those rare diseases that often goes undiagnosed for long periods of time. Diagnosis of these types of rare diseases is considered to be a major use case for machine learning in healthcare. From a collection of over 200,000 patient records in the research data warehouse at our institution, we manually reviewed the top 100 ranking patients identified by the model. Twenty-two of these patients were determined to have the manifestations of AIP but with no occurrence of the string "porph" anywhere in their record, i.e., not in lab tests, notes, or diagnoses. As the test to diagnose AHP is a relatively simple and inexpensive urine porphobilinogen test, we developed a clinical protocol to invite such patients for testing.

We encountered a number of challenges in developing and implementing the clinical protocol. For example, what is the role of the patient's primary care or other provider in the decision to offer testing? Our institutional review board (IRB) determined that the process should a decision by the provider, who would then allow us to contact the patient to offer testing. Once given approval by the provider, we then had to convince the patients, many of them skeptical of the healthcare system long unable to diagnose their symptoms, to come in for testing. Another challenge in implementing the protocol was that it took place during the COVID-19 pandemic, when many people wanted to avoid healthcare settings, though by the time we were recruiting patients, the COVID-19 vaccines had started to become available.

As a result, we could only convince 7 of the 22 patients to undergo the simple and free test we were offering them. And as noted in the paper, none of them were positive.(4) Because these patients did submit to a clinical protocol, and because we believe that any attempt to clinically validate machine learning is important, we decided to submit our results to a journal as a Brief Communication, i.e., not the definitive study but clearly worth reporting. The report has now been published.

Clearly there are many possible reasons for our failure to diagnose any new cases of AIP. Not the least of these is the fact that AIP is a rare disease, and there may have been few if any new cases to be found. Certainly if we had more resources or time, we could have invited for testing more people beyond the first 100 cases we evaluated who may also have been found to have the classic presentation yet for whom the diagnosis was never considered. It is also possible that our machine learning model could have been better at identifying true cases of the disease. There may have been confounders in that those actually coming for testing were not the most likely to have the diagnosis.

Could this study have benefited from proactive genotyping? There may be a role for gene sequencing in this situation, but AIP is a condition of incomplete penetrance, i.e., just because one has the genotype does not mean they will develop the manifestations of the disease. In other words, genotyping would not be enough.

My main hope is that this contribution shows that we should be pursuing such studies aiming to apply machine learning in real-world clinical settings. As one who has been critical that machine learning has not been "translational" enough, I hope there will be a role for larger and more comprehensive studies of many different uses that have been found to be beneficial in model-building studies.

References

1. Chalmers, I., 1990. Underreporting research is scientific misconduct. JAMA 263, 1405–1408.

2. Balwani, M., Sardh, E., Ventura, P., Peiró, P.A., Rees, D.C., Stölzel, U., Bissell, D.M., Bonkovsky, H.L., Windyga, J., Anderson, K.E., Parker, C., Silver, S.M., Keel, S.B., Wang, J.-D., Stein, P.E., Harper, P., Vassiliou, D., Wang, B., Phillips, J., Ivanova, A., Langendonk, J.G., Kauppinen, R., Minder, E., Horie, Y., Penz, C., Chen, J., Liu, S., Ko, J.J., Sweetser, M.T., Garg, P., Vaishnaw, A., Kim, J.B., Simon, A.R., Gouya, L., ENVISION Investigators, 2020. Phase 3 Trial of RNAi Therapeutic Givosiran for Acute Intermittent Porphyria. N Engl J Med 382, 2289–2301.

3. Cohen, A.M., Chamberlin, S., Deloughery, T., Nguyen, M., Bedrick, S., Meninger, S., Ko, J.J., Amin, J.J., Wei, A.J., Hersh, W., 2020. Detecting rare diseases in electronic health records using machine learning and knowledge engineering: Case study of acute hepatic porphyria. PLoS ONE 15, e0235574.

4. Hersh, W.R., Cohen, A.M., Nguyen, M.M., Bensching, K.L., Deloughery, T.G., 2022. Clinical study applying machine learning to detect a rare disease: results and lessons learned. JAMIA Open 5, ooac053.

Wednesday, June 29, 2022

2022 Update of Informatics.Health

I am pleased to announce an update of my Web site that provides an introductory overview of biomedical and health informatics. Entitled, What is Biomedical & Health Informatics?, I created this site two decades ago to provide an answer to that question I used to be asked (less so in modern times). I still maintain and keep the site up to date both to continue to provide an overview of the field as well as demonstrate a portfolio of some of the learning technology we use in our virtual courses in the Biomedical Informatics Graduate Program at Oregon Health & Science University (OHSU).

In recent years I was able to secure the domain name, Informatics.Health.

With the 2022 updating of my larger course that is offered in the American Medical Informatics Association (AMIA) 10x10 ("ten by ten") program, I have now updated the content of the WhatIs site. The main part of the site is the 11 lecture segments on the following topics:

  • What is Biomedical and Health Informatics? (1) (24:32)
  • What is Biomedical and Health Informatics? (2) (18:49)
  • A Short History of Biomedical and Health Informatics (22:30)
  • Resources for Field: Organizations, Information, Education (25:29)
  • Clinical Data (15:08)
  • Examples of the Electronic Health Record (EHR) (24:56)
  • Data Science and Artificial Intelligence (1) (14:15)
  • Data Science and Artificial Intelligence (2) (22:07)
  • Information Retrieval (Search) (23:18)
  • Information Retrieval Content (29:09)

The lectures can be viewed on just about any Web platform, and work fine on mobile devices. The site also contains links to books, articles, organizations, and educational Web sites. The materials on the site are freely available and have been used by many educators and others.

Wednesday, April 6, 2022

3000 by 2022 - A New Milestone for the 10x10 Course

One of the most enjoyable and impactful activities of my career has been the 10x10 ("ten by ten") course, which is an online introductory course in biomedical and health informatics that I teach in partnership with the American Medical Informatics Association (AMIA). The course recently reached a new milestone, surpassing 3000 people completing the course since its inception in 2005. My course from Oregon Health & Science University (OHSU) was the original and is the largest course of the AMIA 10x10 Program.

The OHSU 10x10 course provides an up-to-date detailed introductory overview of biomedical and health informatics. The content delivery is online, followed by an optional in-person session, typically at an AMIA meeting, at its conclusion. About 10-15% of those completing the course take the optional final exam and pursue further study in the field, usually but not exclusively at OHSU.

The 10x10 program gets its name from the program's original goal in 2005 to train 10,000 people in biomedical and health informatics by the year 2010. The goal for 10,000 people came from a stated need by then-President of AMIA Dr. Charles Safran that there should be at least one physician and one nurse trained in informatics in each of the approximately 6000 hospitals in the US. I adapted an already-existing online introductory informatics course and we set the goal of 10,000 people completing the course by 2010, leading me to devise the moniker, 10x10. Even though 10,000 people did not come forward to take the course by 2010, about 1000 people did complete the OHSU offering by 2010.
 
After 2010, the 10x10 program continued to attract enrollment. The positive feedback of those completing the course has always been gratifying. Now, a new milestone has been reached with the cohort who recently completed the course, surpassing a total of 3000 individuals having completed the OHSU 10x10 course since 2005. A total of 111 offerings of the OHSU course have been provided since that time. While most of the course enrollment has come from the US, several international organizations have partnered to offer the course based in their countries, including Gateway Consulting (Singapore), King Saud University (Saudi Arabia), the Ministry of Health (Israel), and Abu Dhabi Health Services (SEHA, United Arab Emirates). There have also been partnerships to offer the course with domestic organizations in the US, including the American College of Emergency Physicians, the Academy of Nutrition and Dietetics (AND), the New York State Academy of Family Practice, the Mayo Clinic, the Centers for Disease Control and Prevention, and others. The table below shows the course partners, the number of offerings with them, and the number of people completing the course.
 
 
The figure below shows the enrollment by year. The largest amount of enrollment came during the heyday of the "meaningful use" era in the early 2010s, dipped somewhat later in the decade, but has continued strong since then.

 
Due to the online delivery of the course, it had no trouble continuing during the COVID-19 pandemic, although the in-person sessions at the end of the course were transitioned to virtual delivery. With the receding of the pandemic, those sessions have returned now to an in-person format. Of course, the content of the course was affected by the pandemic and its impacts on the field. However, the more fundamental topics of the field, including electronic health records, standards and interoperability, machine learning and artificial intelligence, and others have still been covered in a comprehensive and up-to-date manner.

The course will continue to be offered beyond this milestone, and in fact a new offering is beginning this month. In-person sessions for those completing the course will take place at upcoming AMIA meetings this year, including the Clinical Informatics Conference (Houston, TX - May 23-26) and the Annual Symposium (Washington, DC - November 5-9).

As the OHSU agreement with AMIA is non-mutually-exclusive (i.e., AMIA can offer other 10x10 courses and OHSU can use the course materials in other courses), the content of the OHSU 10x10 course has been delivered to students under other course names. The other major use of the content is in the course, BMI 510/610, which is the introductory course in the Health & Clinical Informatics Major of the OHSU Biomedical Informatics Graduate Program. This course has been offered since the OHSU graduate program was launched in 1996 and every academic quarter since 2000. After the most recent academic term, 1598 people have completed this instance of the course.

The course content was also converted into a virtual two-week block course for medical students at the onset of the pandemic. The course was made available to medical students within and outside OHSU, with 44 OHSU medical students participating in three offerings and 178 medical students from outside OHSU participating in eight offerings during 2020. This block course continues to be offered to OHSU medical students.
 
Other offerings of the course have been provided to health systems, including Kaiser Permanente Northwest, Providence Health and Systems, and Bangkok (Thailand) Hospital. The course has also been offered to H3ABioNet, the Pan African Bioinformatics Network for the Human Heredity and Health in Africa (H3Africa) consortium, of which I serve on the Scientific Advisory Board.
 
The course  also inspired federal legislation in the US, leading to the 10,000 Trained by 2010 Act, introduced by Rep. David Wu (D-OR). The bill was passed by the US House in the 111th Congress (2009-2010), and elements of it were incorporated into the HITECH Act. The HITECH program included funding for Health IT Workforce Development, including grants awarded to OHSU.

Probably the most gratifying aspect of the course is the feedback I get from students who appreciate the broad but sufficiently deep content, i.e., getting the big picture while also learning the rationale behind it. I enjoy keeping in touch with those who have taken the course, especially when I run into them in airports, at conferences, or other places. I also value the motivation that the course provides to me in keeping up with advances across the entire field of biomedical and health informatics.

Friday, March 25, 2022

Clinical Informatics Subspecialty Practice Pathway Extended for Three Additional Years

A three-year extension to the Practice Pathway of board certification eligibility for the clinical informatics (CI) subspecialty has been approved by the American Board of Medical Specialties (ABMS) for the American Board of Preventive Medicine (ABPM). This is the second extension of the so-called "grandfathering" pathway that now allows physicians with a primary boarded specialty to achieve board eligibility without formal fellowship training through 2025. This means that physicians who desire to become board-certified in CI will be able to qualify to sit for the board examination by time working in the field or completing "non-traditional" training, the latter which may include a master's degree from a "24 month Masters or PhD program in Biomedical Informatics, Health Sciences Informatics, Clinical Informatics, or a related subject from a university/college in the US and Canada, deemed acceptable by ABPM (e.g. NLM university-based Biomedical Informatics Training)," such as the online Master of Science Program at Oregon Health & Science University (OHSU).

The CI field has expressed mixed feelings on this extension. In particular, a group of CI fellowship Program Directors published a letter in the journal Applied Clinical Informatics (ACI) opposing the extension, noting that the time for grandfathering has passed, and extending the practice pathway will adversely impact pursuit of fellowships.[1] I rebutted this letter, arguing that the field must find alternatives to two-year in-place fellowships to allow broader entry into the field.[2] In particular, a two-year in-place fellowship may place undue burdens on those who wish to become board-certified in CI long after they completed their primary training and would be unable to uproot from job, family, and/or other obligations.

Here are some snippets from my ACI letter arguing for alternatives to in-place fellowships:

I agree that we have passed the point where the Practice Pathway should allow physicians to become board-certified with essentially no formal training. However, I argue instead for this approach to be transformed into a method by where those who are unable to halt careers, salary, and family to pursue a pathway to certification that is mostly virtual and asynchronous yet still rigorous and supervised. Ironically, the pandemic has taught us that CI practice and education can be carried out in a mostly virtual format.

The in-place model for fellowship training made sense in the 20th century model of career development, where one completed education and training in their chosen profession and then entered the workforce for their career. In the 21st century, however, many professionals, especially in knowledge careers, change career pathways long after their primary education and training experience.

I oppose CI fellowships being completely remote, but it would be novel and innovative if there were some sort of hybrid training pathway, with fellows connected to an institution that could offer courses and allow supervised, mentored training experiences in healthcare organizations. Fellows would participate in a mostly remote way, but also have periodic in-person experiences, including stints that might be for several weeks or more and would involve direct interaction with faculty and colleagues. The field of Hospice and Palliative Medicine developed such an approach prior to the COVID-19 pandemic. Even CI somewhat emulates this approach now, as a half-dozen CI fellowships make use of online didactic courses from OHSU.

I applaud that for now the Practice Pathway will still allow those to pursue board certification. Hopefully the CI field can transition to a training process beyond the Practice Pathway that allows entry into the field without an in-place fellowship. As informaticians, we should be at the forefront of pioneering this approach in graduate medical education.

What are eligibility requirements for the CI subspecialty? They are essentially unchanged from the last time I posted about them in this blog in 2019, with the exception that the "grandfathering" pathway is now available through 2025.

References

1. Turer, R.W., Levy, B.P., Hron, J.D., Pageler, N.M., Mize, D.E., Kim, E., Lehmann, C.U., 2022. An Open Letter Arguing for Closure of the Practice Pathway for Clinical Informatics Medical Subspecialty Certification. Appl Clin Inform 13, 301–303. https://doi.org/10.1055/s-0042-1744386.
 
2. Hersh, W.R., 2022. The Clinical Informatics Practice Pathway Should Be Maintained for Now but Transformed into an Alternative to In-Place Fellowships. Appl Clin Inform 13, 398–399. https://doi.org/10.1055/s-0042-1745722.

Friday, March 11, 2022

Receding of the Pandemic: Will the Third Time Be the Charm?

Today, the Governor of Oregon is lifting the state's indoor mask mandate and ending the state's public health COVID-19 emergency. Like most US states, Oregon had a large Omicron wave of cases, hospitalizations, and deaths, although as through all of the pandemic still far below US national averages. At Oregon Health & Science University (OHSU), the number of patients in the hospital and ICU continues to fall each day. The mask mandate in non-healthcare buildings at OHSU will be dropped tomorrow and my department will return to some activities in-person with the start of the spring quarter at the end of March.

Will the pandemic finally recede now and allow us to return to an albeit new normal? We have been down this road before. The first came in the late spring and early summer of 2021. Vaccination had become relatively widespread, and the large wave of hospitalizations and mortality from late 2020 and early 2021 appeared to be subsiding. In Oregon, all mask and other public health mandates were lifted, and life seemed to be returning to normal.
 
Sadly, however, the Delta wave started in the late summer of 2021 and dashed hopes that the fall would see a return to relative normal. As the Delta wave subsided in the late fall, a second era of opportunity seemed to be coming again. Although somewhat more muted than the first reprieve, it looked as if a modified normal might occur in early 2022.

But then, of course, Omicron came, and with it a new wave of hospitalizations and mortality. One fortunate aspect of the Omicron wave is that there has been clear evidence that vaccination provides protection. Even if not preventing SARS-CoV-2 infection completely, vaccination does appear to limit the worst of the infection for most people.

Now the Omicron wave is receding, and hopefully the worst of the pandemic with it. Although some might be gun-shy to feel optimistic, it is clear that there can be a path to living with the virus and a new approach that balances public health requirements with individualizing risk. I look to those physician experts who express cautious optimism and allowing of individual decision-making within the extremes of those at the ends of the spectrum. My favorites include Ashish Jha, Bob Wachter, Leana Wen, and of course the prolific Eric Topol. I also believe that the federal government's new COVID-19 plan is reasonable, with its emphasis on vaccination, testing, and treatment. This is especially the case with new oral anti-viral treatments shown to be highly effective.

The most unfortunate aspects of this pandemic has been its politicization, it becoming part of the culture wars in the US. I am not unsympathetic to those who want to move on. I do not particularly enjoy wearing masks, even though I do so and will continue doing so when it is necessary to protect myself or others. It saddens me that one of the most speedy and effective vaccines ever developed is being met by so much resistance. In addition, the manipulation and misinformation is saddening for a medium I always hoped would lead to dissemination of communication and knowledge across humanity. Wishful thinking, I suppose.

I find particularly sad the misunderstanding of science and the gotcha politics of when knowledge changes based on new research. One of the best quotes comes from Mohamad Safa, an environmental activist who stated on Twitter: "Science is not truth. Science is finding the truth. When science changes its opinion, it didn't lie to you. It learned more."

As the pandemic recedes, what will be my approach? Being relatively healthy and in my early 60s, I would probably weather a COVID-19 infection reasonably well. I will not go out looking for one, but I also will not have dire fear of getting one. One irony about the pandemic period is that I have not been infected so much as a cold, which I normally get once to twice per year. I will also respect the public health concerns for COVID-19. I will not hesitate to wear a mask when I am asked to do so, especially when it protects someone who might be at higher risk of complications from infection. I will also continue to join the chorus of those advocating for more vaccine equity across the world.

The months ahead will be a large natural experiment in the United States, as mandates are lifted. We will see whether the collective immunity we have achieved through vaccination and natural infection will be enough to keep SARS-CoV-2 under control, especially those at highest risk of complications from infection.

Tuesday, January 11, 2022

Considerations and Transitions for Narrating Lecture Slides

For over two decades, a significant portion of my teaching at Oregon Health & Science University (OHSU) has been online, and one of the main modalities I have used is narrated lectures, in particular, voice-over-Powerpoint slides. I know that many bemoan the use of Powerpoint for any presentations, including teaching. However, I find slides a very useful way to organize information, utilize simple graphics, and develop a big picture for the knowledge I aim to convey. As I use my teaching slides for other purposes, such as giving talks, another critical need for me is to maintain the source Powerpoint files.

This makes my choice of tool(s) to create voice-over-Powerpoint files critically important in my work. Over the years, I have used different tools for doing so. My initial foray used a technology that some Internet long-timers might remember, called RealMedia. Another imperative in the early days, circa 1999, was that the files not be too large so as not to require substantial bandwidth, since those were the early days of broadband Internet, and many people were still connecting, especially from home, via telephone modems.

Another lesson learned in those early days was that learners preferred online lectures to be broken down into 15-20 minute segments. This also made recording easier, as one did not have to record an entire lecture in a single session.

Over time, there was development of new tools. RealMedia eventually fell out of favor with many users and developers, and for a while I used a now-defunct tool from Adobe called Presenter. It used an output format that was very popular for many years, Flash, which was retired a few years ago.

In 2007, I discovered another tool that served me well for a long time, Articulate Presenter. One of its key features was the ability to record a single slide at a time. All of the tools I used before then required the lecture over the whole slide deck to be recorded in a single session. Being able to record slide by slide was useful for several reasons, not the least of which was that most of my updating of curricular content involved incremental update of single slides and not the entire lecture segment. In other words, a small part of a topic might change, and by being able to update individual slides, I could update just those slides than needed it.

Another feature I valued from Articulate Presenter was being able to configure the output with a navigation and notes frame on the left, enabling individuals slides to navigate to specific slides, and for the text in the notes field of the slides to be viewed. The output was generated in Flash when it was in its heyday. and then transitioned to HTML 5 as Flash was being retired.

But there is one big limitation of Articulate Presenter, which is that it only runs on Microsoft Windows. My primary computer when I started using Articulate Presenter was a Windows PC, but in the early 2010s, I was feeling the pull to return to the Macintosh platform that I had moved away from in the mid-1990s. Part of the allure was that Apple itself was transitioning to Intel chips, meaning that Macintosh computers could run Windows and its applications.

I made the transition back to the Mac in 2012, and fairly seamlessly moved my Articulate Presenter work to running Windows on Parallels. But although the solution worked, it was somewhat clunky, and having to launch Windows to do any work in Articulate Presenter required a number of steps. Despite the protestations of myself and many other users (evidenced in their support forums), Articulate never developed a native Mac version of Presenter.

In the last couple years, of course, Apple has transitioned away from Intel chips. This is a good thing for Mac users generally, as the speed and power consumption is remarkable. And while there is supposedly a version of Windows that runs on the ARM chips that Apple is now using, it does not run most native applications, including Articulate Presenter.

Fortunately during this same time, Microsoft has been improving the slide narration capabilities, and one can now easily narrate slide by slide directly in Powerpoint. And then the output can be exported to a standard video format such as MP4 or MOV. In addition, OHSU uses the Echo 360 platform, which allows video files to be posted to a server. While the video output of Powerpoint does not have all the navigation features of Articulate Presenter, the learner can still navigate around the lecture. Echo 360 also offers automated closed-captioning, although it is far from perfect.

This transition will require me to re-record all lectures completely, but many lectures are due for that anyways. In addition, I can record in Powerpoint slide by slide. Using Powerpoint on my Mac is also much less clunky that using Articulate Presenter in Windows on a Mac. It is especially easy when I need to make a small update, such as finding and fixing a typo in a slide, and not having to launch Windows, and then Articulate Presenter.

One can see an example of the results of my new approach in the image below from a Web site that I maintain, What is Biomedical and Health Informatics? I have maintained this site for many years, both as a way to introduce the world to my view of the field of informatics and also a showcase for the online learning modalities that our program uses. The site also has a great domain name I was able to snag when .health was made a top-level domain, http://informatics.health/. I have redone the lectures on this site using my new approach.

Certainly a bottom line is that the ease of updating lectures far outweigh the small additional features provided by Articulate Presenter. It may seem somewhat ironic that improvements in Microsoft Powerpoint now give me the ability to move on from Microsoft Windows and Articulate Presenter. But I am pleased that I can now can evolve with Apple as it innovates its hardware and software with its new ARM chips.

Friday, December 31, 2021

Annual Reflections at the End of 2021

As is the tradition with this blog, I end each year with a reflective look at the year past and what the future may hold. The year 2021 is not ending quite like I anticipated. At the beginning of the year, there were stirrings of optimism. The new COVID-19 vaccines had been released, and a new approach to political leadership had just been elected in Washington, DC. On January 2nd, I received my first dose of the Pfizer vaccine, followed by a second one three weeks later.

By the spring, the pandemic was seeming to wane. In late June, Oregon lifted all pandemic restrictions, and life seemed to be returning to relative normal. At work, my department began making plans for returning to the office, aiming to slowly transition over the summer and then return to a new state of normal when the regular academic year started in the fall.
 
Alas, it was not to be. By August, the Delta wave was surging and plans for returning to the office were postponed. Things became more optimistic into the fall, with my getting my booster vaccine in early October, but then, just when it seemed that the Delta wave was on the decline, the new Omicron variant emerged.

Nonetheless, there is some reason for optimism in the long run for the pandemic. The vaccines are having an impact, even if mostly among the vaccinated. Clearly the vaccines are not completely protective against infections, but they do appear to blunt the worst complications of the disease. In addition, there are new COVID-19 treatments becoming available in pill form, such as Paxlovid, which may provide a means to further prevent the worst impacts of infection. With my three doses of Pfizer vaccine on board, I feel relatively well protected, and feel confident that if I were to have a breakthrough infection, the course would be mild. (Though I am not taking any excessive chances.)

Although I look forward to getting back to working in person, I have managed to maintain my own productivity by working mostly from my home office. One thing that the pandemic has taught the informatics field, and certainly the work of my own department, is that we can pretty much function virtually. I still enjoy interacting in person with colleagues, but clearly most of the research and educational work we do can be carried out in a virtual manner.
 
I am sad that this pandemic has been so politicized. Every happening, often having nothing to do with the action of politicians, seems to require being viewed through a political lens. While there is, as with all science, some room for debate on translating scientific findings into public policy, it is unfortunate that many, especially those on the right, have chosen to frame everything in that context and make decisions that defy sound public health practice. I don't enjoy wearing masks or avoiding large gatherings more than anyone, but I recognize these as a small price to pay to protect public health. And it is sad to me that new vaccine technology, one of the greatest public health advances in human history, has also become a political Rorschach test.

The ability to function virtually would not be possible without the maturing of computing and network technology. It is still far from perfect, but we almost take for granted now that we can have a synchronous video call with almost any location on the planet. Social media has played a valuable role too, especially in my case, Facebook. While I am no big fan of Facebook's business model, and I would happily pay a modest subscription fee to get the benefits of it that I like, it has been invaluable in keeping up personally and professionally with friends and colleagues around the world.

In terms of the informatics field, I am optimistic that we will continue the contributions we make to health and larger society. While the pandemic has caused a detour in our priorities, we are still moving forward in core advances, particularly turning the potential of machine learning and artificial intelligence into reality and advancing standards to make data more interoperable. The pandemic has exposed the limits of our current health information systems and processes, and even if too late to provide optimal value for this pandemic, will hopefully put us in a position to benefit the next public health crises.

This blog itself continues to be successful. This is the 361st post, and it is closing in on 768,000 page views over its nearly 13 years. This blog has provided me an opportunity to provide commentary on many topics, mainly professionally but also personally, and is easily accessible from the Internet. My frequency of posting has decreased some over the years, but I still appreciate the opportunity to weigh in on important topics and not feel the need to post at any specific level of frequency.

I head into 2022 with the optimism that things will get better, yet also with the realism that unforeseen events will happen and set us back. The key, as always, is to live life in a way that one has no major regrets for paths not taken, and living in a way that is just and equitable for the rest of society.

Tuesday, December 21, 2021

From Reading to Writing: Next Steps for Patient Data Exchange and Interoperability

The rationale and implementation for reading data from the electronic health record (EHR) and other clinical sources is relatively simple and straightforward. Especially now enshrined into law in the US by the 21st Century Cures Rule, and standardized by the FHIR application programming interface (API), accessing data for reading by clinicians, patients, and others is here to stay.

Writing data to the EHR or other clinical information systems is a little more complicated. As in all aspects of informatics, the technology part is relatively simple, as activating the API in the reverse direction is not difficult technologically. But writing data into the EHR and other systems raises a number of issues. Earlier this year, the Office of the National Coordinator for Health IT (ONC) convened a workshop to address this topic. The workshop discussed stakeholder knowledge, current usage, potential use cases, and lessons learned on “write-back” API functionality. A report from the workshop was released in November, 2021 and provides excellent insights into the usage and challenges for such technology. Five categories of stakeholders were represented: researcher, technologist, healthcare provider, patient, and financial technologist. (The provider perspective was provided by OHSU faculty Dr. Ben Orwoll.)

The report summarized a number of possible use cases and their data sources for API write-back:
  • Data from devices, such as wearables and remote monitors
  • Questionnaires from patients or care activities
  • Results of risk scores and calculators
  • Patient input of symptoms or reported outcomes
  • Recommendations of clinician decision support
  • Annotation or amending of patient notes
  • Results or recommendations of machine learning/artificial intelligence algorithms
  • Data from transitions of care across orgniazations
  • Community sources of data, including social determinants of health

Also identified were a number of technology barriers to writing data back into the EHR:

  • Limitations of FHIR standard
  • Accuracy and completeness of the data
  • Security of data from third-party apps
  • Mapping and coding issues for data entering EHR and other systems
  • Patient-matching accuracy
  • Requirements for manual workflow and/or reconciliation
  • Obligations of organization and clinicians for data written back

In addition, the report raised a number of policy, preference, and data use concerns:

  • Data ownership and expectations for patients and clinicians
  • Compliance with HIPAA and other current laws
  • Relationship to designated record set and legal medical record
  • Regulation needed to support open APIs and their adoption
  • Requirements for future policy
The entry of data into the EHR beyond the usual documentation by clinicians and others changes the nature of the EHR, marking the true transition from electronic medical record to electronic health record.

Monday, December 20, 2021

Kudos for the Informatics Professor - 2021 Update

Although virtually all work remains remote through the end of 2021, the Informatics Professor has still been productive in his work and actively collaborating with others around the world. I am proud of my accomplishments in 2021, and this post lists some of the highlights.

Probably my biggest highlight for this year is the awarding of two new National Institutes of Health (NIH) grants. One is an exciting new project funded by the NIH Harnessing Data Science for Health Discovery and Innovation in Africa Initiative (DS-I), where the Oregon Health & Science University (OHSU) Biomedical Informatics Graduate Program will be teaming up with the University of Cape Town (UCT) to develop new Research Training opportunities. A nice overview in Nature describes the larger initiative.

In this project, OHSU will partner with UCT to develop a new graduate program in Computational Omics and Biomedical Informatics (COBIP) that will start at UCT and aim to expand to other institutions from other countries across southern Africa. I will be on the leadership team and provide my expertise in developing and leading graduate programs in biomedical informatics and data science. The program will draw on the clinical informatics courses that OHSU offers, and other OHSU informatics faculty also be involved in teaching and mentoring of research projects of students in the program. We will aim to build what the overall NIH program seeks to accomplish, namely adding capacity in biomedical informatics and data science across the African continent. With the development of computer networks and sources of health data across Africa, the time is ripe to leverage it for the health of people in Africa. As with other programs we have helped developed over the years (in places like Argentina and Thailand), our goal is to develop sustainable programs in these places so they can eventually function without the need for us. Despite some setbacks from Covid, Africa is really poised to see development of using informatics to improve the healthcare and public health systems, ultimately benefiting health of the people. And hopefully in the long run the program will attract local talent and expertise and become an independent part of the global informatics community.

The second new grant is a five-year renewal of a project, Semi-structured Information Retrieval in Clinical Text for Cohort Identification. This funding continues my work in collaborating with colleagues from the Mayo Clinic and University of Texas Houston at the intersection of information retrieval and re-use of electronic health record data.

In addition to grants funded, I also served as a senior author on  a couple of journal articles published in 2021:
In 2021, I also assumed the Presidency of the International Academy of Health Sciences Informatics and was a co-author on two papers describing some activities of the Academy:
I also published three book chapters in 2021:
  • Hersh W, Biomedical Informatics, in Kutz M (ed.), Biomedical Engineering Fundamentals, Third Edition, McGraw-Hill, 2021, 31-48.
  • Hersh W, Information Retrieval, in Shortliffe EH, Cimino J, Chiang MF (eds.), Biomedical Informatics: Computer Applications in Health Care and Biomedicine, 5th Edition, New York, Springer, 2021, 761-800.
  • Hersh W, A Passion and a Calling, in: Kulikowski, C., Mihalas, G., Yacubsohn, Y., Greenes, R., Park, H.-A. (eds.), IMIA History Book. Healthcare Computing & Communications Canada, 2021, 383–386 (described further in an earlier post to this blog).
During the year, I also participated, mostly remotely, in a number of academic conferences, some of which were international. I gave an international keynote talk, although unfortunately from the confines of my home office at not with my long-time friends in Argentina, at the Jornadas de Informática en Salud del Hospital Italiano de Buenos Aires on November 16th entitled, Artificial Intelligence in Medicine: The Need to Translate From Basic Science to Clinical Value. Both the video part of my talk and my slides and references can be viewed online.

In another international conference in Portugal, I spoke virtually on a panel at the conference, Healthcare in Post-Pandemic – Will Value for Health be the Way?, hosted by the Value for Health Collaborative in Lisbon, Portugal (virtual), June 18th. Domestically, I provided an update virtually, AMIA Update on Academic, Education and Certification Initiatives, at the CAHIIM Fall Festival: Navigating Higher Education 2021 (virtual), October 7th.

Fortunately, I was able to participate in one conference in person. After a year and a half of not attending any conference in person, I was able to make it to the AMIA Annual Symposium in San Diego, CA from October 29-November 3. This meeting marked my 36th consecutive year attending the AMIA Annual Symposium, which was known as the Symposium on Computer Applications in Medical Care (SCAMC) when I first attended it in 1986. Last year, of course, the meeting was completely virtual, but fortunately the meeting took place in San Diego, where the mild climate of southern California allowed all social events to be outdoors. At the conference I participated in a panel entitled, Career Development Issues for Women in Biomedical Informatics Within Professional Organizations, on November 1st.

I was also interviewed for a couple of podcasts. One was, Conversations from the AMIA 2021 Symposium, from The Public Health Networker podcast of the Public Health Podcast Network, in which I gave some history and perspective of AMIA and its Annual Symposium over the years.

Another podcast in which I took part was from the series called Sound Practice, hosted by the American Association for Physician Leadership. My session was At the Intersection of Technology and Healing: Trends in Medical Informatics.

Finally, I also had a chance to do some writing focused on local issues in Portland, OR and the cities challenges in grappling with economic and social issues exacerbated the pandemic. I authored a perspective for a series in the Portland Business Journal entitled, Don't Count Portland Out (apologies for it being behind a paywall).

Wednesday, November 24, 2021

A Part of Informatics History

Although I am no historian, I have always enjoyed reading history, which often provides insights into why the world is the way it is in current times. Although society in the 21st century is changing rapidly, particularly with regards to technology, we can still learn from what happened in the past, both to appreciate what we have now and to understand how we got here.

To this end, I am delighted to see publication of a new volume devoted to the history of biomedical and health informatics. Although the book provides some historical overview of the field, its main content consists of about 160 personal stories of what led many current senior leaders to end up working in this field. I am delighted that my own story is one of those in the collection and am also pleased that the book is published in an open-access manner and is freely available as a PDF on the Web site of the International Medical Informatics Association (IMIA).

The citation for the book is, International Medical Informatics and the Transformation of Healthcare, Casimir A. Kulikowski, Editor-in-Chief; George I. Mihalas, Associate Editor-in-Chief; Robert A. Greenes, Editor; Hyeoun-Ae Park, Editor; Valerio Yácubsohn, Editor. ISSN 1485-7375. Copyright, 2021 by Healthcare Computing & Communications Canada & IMIA. The cover is shown below.

History book cover
Because the book is open-source, I am taking the liberty of copying the text of my chapter into this posting verbatim (resisting temptation to edit or update):

A Passion and a Calling

My interest in biomedical and health informatics goes back to my high school days in the northern suburbs of Chicago in the early 1970s. I was introduced to computers when my school acquired a Hewlett-Packard 9830A, which was the size of a suitcase and had a built-in single-line LED display, thermal printer on top, and cassette tape storage unit. I learned how to program it in BASIC. I was also a cross-country and track runner in high school, which led to my interest in health and medicine. Running also taught me self-discipline, which helped me achieve goals later in life.

I went off to college at the University of Illinois Champaign-Urbana, where I intended to major in computer science (CS). However, I found CS to be very different in college than in high school. My first courses using punch cards and the PL/1 programming language did not excite me. I did, however, enjoy working with PLATO, a networked system with (primitive, by today’s standards) bit-mapped graphics. Two years into college, I left CS to pursue a medical career. My interest in health and preventive medicine, together with youthful rebellion, provided the foundation for my interest in evidence-based medicine.

In medical school, also at University of Illinois, I met my first informatics faculty member, Dr. Allan Levy, who nurtured my interests. Of all my education, medical school was the least enjoyable. I did not like the massive amount of rote memorization, which contributed to my later attraction to informatics. In my third year of medical school in 1983, I purchased my first computer, a Commodore 64: I hooked it up to my television as a monitor and to my phone via a 300-baud modem, and connected to Compuserve, which had a medical bulletin board called MedSIG. There I met Col. Gordon Black, who encouraged a number of us early informaticians, including long-time colleague, Rob McClure.

The reigniting of my interest in computers continued to grow as I started an internal medicine residency in 1984 at University of Illinois Hospital in Chicago. During my residency, it became apparent to me that I wanted to combine medicine and computers in my career. I came to learn about a field called “informatics,” but without Google or other search engines, there was no easy way to find more information. This led me to write letters and make phone calls to people like Ted Shortliffe, Bob Greenes, Clem McDonald, Perry Miller, and Scott Blois. I ultimately learned about National Library of Medicine (NLM)–funded informatics fellowships and chose to pursue one in the Harvard program under Bob Greenes at Brigham and Women’s Hospital in Boston. In 1987, after having lived my whole life in Illinois, I headed off with my wife to start my informatics fellowship. It was quite a change for me, with my previous daytime focus on medicine and intermittent nights-and-weekends focus on computing now flipped. In the fellowship, I could do computing almost all the time and practice medicine on the allowed one day per week. During this fellowship, it quickly became clear that informatics would become my life’s calling.

Like many working in informatics in the 1980s, I initially tried to find a research interest and niche in artificial intelligence (AI) systems of the day. One early attraction was knowledge representation, and this led to Bob involving me in his work on the Unified Medical Language System (UMLS) project that had been launched by the NLM in 1986. But the progress of the first generation of AI was sputtering by then, and almost by accident I came across a report on the topic of information retrieval (IR) authored by Bruce Croft, a computer science professor at the University of Massachusetts at Amherst. There was very little research going on in IR in medical informatics, and the main work emanated from the development of MEDLINE (although Mark Frisse had done some important work during that time in applying IR to the emerging world of hypertext). Croft’s report steered me to the most prolific researcher and author in the IR field, Gerard Salton. Many current senior leaders in IR trained as PhD students under Salton, and I was also profoundly influenced by his work. I had the chance to meet Salton when he came to give a talk at Harvard. He was intrigued by my interest in IR applied in the medical domain. I have always thought it was most unfortunate that Salton never lived to see the wide reach and impact of his ideas and work in modern search engines, as he passed away in 1995.

My clinical background dampened my enthusiasm for the relatively clunky and time-consuming AI systems of the 1980s and heightened it for IR. I was intrigued by the idea of physicians and others being able to access knowledge at the point of care. My perception of IR systems at the time was that they were limited, with systems doing just word-based searching on text or requiring complex Boolean queries over human- assigned indexing terms. My interest in IR, combined with the advancing UMLS project, led me to pursue a line of research that combined concept-based automated indexing to enhance retrieval that applied the statistical approaches developed by Salton and others. This led to me to develop and implement a system called SAPHIRE, which was the focus of my early research.

During my fellowship, I was also briefly involved with a project that would later become a highly successful commercial product. Bob had been visited by Burton Rose, a nephrologist at Brigham and Women’s who was enamored with a new tool that shipped with the Mac called Hypercard. He believed that small chunks of information on each “card” in a Hypercard “stack” could be highly useful to physicians. But as the quantity of information grew, he needed a search capability that was better than that which shipped with Hypercard. I programmed the search capability for the first version of what would later be called UpToDate, which ultimately achieved great commercial success. At the end of my fellowship, I handed this project off to another fellow, Joseph Rush, who continued to work on UpToDate for many years.

As my fellowship was ending, I knew that I wanted to pursue a career in academic medical informatics. One person I came to know was Bob Beck, who at the time was heading the informatics program at Dartmouth College. By the fall of my last year of fellowship, Bob had moved to Oregon Health Sciences University (OHSU) to start a new program there funded by the NLM IAIMS program, bringing with him another faculty member, Kent Spackman.

While I had some other job possibilities, my wife and I, now with a one-year-old daughter, packed up and moved to Portland in July 1990. My first activity in the new job was to submit an NIH R29 proposal that I had been working on in the latter months of my fellowship. Also called a FIRST Award, this type of grant was a common pathway for new researchers to launch their careers. Several months later, I was notified that it would be funded, which jump-started my academic career.

In 1990, Oregon voters passed a property tax limitation measure which ultimately led to Bob Beck losing resources and leaving in 1992. This left behind a very junior faculty, led by Kent, but as Kent wasn’t interested in building a program, he devolved the leadership to me. By 1996, our young academic group was starting to achieve sustained success. This led the Dean of the OHSU School of Medicine at the time, Joseph Bloom, to encourage our unit to become more visible on campus. The usual way of doing this at OHSU was establishing a so-called free-standing division, which was the path to establishing a department. This also provided me a seat at the table of clinical department chairs, which I maintain to this day.

I was interested in teaching from the beginning of my faculty career, and when Kent asked me to organize the introductory informatics course—something I still teach to this day—it led to many others, like the one I teach in the 10x10 program. When I started my fellowship, and then my faculty position at OHSU, I never realized how much of a passion teaching would become for me. I always enjoyed teaching because it gave me a chance to learn as well as develop a coherent organization for various topics. My path down the road to my current leadership in education was also greatly influenced by those I taught. In particular, while I assumed that our educational program would be small and aim to produce researchers like myself, there were a number of students who were interested in more varied careers, such as the small but growing number of professional positions in healthcare settings or industry. This resulted in our new Master of Science program taking on a more practical orientation. But that was fine, as the research of many of our faculty, such as Paul Gorman and Joan Ash, was motivated by real-world concerns in the application of informatics.

Even with my growing interest in education and my leadership responsibilities in our emerging program, I still maintained my interest in research. While it became more difficult to develop new IR systems when giants like Google and PubMed emerged, my interest in evaluating how well people used IR systems for health and medical reasons became the main focus of my research. In 1996, I published the first edition of my book, Information Retrieval: A Health Care Perspective.

By 1999, as I was contemplating ways to expand our educational program, a number of people had asked if we planned to offer our courses via distance learning. I decided to offer my now-mature introductory course in this manner, which was quite successful. There was an untapped market for distance learning in informatics, and the success of my initial course led me to convince the faculty to add this format to the program. This foray into distance learning distracted us from another goal we had in the late 1990s, which was to establish a PhD program. We finally accomplished this when our NLM training grant was renewed in 2002. At this point I became PI of the training grant.

Another pivotal career event for me came when Charlie Safran was President of AMIA (back in the days when the AMIA President was an elected position). He was convinced that the US needed more professionals, especially physicians and nurses, trained in informatics. Charlie believed the US needed at least one physician and one nurse trained in informatics in each of the nearly 6000 hospitals in the US. Also at this time, AMIA was looking to develop some sort of introductory course in biomedical informatics. However, the prices quoted to them by vendors were beyond their means. As I already had my introductory course from our graduate program, I proposed to AMIA that we repackage my online course. I came up with a name, 10x10 (pronounced “ten by ten”), based on Charlie’s one physician and nurse in 5000+ hospitals, and set a goal for doing so by 2010. Because the course already existed, we were able to put in place a Memorandum of Understanding between OHSU and AMIA and launch the first offering of the course in just a few months. The next President of AMIA, Don Detmer, called 10x10 one of the association’s most successful programs ever.

My interest in education and training spurred my interest in workforce development for the field. In 2006, I was invited to organize the surprise retirement event for long- time academic leader, originally from Germany and later from Victoria, Canada, Jochen Moehr. I gave a talk entitled Who are the Informaticians, What We Know and Should Know, which I later published in JAMIA. This interest was fortuitous, since the US economy would soon enter free fall, leading to the American Recovery and Reinvestment Act (ARRA), the economic stimulus bill that included the Health Information Technology for Economic and Clinical Health (HITECH) Act. While HITECH was best known for its $30 billion “meaningful use” program of incentives for EHR adoption, it also included $118 million for workforce development, motivated in part by some research I published showing a need for more informatics professionals. I played a large role in the grants that were competitively awarded by the HITECH Workforce Development Program, including being funded as the National Coordination and Dissemination Center for the health IT curriculum that was funded through the program.

During and after HITECH, I continued to provide leadership for informatics education and its relationship to other careers in the field. I was also a leader in the new clinical informatics physician subspecialty, being appointed by AMIA to direct the Clinical Informatics Board Review Course (CIBRC), which was offered in time for the first board examination in 2013. The next year I laid the groundwork at OHSU to establish one of the first four Accreditation Council for Graduate Medical Education (ACGME)–accredited fellowships for the new subspecialty, which launched in 2015. Around this time, I also had the opportunity to develop informatics education for non- informaticians, namely medical students. Along with colleagues at OHSU, we began to implement informatics education in the MD curriculum (just in time for my younger daughter to become a medical student!).

I have now been at OHSU for nearly 30 years, where I have had the opportunity to continue my research and teaching, and lead my department. Another critical activity of mine now is to mentor young faculty, who one day will sustain and lead our program.

Monday, November 15, 2021

A New Systematic Review Highlights the Current State and Limitations of Clinical AI Use and Efficacy

When I teach about the features of search engines like PubMed, I often quip that if you use the limit function to narrow your search to randomized controlled trials (RCTs), which are the best evidence for medical and health interventions, and you still have many retrievals, there is probably some enterprising researcher who has done a systematic review on the topic. Some actually worry that we have too many systematic reviews these days, not always of the greatest quality.(1) But such reviews, especially when done well, can not only be important catalogs of research on a given topic but also provide an overview of the breadth and quality of studies done.

Sure enough, we have started to see systematic reviews on artificial intelligence (AI) and machine learning (ML) applications. A new systematic review covers all of the RCTs of interventions of AI applications.(2) I hope the authors will keep the review up to date, as one limitation of systematic reviews published in journals is that they become out of date quickly, especially in rapidly moving areas such as AI.

As we know from evidence-based medicine (EBM), the best evidence for the efficacy of interventions (treatment or prevention) comes from RCTs. Ideally, these trials are well-conducted, generalizable, and well-reported. EBM defines four categories of questions that clinicians ask: intervention, diagnosis, harm, and prognosis. As such, there are other clinical questions that can be answered about AI beyond those about interventions. For example, can AI methods improve the ability to diagnose disease? Can AI identify harms from environment, medical care, etc.? And finally, can AI inform the prognosis of health and disease? Ultimately, however, AI interventions must be demonstrated experimentally to benefit patients, clinicians, and populations. There are of course some instances when RCTs are infeasible so observational studies may be justified.

In this context, we can review a recently published systematic review of interventions using AI clinical prediction tools of Zhou et al.(2) This systematic review categorized AI methods into three groups: traditional statistical (TS), mostly regression; machine learning (ML), all ML but deep learning; and deep learning (DL), i.e., applications using multi-layered "deep" neural networks. TL and MS tools were found to be used for three functions: assistive treatment decisions, assistive diagnosis, and risk stratification, whereas DL tools were only assessed for assistive diagnosis.

Typical as happens in most systematic reviews, the authors found over 26,000 papers published and retrieved by their broad MEDLINE search, but of those, there were only 65 RCTs identified. Once identified, the 65 trials were reviewed for a number of characteristics. One important characteristic was whether or not studies demonstrated a benefit for AI, i.e., had a positive result. Of course, counting numbers of positive vs. negative results is not necessarily an indicator of the value or generalizability of a particular method of AI or any other clinical intervention for that matter. Nonetheless, the authors did find that 61.5% of the RCTs had positive results and 38.5% negative results.

As AI can be used for many conditions and functions in medicine, it is important to get a sense of what was studied and what tools were used. The authors found use for AI in a variety of disease categories: acute disease (29%), non-cancer chronic disease (28%), cancer (17%), primary care (14%), and other conditions (12%). Of the predictive tool function used, use was most often for assistive treatment decisions (54%), followed by assistive diagnosis (25%) and risk stratification (19%). There were the most studies used for TS (57%), followed by ML (26%) and DL (17%). These differences may reflect the more recent development and use of ML and especially DL. The rates of positive studies for the tool types were highest for DL (82%), followed by ML (71%) and TS (51%), although it should be noted that the rate of positive results was also inversely related to the number of trials for each tool type.

A table in the paper shows that there were differences by tool categories. TS tools were mostly likely to be used with clinical quantitative data (97%), applied in acute disease (43%) and primary care (24%), and used for assistive treatment decisions (60%) followed by risk stratification (30%). ML tools were also most likely to be used by clinical quantitative data (94%), applied in chronic disease (77%), and used for assistive treatment decisions (77%). DL tools were most likely to be used with imaging data (91%), applied in cancer (91%), and used exclusively for assistive diagnosis (100%). In particular, the DL studies almost exclusively evaluated assistance of gastrointestinal endoscopy, with all nine such RCTs showing positive results and the two trials of other applications and diseases having negative results. Also of note, only two of the 65 RCTs made use of natural language data for input, one ML and one DL.

Systematic reviews typically appraise included studies for risk of bias, or in other words, the quality of their methods to produce an unbiased result. This provides confidence that the research methods were robust and well-reported so readers can have confidence that the results obtain are true. Unfortunately, there were a number of concerns that led to 48 (74%) of the trials being classified as having high or indeterminate risk of bias. This was due to a number of factors:

  • One-third of the trials carried out no sample size estimation to determine what would be the number of subjects needed to achieve a statistically significant benefit
  • Three-fourths of the trials were open-label, so had no masking of the AI system from its users
  • Three-fourths did not reference the CONSORT statement, a 37-item checklist widely used for reporting the details of RCTs and recently extended for AI trials
  • Three-fifths did not apply an intent-to-treat analysis, which evaluates subjects in the study groups into which they were originally assigned
  • Three-fourths did not provide reference to a study protocol for the trial

The rate of outcomes of studies for low risk of bias trials was somewhat comparable to the overall rates, with positive outcomes in 63% of TS, 25% of ML, and 80% of DL trials.

What can be concluded from this systematic review? We certainly know from the vast amount of other literature that a large number of predictive models have been built using AI techniques and shown to function well for a wide variety of clinical conditions and situations. We probably cannot do an RCT of every last application of AI. But at this point in time, the number and variety of RCTs assessing benefit for interventions of AI is modest and uneven. While a number of positive results have been demonstrated, the studies published have not been dispersed across all of the possible clinical applications of AI, and three-fourths of the reports of the trials show indeterminate or high risk of bias. DL methods in particular must be assessed in the myriad of areas in which data sets have been developed and models trained.

There are some problems with the systematic review itself that mar the complete understanding of the work. Table 2 of DL interventions has data missing in its leftmost column that connects the data in the column to its original reference. This table also does not include a recent paper by Yao et al.,(3) which was likely published after the review was completed. It is also difficult to use the data in Supplementary Table 4 of ML interventions, which is provided in a PDF file that is difficult to read or browse. In addition, while the paper references a high-profile study by Wijnberge et al.,(4) it is not listed in ML table. This study may well be classified as TS, but this demonstrates another limitation of the systematic review, which is that there is no data or table that details TS interventions. The authors were kind enough to provide Excel files of the DL and ML tables, but they really should be part of the online materials for the systematic review. I do hope they or someone will keep the review up to date.

As it stands, this systematic review does give us a big-picture view of the clinical use and benefit for AI at this point in time, which is modest, disproportionate, and based on studies using suboptimal methods. We can conclude for now that AI predictive tools show great promise in improving clinical decisions for diagnosis, treatment, and risk stratification but comprehensive evidence for the benefit is lacking.

This systematic review also highlights a point I have written about in this blog before, which is that AI interventions need translation from basic science to clinical value. In particular, we need clinically-driven applications of AI that are assessed in robust clinical trials. There of course must also be attention to patient safety and to clinician workflow. In general, we need robust AI and RCT methods that are replicable and generalizable, and of course we must conduct implementation and trials from a health equity standpoint.

References

1.     Ioannidis JPA. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485–514.
2.     Zhou Q, Chen Z-H, Cao Y-H, Peng S. Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review. NPJ Digit Med. 2021 Oct 28;4(1):154.
3.     Yao X, Rushlow DR, Inselman JW, McCoy RG, Thacher TD, Behnken EM, et al. Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial. Nat Med. 2021 May;27(5):815–9.
4.     Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, et al. Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial. JAMA. 2020 Mar 17;323(11):1052–60.

Saturday, November 13, 2021

This Year's Eco-Event: An E-Bike

If last year's eco-event for me was installing solar panels on the roof of my house, this year's event was the purchase of an electric bike (e-bike). I have to admit I am not completely virtuous when it comes to my carbon footprint. While my time spent in airplanes has reduced dramatically during the pandemic, it is likely to increase, although probably not to pre-pandemic levels, as we all return to travel.

But one thing I do hope as there is eventual return to working in the office is to commute in all but the worst weather by e-bike. One of the challenges for bicycle commuting for me is that although I only live 4-5 miles (depending on the route) away from my office, I must ascend over 500 feet vertically to get between my home and office. (Those in Portland know the Fairmount Loop just below the Council Crest hill that sits above Oregon Health & Science University.)

Likewise, a similar hill separates my home from the short distance to downtown and other parts of Portland that would be wonderful to leisurely ride to on a bicycle. I sometimes do this ride on a regular bicycle, but it is a real workout. There are other times when I would just rather get somewhere without breaking a major sweat.

After doing much online research, I decided to test drive, and eventually purchase, an Aventon Pace 500 Step-Through e-bike. There are many e-bike options, and the market for these products is not yet fully mature. But I have enjoyed my e-bike, and I have to say it is actually quite fun to ride. There is nothing like a little electric assist when riding a bike, especially in the hills of the west side of Portland.

My e-bike
Another aspect of the Pace 500 I enjoy is it being a Class 3 bike, which allows it to have a throttle. While I use the throttle sparingly, I do find it of great value when I need to accelerate quickly, such as at a traffic light turning green or starting up a hill. The Class 3 status also allows the motor to speed the bike to up to 28 mph in its highest (5) pedal assist level, although I try to ride at no higher than pedal assist 2, which gives me plenty of power even up hills.

Tuesday, October 26, 2021

Musings on a Tool Every Academic Should Use

I often muse that there are few computer applications that truly save me time. For all the fun and productive things that computers enable me to do, they are just as often a time sink rather than a time saver, especially when hardware or software go wrong. The transition from being able to talk to a person on the phone to the use of chatbots and other ways to keep people with problems away from costly human support has added even more time, especially during the pandemic.

There is, however, one notable exception, a computer application that saves me a great deal of time. This is bibliographic management software. Any academic who writes a great deal, especially those who publish in journals and other venues across different disciplines, knows the time and effort required to maintain and format references.
 
As with all computer applications, one must choose their bibliographic management software package wisely. I was an early user of EndNote, back to Version 1 when it was first released in the 1990s. It had served me well over the years, but one challenge was that as the formats of bibliographic records changed, I was not able to take advantage of its automatic capture of metadata. I was also not able to easily merge my EndNote database with those of others, again due to the formatting issues. By the end of last year, my database of papers that I have cited once or more in research writing or teaching, had grown to over 12,000 entries, yet my effort to use the product put me in a silo.
 
This year I decided to make a decision to start over. My first decision was whether to continue with EndNote or move to a different package. One package that many of my colleagues seemed to be adopting was Zotero. This package has the advantage of being open-source, with a large developer community. I decided to make the switch.
 
The transition has not been simple, as my existing EndNote library had too many irregularities for me to simply import it into Zotero. However, I have been able to build up my new Zotero database relatively quickly due to its automated capture of metadata. (In fairness to EndNote, they have this feature as well.) The automated capture of metadata is not perfect, mainly because many Web sites and pages do not adhere to standards. But many key sources, such as PubMed, most journals, Amazon (for books), and others make entry into the database quick. One notable feature for someone concerned with the big picture of science is Zotero's ability to flag that a scientific paper has been retracted.
 
Zotero is not perfect, and one feature I hope is added soon is the ability to easily update a record, such as when a journal paper goes from "online ahead of print" to actually being "in print," i.e., having a volume, issue, and page numbers (even if many journals are not physically printed these days). The metadata of journal articles does change over time, and the ability to automate the capture of its changing as easily as its initial capture would be a great feature.
 
Nonetheless, bibliographic management software is a vitally important tool for those who write scientific papers, especially in inter-disciplinary fields like informatics. And the decision on which package to use is important, as changing from one to another can be time-consuming. But it is certainly an application whose proper use can save time overall.

Wednesday, October 6, 2021

Certification for the Rest of Informatics

After several years of planning, professional certification is coming to the rest of the informatics field, i.e., moving beyond just board certification for eligible physicians. While certification is somewhat easier to apply in the context of the physician board model, the American Medical Informatics Association (AMIA) has now rolled out the AMIA Health Informatics Certification (AHIC, formerly Advanced Health Informatics Certification). Those who are certified will be designated as ACHIP, the AMIA Certified Health Informatics Professional. A section of the AMIA Web site provides detailed on the certification, eligibility for it, applying for and taking the exam, and recertification.

While AHIC is open to all who have a master's or doctoral degree in health informatics or a related discipline, the certification process is not conferred upon initial completion of one's education. Rather, individuals also need to have completed qualifying work experience to be eligible for certification. This is different from some fields, such as medicine, including the clinical informatics subspecialty, where one takes the board certification exam shortly after completing formal training. There are a number of healthcare disciplines that require significant work experience for certification, such as some of the advanced certifications offered by the American Nurses Credentialing Center.

The qualifications for AHIC are listed in a table on the AHIC Web site. There are two tracks of eligibility. Track 1 is for those who have a graduate degree in a health informatics-related area, e.g., health informatics, biomedical informatics, nursing informatics, public health informatics, translational bioinformatics, etc. Track 2 is for those who have a graduate degree in a related field, e.g., health professions such as nursing, pharmacy, and medicine, and other fields such as computer science and public health. The work time required for those in Track 1 is 50-100% work time over the last four of six years or 20-49% time over the last six of eight years. The work time required for those in Track 2 is 50-100% work time over the last six of eight years or 20-49% time over the last eight of 10 years.

The certification process is being developed and managed by the Health Informatics Certification Commission (HICC), a 14-member commission that is part of AMIA yet has considerable autonomy from AMIA, especially with regards to AMIA's educational programs. The HICC is responsible for eligibility, examination development, and recertification requirements for AHIC.

The first offering of the certification exam is taking place this fall. The outline of exam topics follows the health informatics workforce analysis commissioned by AMIA (Gadd, C.S., Steen, E.B., Caro, C.M., Greenberg, S., Williamson, J.J., Fridsma, D.B., 2020. Domains, tasks, and knowledge for health informatics practice: results of a practice analysis. J Am Med Inform Assoc 27, 845–852. https://doi.org/10.1093/jamia/ocaa018), just as the clinical informatics subspecialty exams now uses the complementary clinical informatics subspecialty workforce analysis for its exam blueprint.

The two questions someone enrolled in or contemplating seeking a degree in informatics will likely ask are: (1) Is this certification process for me? and (2) Will it benefit my career? Since this form of certification is new for professionals who work in informatics, the benefits at this time are unknown. The main drivers of the uptake will be employers who make hiring decisions that are influenced by job candidates having the certification. Similar to the clinical informatics subspecialty, we will probably see a gradual uptake of the AHIC over time. It may never be an absolute requirement for a job but it will be an important "feather in one's cap" when competing with others for a given position.

Tuesday, August 24, 2021

Scientific Rankings for the Informatics Professor

While I agree with those who argue that scientific rankings, especially based on bibliographic citation indicators, are limited in their measurement of a scientist's impact, I must admit a certain fascination with them. Perhaps that stems from my interest in dissemination and retrieval of scientific information generally. And perhaps also, because I enjoy writing and tend to measure well by these metrics.

I do show up in most rankings of my primary and related scientific fields. As I consider my primary scientific discipline to be biomedical informatics, I can report that I rank 57th on one list of researchers in the field. On another more focused list of those who work in medical informatics, I rank 14th, although my ranking falls to 33rd when the list is less focused (details below - also see [1,2]). I am also on a global list of the top 1000 computer science and electronics researchers, where I rank 932nd globally and 569th among Americans. On a more focused computer science list for the information retrieval field, I rank 23rd.

A description of how these rankings are calculated gives some perspective into my positions on them. All of these rankings make use of the well-known citation measure, the h-index, although one uses additional factors. The h-index is a measure of the number of one's publications that have been cited by at least that same number of publications. So for example, if one has 15 papers that have been cited 15 or more times, their h-index is 15. There are two main public sources of h-index values that are most commonly used, which give different results due to the way they are calculated. The two are Google Scholar and Scopus, the latter an arm of the scientific publishing conglomerate, Elsevier. The Google Scholar h-index is usually higher than the Scopus h-index due to the former including a wide variety of academic products, such as conference proceedings, books, non-peer-reviewed reports, and other publications on the Internet, whereas the latter is limited to journal publications. As the Google Scholar value is also generated automatically, it is more likely to contain erroneously included papers, especially when author names are ambiguous. My current Google Scholar and Scopus h-index values are, as of this writing, 75 and 46 respectively.

Obviously the h-index is related to the duration of one's career, and as citation patterns vary in different fields, one must compare the h-index of different individuals with caution. With those caveats, we can explore further my own ranks. The list of biomedical informatics researchers is maintained by Allison McCoy of Vanderbilt University. One concern about this list is that it contains a number of researchers who, although published in the biomedical informatics literature, do not primarily work in the biomedical informatics field. This list is generated from software developed by Jimmy Lin of University of Waterloo, who maintains the list of information retrieval researchers (and several other fields within computer science). The list of top worldwide computer science researchers is maintained by a Web site devoted to computer science research, Guide2Research.

An additional ranking in which I appear is one compiled by John Ioannidis of Stanford University and colleagues [1,2]. This analysis includes the top 100,000 scientists across all fields, with additional enrichment from those in the top 2% of their field but not in the top 100,000. Unlike the other sources, this analysis is fixed, with data through 2019 and taking more factors into account than just the h-index. A composite C-score is made up of six factors measured from citations through 2019 and excludes self-citations:

  • h19 (ns)    h-index as of end-2019
  • hm19 (ns)    hm-index as of end-2019
  • ncs (ns)    total citations to single authored papers
  • ncsf (ns)    total citations to single+first authored papers
  • npsfl (ns)    number of single+first+last authored papers
  • ncsfl (ns)    total citations to single+first+last authored papers

The rationale for this more complex measure is based on observations that (a) in some fields, many papers have vast numbers of authors, (b) these large numbers of authors give great weight to measures based purely on citations, (c) many Nobel laureates do not rate highly in simple citation measures such as h-index, (d) many of those who rank highly in simple citation measures have few or no first-authored or last-authored papers, and (e) Nobel laureates rank higher when more complex measures such as a C-score are employed.

In this cast of more than a hundred thousand, my C-score of 3.938 gives me an overall rank of 22,034, which is based on 241 papers published and 6109 citations to them through 2019. As noted above, I rank 15th among those whose primary field is medical informatics. There are also others for whom medical informatics is listed as their secondary field, and when combined with those for whom it is primary, my ranking is 33rd. There is a separate ranking for those whose primary field is bioinformatics. 

I can also extract out all researchers in the ranking from my institution and its affiliates (Oregon Health & Science University, OHSU School of Medicine, Oregon National Primate Research Center, and Portland VA Medical Center) and note that I rank 49th out of 256 included in this list. (I am also pleased to note that 10 people from my department make it on to the overall OHSU list, including Roger Chou, Heidi Nelson, Mark Helfand, Joan Ash, Cynthia Morris, Paul Gorman, Rochelle Fu, Linda Humphrey, and Aaron Cohen.)

One interesting aspect of the Ioannidis et al. analysis is that I rank better using the composite score than just by my h-index. Based solely on the h-index, I would rank only 131st for OHSU and 37th in the primary medical informatics list. My C-score is improved by my relatively higher number of first-author and single-author papers, and citations to them. I also must have fewer co-authors on my papers than my colleagues at OHSU and in informatics, as I do better with the hm-index, which adjusts for the number of authors on a paper. At OHSU in particular, where I rank 49th overall and 131st by h-index, I rank 51st in hm-index, 37th in citations to single-authored papers, and 41st in citations to single- and first-authored papers. My 104 single- and first-authored papers rank me 22nd at OHSU. My data for the Ioannidis et al. analysis is available in a spreadsheet (Enjoy!).

On a final note, I am pleased to report that citation indices are a family affair for me. My daughter Alyssa Hersh, MD, MPH is currently a resident in Obstetrics & Gynecology at OHSU. She is also a rising researcher, and as of this writing has a Google Scholar h-index of 6 and a Scopus h-index of 4. I have no doubt she will surpass my current citation metrics long before she reaches my current age!

References

[1] Ioannidis, J.P.A., Klavans, R., Boyack, K.W., 2016. Multiple Citation Indicators and Their Composite across Scientific Disciplines. PLoS Biol 14, e1002501. https://doi.org/10.1371/journal.pbio.1002501.

[2] Ioannidis, J.P.A., Boyack, K.W., Baas, J., 2020. Updated science-wide author databases of standardized citation indicators. PLoS Biol 18, e3000918. https://doi.org/10.1371/journal.pbio.3000918.