Annual Reflections at the End of 2021

As is the tradition with this blog, I end each year with a reflective look at the year past and what the future may hold. The year 2021 is not ending quite like I anticipated. At the beginning of the year, there were stirrings of optimism. The new COVID-19 vaccines had been released, and a new approach to political leadership had just been elected in Washington, DC. On January 2nd, I received my first dose of the Pfizer vaccine, followed by a second one three weeks later.

By the spring, the pandemic was seeming to wane. In late June, Oregon lifted all pandemic restrictions, and life seemed to be returning to relative normal. At work, my department began making plans for returning to the office, aiming to slowly transition over the summer and then return to a new state of normal when the regular academic year started in the fall.

 

Alas, it was not to be. By August, the Delta wave was surging and plans for returning to the office were postponed. Things became more optimistic into the fall, with my getting my booster vaccine in early October, but then, just when it seemed that the Delta wave was on the decline, the new Omicron variant emerged.

Nonetheless, there is some reason for optimism in the long run for the pandemic. The vaccines are having an impact, even if mostly among the vaccinated. Clearly the vaccines are not completely protective against infections, but they do appear to blunt the worst complications of the disease. In addition, there are new COVID-19 treatments becoming available in pill form, such as Paxlovid, which may provide a means to further prevent the worst impacts of infection. With my three doses of Pfizer vaccine on board, I feel relatively well protected, and feel confident that if I were to have a breakthrough infection, the course would be mild. (Though I am not taking any excessive chances.)

Although I look forward to getting back to working in person, I have managed to maintain my own productivity by working mostly from my home office. One thing that the pandemic has taught the informatics field, and certainly the work of my own department, is that we can pretty much function virtually. I still enjoy interacting in person with colleagues, but clearly most of the research and educational work we do can be carried out in a virtual manner.

 

I am sad that this pandemic has been so politicized. Every happening, often having nothing to do with the action of politicians, seems to require being viewed through a political lens. While there is, as with all science, some room for debate on translating scientific findings into public policy, it is unfortunate that many, especially those on the right, have chosen to frame everything in that context and make decisions that defy sound public health practice. I don't enjoy wearing masks or avoiding large gatherings more than anyone, but I recognize these as a small price to pay to protect public health. And it is sad to me that new vaccine technology, one of the greatest public health advances in human history, has also become a political Rorschach test.

The ability to function virtually would not be possible without the maturing of computing and network technology. It is still far from perfect, but we almost take for granted now that we can have a synchronous video call with almost any location on the planet. Social media has played a valuable role too, especially in my case, Facebook. While I am no big fan of Facebook's business model, and I would happily pay a modest subscription fee to get the benefits of it that I like, it has been invaluable in keeping up personally and professionally with friends and colleagues around the world.

In terms of the informatics field, I am optimistic that we will continue the contributions we make to health and larger society. While the pandemic has caused a detour in our priorities, we are still moving forward in core advances, particularly turning the potential of machine learning and artificial intelligence into reality and advancing standards to make data more interoperable. The pandemic has exposed the limits of our current health information systems and processes, and even if too late to provide optimal value for this pandemic, will hopefully put us in a position to benefit the next public health crises.

This blog itself continues to be successful. This is the 361st post, and it is closing in on 768,000 page views over its nearly 13 years. This blog has provided me an opportunity to provide commentary on many topics, mainly professionally but also personally, and is easily accessible from the Internet. My frequency of posting has decreased some over the years, but I still appreciate the opportunity to weigh in on important topics and not feel the need to post at any specific level of frequency.

I head into 2022 with the optimism that things will get better, yet also with the realism that unforeseen events will happen and set us back. The key, as always, is to live life in a way that one has no major regrets for paths not taken, and living in a way that is just and equitable for the rest of society.

From Reading to Writing: Next Steps for Patient Data Exchange and Interoperability

The rationale and implementation for reading data from the electronic health record (EHR) and other clinical sources is relatively simple and straightforward. Especially now enshrined into law in the US by the 21st Century Cures Rule, and standardized by the FHIR application programming interface (API), accessing data for reading by clinicians, patients, and others is here to stay.

Writing data to the EHR or other clinical information systems is a little more complicated. As in all aspects of informatics, the technology part is relatively simple, as activating the API in the reverse direction is not difficult technologically. But writing data into the EHR and other systems raises a number of issues. Earlier this year, the Office of the National Coordinator for Health IT (ONC) convened a workshop to address this topic. The workshop discussed stakeholder knowledge, current usage, potential use cases, and lessons learned on “write-back” API functionality. A report from the workshop was released in November, 2021 and provides excellent insights into the usage and challenges for such technology. Five categories of stakeholders were represented: researcher, technologist, healthcare provider, patient, and financial technologist. (The provider perspective was provided by OHSU faculty Dr. Ben Orwoll.)

The report summarized a number of possible use cases and their data sources for API write-back:

• Data from devices, such as wearables and remote monitors
• Questionnaires from patients or care activities
• Results of risk scores and calculators
• Patient input of symptoms or reported outcomes
• Recommendations of clinician decision support
• Annotation or amending of patient notes
• Results or recommendations of machine learning/artificial intelligence algorithms
• Data from transitions of care across orgniazations
• Community sources of data, including social determinants of health

Also identified were a number of technology barriers to writing data back into the EHR:

• Limitations of FHIR standard
• Accuracy and completeness of the data
• Security of data from third-party apps
• Mapping and coding issues for data entering EHR and other systems
• Patient-matching accuracy
• Requirements for manual workflow and/or reconciliation
• Obligations of organization and clinicians for data written back

In addition, the report raised a number of policy, preference, and data use concerns:

• Data ownership and expectations for patients and clinicians
• Compliance with HIPAA and other current laws
  
• Relationship to designated record set and legal medical record
• Regulation needed to support open APIs and their adoption
• Requirements for future policy

The entry of data into the EHR beyond the usual documentation by clinicians and others changes the nature of the EHR, marking the true transition from electronic medical record to electronic health record.

Kudos for the Informatics Professor - 2021 Update

Although virtually all work remains remote through the end of 2021, the Informatics Professor has still been productive in his work and actively collaborating with others around the world. I am proud of my accomplishments in 2021, and this post lists some of the highlights.

Probably my biggest highlight for this year is the awarding of two new National Institutes of Health (NIH) grants. One is an exciting new project funded by the NIH Harnessing Data Science for Health Discovery and Innovation in Africa Initiative (DS-I), where the Oregon Health & Science University (OHSU) Biomedical Informatics Graduate Program will be teaming up with the University of Cape Town (UCT) to develop new Research Training opportunities. A nice overview in Nature describes the larger initiative.

In this project, OHSU will partner with UCT to develop a new graduate program in Computational Omics and Biomedical Informatics (COBIP) that will start at UCT and aim to expand to other institutions from other countries across southern Africa. I will be on the leadership team and provide my expertise in developing and leading graduate programs in biomedical informatics and data science. The program will draw on the clinical informatics courses that OHSU offers, and other OHSU informatics faculty also be involved in teaching and mentoring of research projects of students in the program. We will aim to build what the overall NIH program seeks to accomplish, namely adding capacity in biomedical informatics and data science across the African continent. With the development of computer networks and sources of health data across Africa, the time is ripe to leverage it for the health of people in Africa. As with other programs we have helped developed over the years (in places like Argentina and Thailand), our goal is to develop sustainable programs in these places so they can eventually function without the need for us. Despite some setbacks from Covid, Africa is really poised to see development of using informatics to improve the healthcare and public health systems, ultimately benefiting health of the people. And hopefully in the long run the program will attract local talent and expertise and become an independent part of the global informatics community.

The second new grant is a five-year renewal of a project, Semi-structured Information Retrieval in Clinical Text for Cohort Identification. This funding continues my work in collaborating with colleagues from the Mayo Clinic and University of Texas Houston at the intersection of information retrieval and re-use of electronic health record data.

In addition to grants funded, I also served as a senior author on  a couple of journal articles published in 2021:

• Chen J, Hersh W, A comparative analysis of system features used in the TREC-COVID information retrieval challenge, Journal of Biomedical Informatics, 2021, 117:103745.
• Roberts K, Alam T, Bedrick S, Demner-Fushman S, Lo S, Soboroff I, Voorhees E, Wang LL, Hersh WR, Searching for answers in a pandemic: an overview of TREC-COVID, Journal of Biomedical Informatics, 2021, 121:103865.

In 2021, I also assumed the Presidency of the International Academy of Health Sciences Informatics and was a co-author on two papers describing some activities of the Academy:

• Koch S, Hersh WR, Bellazzi R, Leong TY, Yedaly M, Al-Shorbaji N. Digital Health during COVID-19: Informatics Dialogue with the World Health Organization. Yearbook of Medical Informatics. 2021, 30(1):13-16.
• Haux R, Ball MJ, Hersh WR, Huesing E, Kimura M, Koch S, Martin-Sanchez F, Otero P. The International Academy of Health Sciences Informatics (IAHSI): 2020 Report. Yearbook of Medical Informatics. 2021, 30(1):8-12.

I also published three book chapters in 2021:

• Hersh W, Biomedical Informatics, in Kutz M (ed.), Biomedical Engineering Fundamentals, Third Edition, McGraw-Hill, 2021, 31-48.
• Hersh W, Information Retrieval, in Shortliffe EH, Cimino J, Chiang MF (eds.), Biomedical Informatics: Computer Applications in Health Care and Biomedicine, 5th Edition, New York, Springer, 2021, 761-800.
• Hersh W, A Passion and a Calling, in: Kulikowski, C., Mihalas, G., Yacubsohn, Y., Greenes, R., Park, H.-A. (eds.), IMIA History Book. Healthcare Computing & Communications Canada, 2021, 383–386 (described further in an earlier post to this blog).

During the year, I also participated, mostly remotely, in a number of academic conferences, some of which were international. I gave an international keynote talk, although unfortunately from the confines of my home office at not with my long-time friends in Argentina, at the Jornadas de Informática en Salud del Hospital Italiano de Buenos Aires on November 16th entitled, Artificial Intelligence in Medicine: The Need to Translate From Basic Science to Clinical Value. Both the video part of my talk and my slides and references can be viewed online.

In another international conference in Portugal, I spoke virtually on a panel at the conference, Healthcare in Post-Pandemic – Will Value for Health be the Way?, hosted by the Value for Health Collaborative in Lisbon, Portugal (virtual), June 18th. Domestically, I provided an update virtually, AMIA Update on Academic, Education and Certification Initiatives, at the CAHIIM Fall Festival: Navigating Higher Education 2021 (virtual), October 7th.

Fortunately, I was able to participate in one conference in person. After a year and a half of not attending any conference in person, I was able to make it to the AMIA Annual Symposium in San Diego, CA from October 29-November 3. This meeting marked my 36th consecutive year attending the AMIA Annual Symposium, which was known as the Symposium on Computer Applications in Medical Care (SCAMC) when I first attended it in 1986. Last year, of course, the meeting was completely virtual, but fortunately the meeting took place in San Diego, where the mild climate of southern California allowed all social events to be outdoors. At the conference I participated in a panel entitled, Career Development Issues for Women in Biomedical Informatics Within Professional Organizations, on November 1st.

I was also interviewed for a couple of podcasts. One was, Conversations from the AMIA 2021 Symposium, from The Public Health Networker podcast of the Public Health Podcast Network, in which I gave some history and perspective of AMIA and its Annual Symposium over the years.

Another podcast in which I took part was from the series called Sound Practice, hosted by the American Association for Physician Leadership. My session was At the Intersection of Technology and Healing: Trends in Medical Informatics.

Finally, I also had a chance to do some writing focused on local issues in Portland, OR and the cities challenges in grappling with economic and social issues exacerbated the pandemic. I authored a perspective for a series in the Portland Business Journal entitled, Don't Count Portland Out (apologies for it being behind a paywall).

A Part of Informatics History

Although I am no historian, I have always enjoyed reading history, which often provides insights into why the world is the way it is in current times. Although society in the 21st century is changing rapidly, particularly with regards to technology, we can still learn from what happened in the past, both to appreciate what we have now and to understand how we got here.

To this end, I am delighted to see publication of a new volume devoted to the history of biomedical and health informatics. Although the book provides some historical overview of the field, its main content consists of about 160 personal stories of what led many current senior leaders to end up working in this field. I am delighted that my own story is one of those in the collection and am also pleased that the book is published in an open-access manner and is freely available as a PDF on the Web site of the International Medical Informatics Association (IMIA).

The citation for the book is, International Medical Informatics and the Transformation of Healthcare, Casimir A. Kulikowski, Editor-in-Chief; George I. Mihalas, Associate Editor-in-Chief; Robert A. Greenes, Editor; Hyeoun-Ae Park, Editor; Valerio Yácubsohn, Editor. ISSN 1485-7375. Copyright, 2021 by Healthcare Computing & Communications Canada & IMIA. The cover is shown below.

Because the book is open-source, I am taking the liberty of copying the text of my chapter into this posting verbatim (resisting temptation to edit or update):

A Passion and a Calling

My interest in biomedical and health informatics goes back to my high school days in the northern suburbs of Chicago in the early 1970s. I was introduced to computers when my school acquired a Hewlett-Packard 9830A, which was the size of a suitcase and had a built-in single-line LED display, thermal printer on top, and cassette tape storage unit. I learned how to program it in BASIC. I was also a cross-country and track runner in high school, which led to my interest in health and medicine. Running also taught me self-discipline, which helped me achieve goals later in life.

I went off to college at the University of Illinois Champaign-Urbana, where I intended to major in computer science (CS). However, I found CS to be very different in college than in high school. My first courses using punch cards and the PL/1 programming language did not excite me. I did, however, enjoy working with PLATO, a networked system with (primitive, by today’s standards) bit-mapped graphics. Two years into college, I left CS to pursue a medical career. My interest in health and preventive medicine, together with youthful rebellion, provided the foundation for my interest in evidence-based medicine.

In medical school, also at University of Illinois, I met my first informatics faculty member, Dr. Allan Levy, who nurtured my interests. Of all my education, medical school was the least enjoyable. I did not like the massive amount of rote memorization, which contributed to my later attraction to informatics. In my third year of medical school in 1983, I purchased my first computer, a Commodore 64: I hooked it up to my television as a monitor and to my phone via a 300-baud modem, and connected to Compuserve, which had a medical bulletin board called MedSIG. There I met Col. Gordon Black, who encouraged a number of us early informaticians, including long-time colleague, Rob McClure.

The reigniting of my interest in computers continued to grow as I started an internal medicine residency in 1984 at University of Illinois Hospital in Chicago. During my residency, it became apparent to me that I wanted to combine medicine and computers in my career. I came to learn about a field called “informatics,” but without Google or other search engines, there was no easy way to find more information. This led me to write letters and make phone calls to people like Ted Shortliffe, Bob Greenes, Clem McDonald, Perry Miller, and Scott Blois. I ultimately learned about National Library of Medicine (NLM)–funded informatics fellowships and chose to pursue one in the Harvard program under Bob Greenes at Brigham and Women’s Hospital in Boston. In 1987, after having lived my whole life in Illinois, I headed off with my wife to start my informatics fellowship. It was quite a change for me, with my previous daytime focus on medicine and intermittent nights-and-weekends focus on computing now flipped. In the fellowship, I could do computing almost all the time and practice medicine on the allowed one day per week. During this fellowship, it quickly became clear that informatics would become my life’s calling.

Like many working in informatics in the 1980s, I initially tried to find a research interest and niche in artificial intelligence (AI) systems of the day. One early attraction was knowledge representation, and this led to Bob involving me in his work on the Unified Medical Language System (UMLS) project that had been launched by the NLM in 1986. But the progress of the first generation of AI was sputtering by then, and almost by accident I came across a report on the topic of information retrieval (IR) authored by Bruce Croft, a computer science professor at the University of Massachusetts at Amherst. There was very little research going on in IR in medical informatics, and the main work emanated from the development of MEDLINE (although Mark Frisse had done some important work during that time in applying IR to the emerging world of hypertext). Croft’s report steered me to the most prolific researcher and author in the IR field, Gerard Salton. Many current senior leaders in IR trained as PhD students under Salton, and I was also profoundly influenced by his work. I had the chance to meet Salton when he came to give a talk at Harvard. He was intrigued by my interest in IR applied in the medical domain. I have always thought it was most unfortunate that Salton never lived to see the wide reach and impact of his ideas and work in modern search engines, as he passed away in 1995.

My clinical background dampened my enthusiasm for the relatively clunky and time-consuming AI systems of the 1980s and heightened it for IR. I was intrigued by the idea of physicians and others being able to access knowledge at the point of care. My perception of IR systems at the time was that they were limited, with systems doing just word-based searching on text or requiring complex Boolean queries over human- assigned indexing terms. My interest in IR, combined with the advancing UMLS project, led me to pursue a line of research that combined concept-based automated indexing to enhance retrieval that applied the statistical approaches developed by Salton and others. This led to me to develop and implement a system called SAPHIRE, which was the focus of my early research.

During my fellowship, I was also briefly involved with a project that would later become a highly successful commercial product. Bob had been visited by Burton Rose, a nephrologist at Brigham and Women’s who was enamored with a new tool that shipped with the Mac called Hypercard. He believed that small chunks of information on each “card” in a Hypercard “stack” could be highly useful to physicians. But as the quantity of information grew, he needed a search capability that was better than that which shipped with Hypercard. I programmed the search capability for the first version of what would later be called UpToDate, which ultimately achieved great commercial success. At the end of my fellowship, I handed this project off to another fellow, Joseph Rush, who continued to work on UpToDate for many years.

As my fellowship was ending, I knew that I wanted to pursue a career in academic medical informatics. One person I came to know was Bob Beck, who at the time was heading the informatics program at Dartmouth College. By the fall of my last year of fellowship, Bob had moved to Oregon Health Sciences University (OHSU) to start a new program there funded by the NLM IAIMS program, bringing with him another faculty member, Kent Spackman.

While I had some other job possibilities, my wife and I, now with a one-year-old daughter, packed up and moved to Portland in July 1990. My first activity in the new job was to submit an NIH R29 proposal that I had been working on in the latter months of my fellowship. Also called a FIRST Award, this type of grant was a common pathway for new researchers to launch their careers. Several months later, I was notified that it would be funded, which jump-started my academic career.

In 1990, Oregon voters passed a property tax limitation measure which ultimately led to Bob Beck losing resources and leaving in 1992. This left behind a very junior faculty, led by Kent, but as Kent wasn’t interested in building a program, he devolved the leadership to me. By 1996, our young academic group was starting to achieve sustained success. This led the Dean of the OHSU School of Medicine at the time, Joseph Bloom, to encourage our unit to become more visible on campus. The usual way of doing this at OHSU was establishing a so-called free-standing division, which was the path to establishing a department. This also provided me a seat at the table of clinical department chairs, which I maintain to this day.

I was interested in teaching from the beginning of my faculty career, and when Kent asked me to organize the introductory informatics course—something I still teach to this day—it led to many others, like the one I teach in the 10x10 program. When I started my fellowship, and then my faculty position at OHSU, I never realized how much of a passion teaching would become for me. I always enjoyed teaching because it gave me a chance to learn as well as develop a coherent organization for various topics. My path down the road to my current leadership in education was also greatly influenced by those I taught. In particular, while I assumed that our educational program would be small and aim to produce researchers like myself, there were a number of students who were interested in more varied careers, such as the small but growing number of professional positions in healthcare settings or industry. This resulted in our new Master of Science program taking on a more practical orientation. But that was fine, as the research of many of our faculty, such as Paul Gorman and Joan Ash, was motivated by real-world concerns in the application of informatics.

Even with my growing interest in education and my leadership responsibilities in our emerging program, I still maintained my interest in research. While it became more difficult to develop new IR systems when giants like Google and PubMed emerged, my interest in evaluating how well people used IR systems for health and medical reasons became the main focus of my research. In 1996, I published the first edition of my book, Information Retrieval: A Health Care Perspective.

By 1999, as I was contemplating ways to expand our educational program, a number of people had asked if we planned to offer our courses via distance learning. I decided to offer my now-mature introductory course in this manner, which was quite successful. There was an untapped market for distance learning in informatics, and the success of my initial course led me to convince the faculty to add this format to the program. This foray into distance learning distracted us from another goal we had in the late 1990s, which was to establish a PhD program. We finally accomplished this when our NLM training grant was renewed in 2002. At this point I became PI of the training grant.

Another pivotal career event for me came when Charlie Safran was President of AMIA (back in the days when the AMIA President was an elected position). He was convinced that the US needed more professionals, especially physicians and nurses, trained in informatics. Charlie believed the US needed at least one physician and one nurse trained in informatics in each of the nearly 6000 hospitals in the US. Also at this time, AMIA was looking to develop some sort of introductory course in biomedical informatics. However, the prices quoted to them by vendors were beyond their means. As I already had my introductory course from our graduate program, I proposed to AMIA that we repackage my online course. I came up with a name, 10x10 (pronounced “ten by ten”), based on Charlie’s one physician and nurse in 5000+ hospitals, and set a goal for doing so by 2010. Because the course already existed, we were able to put in place a Memorandum of Understanding between OHSU and AMIA and launch the first offering of the course in just a few months. The next President of AMIA, Don Detmer, called 10x10 one of the association’s most successful programs ever.

My interest in education and training spurred my interest in workforce development for the field. In 2006, I was invited to organize the surprise retirement event for long- time academic leader, originally from Germany and later from Victoria, Canada, Jochen Moehr. I gave a talk entitled Who are the Informaticians, What We Know and Should Know, which I later published in JAMIA. This interest was fortuitous, since the US economy would soon enter free fall, leading to the American Recovery and Reinvestment Act (ARRA), the economic stimulus bill that included the Health Information Technology for Economic and Clinical Health (HITECH) Act. While HITECH was best known for its $30 billion “meaningful use” program of incentives for EHR adoption, it also included $118 million for workforce development, motivated in part by some research I published showing a need for more informatics professionals. I played a large role in the grants that were competitively awarded by the HITECH Workforce Development Program, including being funded as the National Coordination and Dissemination Center for the health IT curriculum that was funded through the program.

During and after HITECH, I continued to provide leadership for informatics education and its relationship to other careers in the field. I was also a leader in the new clinical informatics physician subspecialty, being appointed by AMIA to direct the Clinical Informatics Board Review Course (CIBRC), which was offered in time for the first board examination in 2013. The next year I laid the groundwork at OHSU to establish one of the first four Accreditation Council for Graduate Medical Education (ACGME)–accredited fellowships for the new subspecialty, which launched in 2015. Around this time, I also had the opportunity to develop informatics education for non- informaticians, namely medical students. Along with colleagues at OHSU, we began to implement informatics education in the MD curriculum (just in time for my younger daughter to become a medical student!).

I have now been at OHSU for nearly 30 years, where I have had the opportunity to continue my research and teaching, and lead my department. Another critical activity of mine now is to mentor young faculty, who one day will sustain and lead our program.

A New Systematic Review Highlights the Current State and Limitations of Clinical AI Use and Efficacy

When I teach about the features of search engines like PubMed, I often quip that if you use the limit function to narrow your search to randomized controlled trials (RCTs), which are the best evidence for medical and health interventions, and you still have many retrievals, there is probably some enterprising researcher who has done a systematic review on the topic. Some actually worry that we have too many systematic reviews these days, not always of the greatest quality.(1) But such reviews, especially when done well, can not only be important catalogs of research on a given topic but also provide an overview of the breadth and quality of studies done.

Sure enough, we have started to see systematic reviews on artificial intelligence (AI) and machine learning (ML) applications. A new systematic review covers all of the RCTs of interventions of AI applications.(2) I hope the authors will keep the review up to date, as one limitation of systematic reviews published in journals is that they become out of date quickly, especially in rapidly moving areas such as AI.

As we know from evidence-based medicine (EBM), the best evidence for the efficacy of interventions (treatment or prevention) comes from RCTs. Ideally, these trials are well-conducted, generalizable, and well-reported. EBM defines four categories of questions that clinicians ask: intervention, diagnosis, harm, and prognosis. As such, there are other clinical questions that can be answered about AI beyond those about interventions. For example, can AI methods improve the ability to diagnose disease? Can AI identify harms from environment, medical care, etc.? And finally, can AI inform the prognosis of health and disease? Ultimately, however, AI interventions must be demonstrated experimentally to benefit patients, clinicians, and populations. There are of course some instances when RCTs are infeasible so observational studies may be justified.

In this context, we can review a recently published systematic review of interventions using AI clinical prediction tools of Zhou et al.(2) This systematic review categorized AI methods into three groups: traditional statistical (TS), mostly regression; machine learning (ML), all ML but deep learning; and deep learning (DL), i.e., applications using multi-layered "deep" neural networks. TL and MS tools were found to be used for three functions: assistive treatment decisions, assistive diagnosis, and risk stratification, whereas DL tools were only assessed for assistive diagnosis.

Typical as happens in most systematic reviews, the authors found over 26,000 papers published and retrieved by their broad MEDLINE search, but of those, there were only 65 RCTs identified. Once identified, the 65 trials were reviewed for a number of characteristics. One important characteristic was whether or not studies demonstrated a benefit for AI, i.e., had a positive result. Of course, counting numbers of positive vs. negative results is not necessarily an indicator of the value or generalizability of a particular method of AI or any other clinical intervention for that matter. Nonetheless, the authors did find that 61.5% of the RCTs had positive results and 38.5% negative results.

As AI can be used for many conditions and functions in medicine, it is important to get a sense of what was studied and what tools were used. The authors found use for AI in a variety of disease categories: acute disease (29%), non-cancer chronic disease (28%), cancer (17%), primary care (14%), and other conditions (12%). Of the predictive tool function used, use was most often for assistive treatment decisions (54%), followed by assistive diagnosis (25%) and risk stratification (19%). There were the most studies used for TS (57%), followed by ML (26%) and DL (17%). These differences may reflect the more recent development and use of ML and especially DL. The rates of positive studies for the tool types were highest for DL (82%), followed by ML (71%) and TS (51%), although it should be noted that the rate of positive results was also inversely related to the number of trials for each tool type.

A table in the paper shows that there were differences by tool categories. TS tools were mostly likely to be used with clinical quantitative data (97%), applied in acute disease (43%) and primary care (24%), and used for assistive treatment decisions (60%) followed by risk stratification (30%). ML tools were also most likely to be used by clinical quantitative data (94%), applied in chronic disease (77%), and used for assistive treatment decisions (77%). DL tools were most likely to be used with imaging data (91%), applied in cancer (91%), and used exclusively for assistive diagnosis (100%). In particular, the DL studies almost exclusively evaluated assistance of gastrointestinal endoscopy, with all nine such RCTs showing positive results and the two trials of other applications and diseases having negative results. Also of note, only two of the 65 RCTs made use of natural language data for input, one ML and one DL.

Systematic reviews typically appraise included studies for risk of bias, or in other words, the quality of their methods to produce an unbiased result. This provides confidence that the research methods were robust and well-reported so readers can have confidence that the results obtain are true. Unfortunately, there were a number of concerns that led to 48 (74%) of the trials being classified as having high or indeterminate risk of bias. This was due to a number of factors:

• One-third of the trials carried out no sample size estimation to determine what would be the number of subjects needed to achieve a statistically significant benefit
• Three-fourths of the trials were open-label, so had no masking of the AI system from its users
• Three-fourths did not reference the CONSORT statement, a 37-item checklist widely used for reporting the details of RCTs and recently extended for AI trials
• Three-fifths did not apply an intent-to-treat analysis, which evaluates subjects in the study groups into which they were originally assigned
• Three-fourths did not provide reference to a study protocol for the trial

The rate of outcomes of studies for low risk of bias trials was somewhat comparable to the overall rates, with positive outcomes in 63% of TS, 25% of ML, and 80% of DL trials.

What can be concluded from this systematic review? We certainly know from the vast amount of other literature that a large number of predictive models have been built using AI techniques and shown to function well for a wide variety of clinical conditions and situations. We probably cannot do an RCT of every last application of AI. But at this point in time, the number and variety of RCTs assessing benefit for interventions of AI is modest and uneven. While a number of positive results have been demonstrated, the studies published have not been dispersed across all of the possible clinical applications of AI, and three-fourths of the reports of the trials show indeterminate or high risk of bias. DL methods in particular must be assessed in the myriad of areas in which data sets have been developed and models trained.

There are some problems with the systematic review itself that mar the complete understanding of the work. Table 2 of DL interventions has data missing in its leftmost column that connects the data in the column to its original reference. This table also does not include a recent paper by Yao et al.,(3) which was likely published after the review was completed. It is also difficult to use the data in Supplementary Table 4 of ML interventions, which is provided in a PDF file that is difficult to read or browse. In addition, while the paper references a high-profile study by Wijnberge et al.,(4) it is not listed in ML table. This study may well be classified as TS, but this demonstrates another limitation of the systematic review, which is that there is no data or table that details TS interventions. The authors were kind enough to provide Excel files of the DL and ML tables, but they really should be part of the online materials for the systematic review. I do hope they or someone will keep the review up to date.

As it stands, this systematic review does give us a big-picture view of the clinical use and benefit for AI at this point in time, which is modest, disproportionate, and based on studies using suboptimal methods. We can conclude for now that AI predictive tools show great promise in improving clinical decisions for diagnosis, treatment, and risk stratification but comprehensive evidence for the benefit is lacking.

This systematic review also highlights a point I have written about in this blog before, which is that AI interventions need translation from basic science to clinical value. In particular, we need clinically-driven applications of AI that are assessed in robust clinical trials. There of course must also be attention to patient safety and to clinician workflow. In general, we need robust AI and RCT methods that are replicable and generalizable, and of course we must conduct implementation and trials from a health equity standpoint.

References

1.     Ioannidis JPA. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485–514.
2.     Zhou Q, Chen Z-H, Cao Y-H, Peng S. Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review. NPJ Digit Med. 2021 Oct 28;4(1):154.
3.     Yao X, Rushlow DR, Inselman JW, McCoy RG, Thacher TD, Behnken EM, et al. Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial. Nat Med. 2021 May;27(5):815–9.
4.     Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, et al. Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial. JAMA. 2020 Mar 17;323(11):1052–60.

This Year's Eco-Event: An E-Bike

If last year's eco-event for me was installing solar panels on the roof of my house, this year's event was the purchase of an electric bike (e-bike). I have to admit I am not completely virtuous when it comes to my carbon footprint. While my time spent in airplanes has reduced dramatically during the pandemic, it is likely to increase, although probably not to pre-pandemic levels, as we all return to travel.

But one thing I do hope as there is eventual return to working in the office is to commute in all but the worst weather by e-bike. One of the challenges for bicycle commuting for me is that although I only live 4-5 miles (depending on the route) away from my office, I must ascend over 500 feet vertically to get between my home and office. (Those in Portland know the Fairmount Loop just below the Council Crest hill that sits above Oregon Health & Science University.)

Likewise, a similar hill separates my home from the short distance to downtown and other parts of Portland that would be wonderful to leisurely ride to on a bicycle. I sometimes do this ride on a regular bicycle, but it is a real workout. There are other times when I would just rather get somewhere without breaking a major sweat.

After doing much online research, I decided to test drive, and eventually purchase, an Aventon Pace 500 Step-Through e-bike. There are many e-bike options, and the market for these products is not yet fully mature. But I have enjoyed my e-bike, and I have to say it is actually quite fun to ride. There is nothing like a little electric assist when riding a bike, especially in the hills of the west side of Portland.

Another aspect of the Pace 500 I enjoy is it being a Class 3 bike, which allows it to have a throttle. While I use the throttle sparingly, I do find it of great value when I need to accelerate quickly, such as at a traffic light turning green or starting up a hill. The Class 3 status also allows the motor to speed the bike to up to 28 mph in its highest (5) pedal assist level, although I try to ride at no higher than pedal assist 2, which gives me plenty of power even up hills.

Musings on a Tool Every Academic Should Use

I often muse that there are few computer applications that truly save me time. For all the fun and productive things that computers enable me to do, they are just as often a time sink rather than a time saver, especially when hardware or software go wrong. The transition from being able to talk to a person on the phone to the use of chatbots and other ways to keep people with problems away from costly human support has added even more time, especially during the pandemic.

There is, however, one notable exception, a computer application that saves me a great deal of time. This is bibliographic management software. Any academic who writes a great deal, especially those who publish in journals and other venues across different disciplines, knows the time and effort required to maintain and format references.

 

As with all computer applications, one must choose their bibliographic management software package wisely. I was an early user of EndNote, back to Version 1 when it was first released in the 1990s. It had served me well over the years, but one challenge was that as the formats of bibliographic records changed, I was not able to take advantage of its automatic capture of metadata. I was also not able to easily merge my EndNote database with those of others, again due to the formatting issues. By the end of last year, my database of papers that I have cited once or more in research writing or teaching, had grown to over 12,000 entries, yet my effort to use the product put me in a silo.

 

This year I decided to make a decision to start over. My first decision was whether to continue with EndNote or move to a different package. One package that many of my colleagues seemed to be adopting was Zotero. This package has the advantage of being open-source, with a large developer community. I decided to make the switch.

 

The transition has not been simple, as my existing EndNote library had too many irregularities for me to simply import it into Zotero. However, I have been able to build up my new Zotero database relatively quickly due to its automated capture of metadata. (In fairness to EndNote, they have this feature as well.) The automated capture of metadata is not perfect, mainly because many Web sites and pages do not adhere to standards. But many key sources, such as PubMed, most journals, Amazon (for books), and others make entry into the database quick. One notable feature for someone concerned with the big picture of science is Zotero's ability to flag that a scientific paper has been retracted.

 

Zotero is not perfect, and one feature I hope is added soon is the ability to easily update a record, such as when a journal paper goes from "online ahead of print" to actually being "in print," i.e., having a volume, issue, and page numbers (even if many journals are not physically printed these days). The metadata of journal articles does change over time, and the ability to automate the capture of its changing as easily as its initial capture would be a great feature.

 

Nonetheless, bibliographic management software is a vitally important tool for those who write scientific papers, especially in inter-disciplinary fields like informatics. And the decision on which package to use is important, as changing from one to another can be time-consuming. But it is certainly an application whose proper use can save time overall.

Certification for the Rest of Informatics

After several years of planning, professional certification is coming to the rest of the informatics field, i.e., moving beyond just board certification for eligible physicians. While certification is somewhat easier to apply in the context of the physician board model, the American Medical Informatics Association (AMIA) has now rolled out the AMIA Health Informatics Certification (AHIC, formerly Advanced Health Informatics Certification). Those who are certified will be designated as ACHIP, the AMIA Certified Health Informatics Professional. A section of the AMIA Web site provides detailed on the certification, eligibility for it, applying for and taking the exam, and recertification.

While AHIC is open to all who have a master's or doctoral degree in health informatics or a related discipline, the certification process is not conferred upon initial completion of one's education. Rather, individuals also need to have completed qualifying work experience to be eligible for certification. This is different from some fields, such as medicine, including the clinical informatics subspecialty, where one takes the board certification exam shortly after completing formal training. There are a number of healthcare disciplines that require significant work experience for certification, such as some of the advanced certifications offered by the American Nurses Credentialing Center.

The qualifications for AHIC are listed in a table on the AHIC Web site. There are two tracks of eligibility. Track 1 is for those who have a graduate degree in a health informatics-related area, e.g., health informatics, biomedical informatics, nursing informatics, public health informatics, translational bioinformatics, etc. Track 2 is for those who have a graduate degree in a related field, e.g., health professions such as nursing, pharmacy, and medicine, and other fields such as computer science and public health. The work time required for those in Track 1 is 50-100% work time over the last four of six years or 20-49% time over the last six of eight years. The work time required for those in Track 2 is 50-100% work time over the last six of eight years or 20-49% time over the last eight of 10 years.

The certification process is being developed and managed by the Health Informatics Certification Commission (HICC), a 14-member commission that is part of AMIA yet has considerable autonomy from AMIA, especially with regards to AMIA's educational programs. The HICC is responsible for eligibility, examination development, and recertification requirements for AHIC.

The first offering of the certification exam is taking place this fall. The outline of exam topics follows the health informatics workforce analysis commissioned by AMIA (Gadd, C.S., Steen, E.B., Caro, C.M., Greenberg, S., Williamson, J.J., Fridsma, D.B., 2020. Domains, tasks, and knowledge for health informatics practice: results of a practice analysis. J Am Med Inform Assoc 27, 845–852. https://doi.org/10.1093/jamia/ocaa018), just as the clinical informatics subspecialty exams now uses the complementary clinical informatics subspecialty workforce analysis for its exam blueprint.

The two questions someone enrolled in or contemplating seeking a degree in informatics will likely ask are: (1) Is this certification process for me? and (2) Will it benefit my career? Since this form of certification is new for professionals who work in informatics, the benefits at this time are unknown. The main drivers of the uptake will be employers who make hiring decisions that are influenced by job candidates having the certification. Similar to the clinical informatics subspecialty, we will probably see a gradual uptake of the AHIC over time. It may never be an absolute requirement for a job but it will be an important "feather in one's cap" when competing with others for a given position.

Scientific Rankings for the Informatics Professor

While I agree with those who argue that scientific rankings, especially based on bibliographic citation indicators, are limited in their measurement of a scientist's impact, I must admit a certain fascination with them. Perhaps that stems from my interest in dissemination and retrieval of scientific information generally. And perhaps also, because I enjoy writing and tend to measure well by these metrics.

I do show up in most rankings of my primary and related scientific fields. As I consider my primary scientific discipline to be biomedical informatics, I can report that I rank 57th on one list of researchers in the field. On another more focused list of those who work in medical informatics, I rank 14th, although my ranking falls to 33rd when the list is less focused (details below - also see [1,2]). I am also on a global list of the top 1000 computer science and electronics researchers, where I rank 932nd globally and 569th among Americans. On a more focused computer science list for the information retrieval field, I rank 23rd.

A description of how these rankings are calculated gives some perspective into my positions on them. All of these rankings make use of the well-known citation measure, the h-index, although one uses additional factors. The h-index is a measure of the number of one's publications that have been cited by at least that same number of publications. So for example, if one has 15 papers that have been cited 15 or more times, their h-index is 15. There are two main public sources of h-index values that are most commonly used, which give different results due to the way they are calculated. The two are Google Scholar and Scopus, the latter an arm of the scientific publishing conglomerate, Elsevier. The Google Scholar h-index is usually higher than the Scopus h-index due to the former including a wide variety of academic products, such as conference proceedings, books, non-peer-reviewed reports, and other publications on the Internet, whereas the latter is limited to journal publications. As the Google Scholar value is also generated automatically, it is more likely to contain erroneously included papers, especially when author names are ambiguous. My current Google Scholar and Scopus h-index values are, as of this writing, 75 and 46 respectively.

Obviously the h-index is related to the duration of one's career, and as citation patterns vary in different fields, one must compare the h-index of different individuals with caution. With those caveats, we can explore further my own ranks. The list of biomedical informatics researchers is maintained by Allison McCoy of Vanderbilt University. One concern about this list is that it contains a number of researchers who, although published in the biomedical informatics literature, do not primarily work in the biomedical informatics field. This list is generated from software developed by Jimmy Lin of University of Waterloo, who maintains the list of information retrieval researchers (and several other fields within computer science). The list of top worldwide computer science researchers is maintained by a Web site devoted to computer science research, Guide2Research.

An additional ranking in which I appear is one compiled by John Ioannidis of Stanford University and colleagues [1,2]. This analysis includes the top 100,000 scientists across all fields, with additional enrichment from those in the top 2% of their field but not in the top 100,000. Unlike the other sources, this analysis is fixed, with data through 2019 and taking more factors into account than just the h-index. A composite C-score is made up of six factors measured from citations through 2019 and excludes self-citations:

• h19 (ns)    h-index as of end-2019
• hm19 (ns)    hm-index as of end-2019
• ncs (ns)    total citations to single authored papers
• ncsf (ns)    total citations to single+first authored papers
• npsfl (ns)    number of single+first+last authored papers
• ncsfl (ns)    total citations to single+first+last authored papers

The rationale for this more complex measure is based on observations that (a) in some fields, many papers have vast numbers of authors, (b) these large numbers of authors give great weight to measures based purely on citations, (c) many Nobel laureates do not rate highly in simple citation measures such as h-index, (d) many of those who rank highly in simple citation measures have few or no first-authored or last-authored papers, and (e) Nobel laureates rank higher when more complex measures such as a C-score are employed.

In this cast of more than a hundred thousand, my C-score of 3.938 gives me an overall rank of 22,034, which is based on 241 papers published and 6109 citations to them through 2019. As noted above, I rank 15th among those whose primary field is medical informatics. There are also others for whom medical informatics is listed as their secondary field, and when combined with those for whom it is primary, my ranking is 33rd. There is a separate ranking for those whose primary field is bioinformatics. 

I can also extract out all researchers in the ranking from my institution and its affiliates (Oregon Health & Science University, OHSU School of Medicine, Oregon National Primate Research Center, and Portland VA Medical Center) and note that I rank 49th out of 256 included in this list. (I am also pleased to note that 10 people from my department make it on to the overall OHSU list, including Roger Chou, Heidi Nelson, Mark Helfand, Joan Ash, Cynthia Morris, Paul Gorman, Rochelle Fu, Linda Humphrey, and Aaron Cohen.)

One interesting aspect of the Ioannidis et al. analysis is that I rank better using the composite score than just by my h-index. Based solely on the h-index, I would rank only 131st for OHSU and 37th in the primary medical informatics list. My C-score is improved by my relatively higher number of first-author and single-author papers, and citations to them. I also must have fewer co-authors on my papers than my colleagues at OHSU and in informatics, as I do better with the hm-index, which adjusts for the number of authors on a paper. At OHSU in particular, where I rank 49th overall and 131st by h-index, I rank 51st in hm-index, 37th in citations to single-authored papers, and 41st in citations to single- and first-authored papers. My 104 single- and first-authored papers rank me 22nd at OHSU. My data for the Ioannidis et al. analysis is available in a spreadsheet (Enjoy!).

On a final note, I am pleased to report that citation indices are a family affair for me. My daughter Alyssa Hersh, MD, MPH is currently a resident in Obstetrics & Gynecology at OHSU. She is also a rising researcher, and as of this writing has a Google Scholar h-index of 6 and a Scopus h-index of 4. I have no doubt she will surpass my current citation metrics long before she reaches my current age!

References

[1] Ioannidis, J.P.A., Klavans, R., Boyack, K.W., 2016. Multiple Citation Indicators and Their Composite across Scientific Disciplines. PLoS Biol 14, e1002501. https://doi.org/10.1371/journal.pbio.1002501.

[2] Ioannidis, J.P.A., Boyack, K.W., Baas, J., 2020. Updated science-wide author databases of standardized citation indicators. PLoS Biol 18, e3000918. https://doi.org/10.1371/journal.pbio.3000918.

Translational Artificial Intelligence: A Grand Challenge for AI

The potential for artificial intelligence (AI) to transform biomedicine and health is immense, yet at this time that potential remains largely unfulfilled. As I have noted in this blog over the years, there have been many impressive achievements in applying AI to biomedical and health data using clean and well-curated data sets, yet its use in every day medical care or health pursuits is modest. I have noted the reasons for this gap over the years in various postings, namely in the need to move capabilities beyond prediction to the ability to take action on them and the importance of showing actual clinical value, whether in leading to better health outcomes or care delivery. Failure to achieve these will relegate AI to the same fate as its first generation in the last century.

The premiere biomedical research agency in the US, the National Institutes of Health (NIH), has recognized the need to advance the science of AI in biomedicine. As such, the NIH has launched a new initiative, Bridge to Artificial Intelligence (Bridge2AI), which aims to "propel biomedical research forward by setting the stage for widespread adoption of artificial intelligence (AI) that tackles complex biomedical challenges beyond human intuition."

I applaud this program, and hope it will address the larger picture of translating current advances in AI into algorithms and systems that truly lead to improved health outcomes and care delivery. Along the way, it will hopefully address other issues related to AI, such as deploying AI ethically, focusing on solving important problems, and developing human expertise, not only of those who implement and evaluate AI systems but also the clinicians who employ them in their professional practice and patients who understand the benefits and limitations.

The Bridge2AI program recently hosted a virtual workshop series. One of these was devoted to enumerating grand challenges for AI in biomedicine. Participants were allowed to suggest grand challenges related to any aspects of the problems or solutions. I submitted a grand challenge focused on what I perceive is the need for translational AI, i.e., building on the successes in the "basic science" of showing the predictive value of algorithms to studying and ultimately deploying evidence-based AI in the real world. My grand challenge was one of 25 accepted by the meeting organizers for presentation to workshop attendees. We were allowed to use one slide, which is shown below.

To reiterate from my slide and presentation, I noted that most AI and machine learning research is still at the "basic science" stage of the biomedical research pipeline. Advances have been made with clean and well-curated data sets in simulated settings. Just as drugs and devices must progress from the lab to patient care, AI advances must do the same. In other words, moving from prediction to prescription and providing value. There are all sorts of potential ethical issues that must be addressed in the translational activities, such as balancing privacy protection vs. the public good, identifying and eliminating bias in data and algorithms, and ensuring the resulting actions do not exacerbate groups that have historically been discriminated against in health and healthcare. It is also critical that the "people perspectives" are addressed to develop the expertise in scientific as well as sociotechnical issues, including patients/consumers, clinicians, and researchers. In particular, appropriate education must be provided for clinical users and leaders to maximize value and minimize harm from these approaches. I am hardly the only person to hold these views, and a number of others have articulated the importance of AI providing real-world value in its application in biomedicine and health.[1-3]

One concern I have for Bridge2AI is that the first round of funding opportunities is focused on data generation projects. Generating high-quality, ethically-sourced, and relevant data is of course necessary, but is not sufficient. In fact, I have a hard time aligning what is proposed in my grand challenge with this initial funding opportunity. I hope that Bridge2AI will broad the focus of research and usher in translational research and ultimately demonstrates its true potential to improve human health.

References

1. Allen, B., Agarwal, S., Kalpathy-Cramer, J., Dreyer, K., 2019. Democratizing AI. Journal of the American College of Radiology 16, 961–963. https://doi.org/10.1016/j.jacr.2019.04.023

2. Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V.X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., Ossorio, P.N., Thadaney-Israni, S., Goldenberg, A., 2019. Do no harm: a roadmap for responsible machine learning for health care. Nat. Med. 25, 1337–1340. https://doi.org/10.1038/s41591-019-0548-6

3. Faes, L., Liu, X., Wagner, S.K., Fu, D.J., Balaskas, K., Sim, D.A., Bachmann, L.M., Keane, P.A., Denniston, A.K., 2020. A Clinician’s Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies. Transl Vis Sci Technol 9, 7. https://doi.org/10.1167/tvst.9.2.7

Virtual Graduation Message for 2021 OHSU Biomedical Informatics Graduates

I never imagined before 2020 that I would ever take part in a virtual graduation ceremony as I did in 2020, and now it is hard to believe that I am doing the same in 2021. As I noted last year, one of my favorite activities of the academic year is the Convocation and Hooding Ceremony, where we honor graduates of Oregon Health & Science University (OHSU), including those graduating from the OHSU Biomedical Informatics Graduate Program. Despite the pandemic over the last year and a quarter, we have 33 graduates this year from our PhD, Master of Science and Graduate Certificate programs. Since the programs inception in 1996, we have awarded 905 degrees and certificates to 815 people. I am very proud of all they accomplished during their time at OHSU and now beyond in various academic, industry, government, and other settings.

This year’s annual OHSU Convocation and Hooding Ceremony will take place virtually on Sunday, June 6, as noted in our department blog. Here is the transcript of my video message to the Class of 2021 graduates of the OHSU Biomedical Informatics Graduate Program:

It gives me great pleasure to welcome the 2021 graduates of the OHSU Biomedical Informatics Graduate Program to this year’s virtual Commencement ceremony. I never thought we would have a second virtual Commencement this year, but the pandemic is not yet fully behind us. With more and more people getting vaccinated each day, however, I am confident that next year’s ceremony will be in person, and I hope OHSU finds a way for those participating virtually the last couple years to take part in person in some way. If nothing else, we will celebrate next year at our annual DMICE banquet around graduation time that all alumni are always invited to attend. I do miss the pomp and circumstance of graduation, and getting to wear regalia and march in the procession.

As many of you know, the annual Commencement ceremony is an important event for me, as I enjoy every year celebrating the success of our graduates and their moving on to new paths in their lives. We have been awarding degrees and certificates from our program since 1998, and only once have I had to miss Commencement.

In any case, most of you are now moving from your studies into jobs where the contributions of informatics are more critical than ever. Just as the pandemic has exposed problems in our healthcare system, it has also exposed limitations in our data and information systems. It is critical for all informatics graduates, and everyone else in the informatics field, to keep improving how we use data and information, not only to overcome COVID-19 but also to improve health, health care, public health, and biomedical research generally. From bio- to imaging to clinical and public health informatics, the challenges have never been greater. I am confident that you all have the talent, and the knowledge and skills you have acquired in your studies, to meet those challenges.

I am pleased to report that with the 33 of you graduating today, our program has now awarded over 900 degrees and certificates dating back to 1998. These include 398 master’s degrees and 34 PhD degrees. Our graduates have achieved success in academia, industry, government, and just about every other place where informaticians work. Your success is one of the main aspects of our work that gives faculty and staff great satisfaction.

Let me close as always to remind you that even though you are moving on from OHSU and DMICE, we are still here for you and hope you will keep in touch with us as your careers develop and prosper.

Response to NIH RFI: Comments and Suggestions to Advance and Strengthen Racial Equity, Diversity, and Inclusion in the Biomedical Research Workforce and Advance Health Disparities and Health Equity Research

In addition to its public health problems, the COVID-19 pandemic has exposed other fault lines in society, not the least of which is systematic racism that still pervades American society. A more equitable society would ensure diversity and inclusion in all aspects of life, including in healthcare and in biomedical research. The National Institutes of Health (NIH), the premier US federal agency that funds biomedical research, has recognized the need for its activities to be more inclusive of all Americans from every background. Not only must biomedical research reflect the health issues for the entire US population, its workforce should ideally reflect the ethnic and racial makeup of our larger society. The NIH recently issued a request for information (RFI), asking for Comments and Suggestions to Advance and Strengthen Racial Equity, Diversity, and Inclusion in the Biomedical Research Workforce and Advance Health Disparities and Health Equity Research. This posting contains the comments I submitted in response to this RFI.

While others will likely comment on the need for biomedical research itself to address health disparities and move toward health equity, it is equally important to address the needs of the biomedical research workforce that will contribute solutions to these problems. The NIH has already made a tremendous commitment to diversity and inclusion, including in building career pathways the biomedical research workforce, but additional efforts must be made to facilitate access to these programs by extramural researchers and leaders.

One example is the Building Infrastructure Leading to Diversity (BUILD) Initiative, which has a prominent program in our region.

Nonetheless, there are still challenges for engaging the potential future biomedical research workforce. My particular concern is how to increase diversity and inclusion among faculty of academic health science centers.

One of the challenges is explaining to young minds the opportunity and the work of biomedical research. While most young people are familiar with healthcare professionals - i.e., physicians, nurses, and pharmacists - fewer are familiar with the work and importance of researchers. Schools and communities themselves may not be aware of career opportunities. There should be resources committed, and easy-to-use tools made available, so that academic health science centers and others can disseminate information about careers and the rewarding work of biomedical research.

A second challenge is for researchers themselves to have the time to engage in such mentoring and teaching. As demands for productivity by biomedical research faculty in academic health science centers increase - i.e., keep grants funded and students taught - there is less time in their busy schedules for this critical activity. Such activity is also unlikely to "count" toward promotion or lead to that next grant. There should be standards for promotion committees in academic health science centers to require diversity and inclusion outreach. Of course, this must not be an "unfunded mandate," and instead be an activity that has committed time from institutions.

A third challenge is that success in biomedical research typically requires a graduate degree. As such, the road to college and then graduate or professional school is long and can be expensive. There must be pathways for students, especially for those of limited resources with few parental or other role models, to be helped through that long path. There should be opportunities provided, along with appropriate mentoring, for students to enable them to stay engaged during the long journey. Students should not only be given sustained exposure, but also be taught knowledge and skills along the way.

My own work is as an academic faculty in biomedical and health informatics, where I daily experience the satisfaction of research and teaching. While my field has made some strides in diversity and inclusion, it still has a long way to go to reflect our the racial and ethnic distribution of our larger society. Many who want to spend time engaging with future diverse researchers and professionals in the field need help in overcoming the above barriers. This leads to questions that must be answered:

1. How do we engage with schools, community organizations, and others to expose high school and perhaps even younger students to biomedical research?
2. How do we provide academic faculty with the protected time and academic credit for this work of critical importance?
3. How do we develop pathways to sustain the interest and achievement for students, especially those from backgrounds that include little exposure to higher education?

We can and should require our academic health science centers and their faculty and others to engage with historically underrepresented groups in biomedical research, but make sure that they have the time and the tools, with milestones and outcomes measured, to achieve these goals. This should consist of:

1. The availability of resources and tools to engage young minds in the possibilities for careers in biomedical research
2. Expectations and protected time for existing faculty to devote effort to engaging with young students, including requirements to achieve promotion
3. Developing pathways to sustain interest and achievement toward careers in biomedical research

By making diversity and inclusion efforts an expected activity of all biomedical research faculty and providing such faculty the resources and opportunities, we can achieve the shared aim of the biomedical research workforce and its activities reflecting the larger population of our country.

A New OHSU Course in Applied Clinical Data Science and Machine Learning for Health & Clinical Informatics (HCIN) Students

I have written over the years about the need for all who work in biomedical and health informatics to have appropriate knowledge and skills in data science, machine learning (ML), artificial intelligence (AI), and related topics. I am now excited to announce that our OHSU Biomedical Informatics Graduate Program is launching a new course in Applied Clinical Data Science and Machine Learning for Health & Clinical Informatics (HCIN) majors.

The goal of this new course is not to provide students with the mastery of ML and AI tools and techniques; rather, it is to provide a conceptual understanding of their practical application in health and biomedicine. The course is not meant to be a substitute for the sequence of courses available in the other major in our program, Bioinformatics & Computational Biomedicine (BCB), whose offerings delve far more into the theory, mathematics, and programming of these topics and include:

• BMI 551/651 - Statistical Methods
• BMI 531/631 - Probability and Statistical Inference
• BMI 543/643 - Machine Learning
• BMI 525/625 - Principles and Practice of Data Visualization

The new HCIN course will be focused on applied data science and machine learning, with a focus on clinical data sets as well as clinical issues and challenges in their application. While the course will have some programming activity (requiring Python programming as a prerequisite), it will focus on a hands-on, high-level view of the different types of machine learning methods and their applications. It will also cover the topics of data management and selection, pitfalls in building and deploying models, and critical appraisal of clinical machine learning literature. The course will aim to provide an in-depth understanding for those who will work alongside experts who develop, build models, implement, and evaluate machine learning applications in health and clinical settings.

The textbook for the course will be: Hoyt, R. and Muenchen, R. (Eds.), 2019. Introduction to Biomedical Data Science, Lulu.com. The course syllabus provides further details on the topics to be covered.

The content of the course will be based on a combination of what faculty and students believe is most important for a course like this. Among the topics that be included are:

• Data sources - electronic health records, registries (e.g., N3C, AllOfUs), patient-generated, social media, public health
• Data preparation (wrangling) - cleaning, quality analysis, feature selection, de-biasing
• Exploratory data analysis - summaries, correlations, visualizations
• Machine learning approaches and models - supervised, unsupervised, reinforcement, deep learning
• Software and tools available
• Common pitfalls and misunderstandings of applying machine learning
• Critical appraisal of clinical machine learning literature
• Ethical issues and challenges

The 3-credit course will be taught in the OHSU spring academic quarter, which runs from late March to early June. The lead instructors will be Steven Chamberlin, ND and myself, with other department faculty contributing. As with all courses in the HCIN major, it will be mostly online and asynchronous, with some option synchronous activities (which will be recorded for those not able to attend). This course will be different from to complementary to other data science-related courses in the HCIN major, including:

• BSTA 525 - Introduction to Biostatistics
• BMI 540/640 - Computer Science and Programming for Clinical Informatics
• BMI 544/644 - Databases
• BMI 524/624 - Data Analytics for Healthcare
• BMI 516/616 - Standards/Interoperability in Healthcare
• BMI 537/637 - Healthcare Quality
• BMI 525/625 - Principles and Practice of Data Visualization
  

I will be excited to see how this course is accepted and how it evolves based on feedback of students and others. I suspect there will be interest beyond our graduate program.

Vaccinated and Vaccinating: The End May Be Near?

I was delighted to learn in early January that my institution, Oregon Health & Science University (OHSU), made the decision like many medical centers to offer the SARS-CoV-2 vaccine to all employees, not just those at the front line of care delivery. I received my first and second doses of the Pfizer vaccine on January 2nd and 23rd. I had some minor malaise the day after the second dose, but was thrilled to have received the vaccine.

I also decided that since I received an early dose, I would do everything I could to support the national and global effort to disseminate the vaccine. To that end, I have volunteered to work shifts at the OHSU Portland International Airport Vaccine Clinic. While I thought I might put my medical training to use giving injections, it turns out that the greater need was for registration and check-in personnel. I suppose it is most appropriate for the Chair of the informatics department to be checking in and scheduling follow-up appointments in Epic for those coming for their shots. But I actually enjoy the job I am doing at the site, interacting with people driving through the site and expressing gratitude they are able to get vaccinated. It is also nice to put on a friendly face for our university.

Overall, I feel a sense that the end may be near for the worst of this pandemic that has upended our lives. While the complete end will not come any time soon, and we will likely need to be vigilant about SARS-CoV-2 for years to come, I am hopeful that the vaccine rollout will continue at a strong pace and allow us to gradually resume more normal living. I am also encouraged that the COVID-19 numbers of cases, hospitalizations, and deaths are trending downward, and that we have new science-driven leadership in our federal government.

Looking ahead, I yearn to be around people at work, in social settings, and, yes, traveling. Regarding the latter, it has been almost a year since I have been on an airplane, although I am planning to visit my elderly stepfather, my last living adult relative, next month in Florida. He will have received his second dose a couple weeks before I visit.

There are many unanswered questions about what life will be like in the long run. Will work move to a more virtual arrangement? What will come of city centers that have been hurt by the pandemic and resulting economic and social upheaval? What will come of academic meetings and conferences, many of which probably could be done more virtually? Even though I spend a great deal of work time in front of a computer, I am still a social being. Social media has taken the sting off of the interpersonal isolation, but there is nothing like being around other people, and I am hopeful that much of that will eventually return. We will see as 2021 unfolds.