Showing posts sorted by relevance for query data science machine learning. Sort by date Show all posts
Showing posts sorted by relevance for query data science machine learning. Sort by date Show all posts

Thursday, February 25, 2021

A New OHSU Course in Applied Clinical Data Science and Machine Learning for Health & Clinical Informatics (HCIN) Students

I have written over the years about the need for all who work in biomedical and health informatics to have appropriate knowledge and skills in data science, machine learning (ML), artificial intelligence (AI), and related topics. I am now excited to announce that our OHSU Biomedical Informatics Graduate Program is launching a new course in Applied Clinical Data Science and Machine Learning for Health & Clinical Informatics (HCIN) majors.

The goal of this new course is not to provide students with the mastery of ML and AI tools and techniques; rather, it is to provide a conceptual understanding of their practical application in health and biomedicine. The course is not meant to be a substitute for the sequence of courses available in the other major in our program, Bioinformatics & Computational Biomedicine (BCB), whose offerings delve far more into the theory, mathematics, and programming of these topics and include:

  • BMI 551/651 - Statistical Methods
  • BMI 531/631 - Probability and Statistical Inference
  • BMI 543/643 - Machine Learning
  • BMI 525/625 - Principles and Practice of Data Visualization

The new HCIN course will be focused on applied data science and machine learning, with a focus on clinical data sets as well as clinical issues and challenges in their application. While the course will have some programming activity (requiring Python programming as a prerequisite), it will focus on a hands-on, high-level view of the different types of machine learning methods and their applications. It will also cover the topics of data management and selection, pitfalls in building and deploying models, and critical appraisal of clinical machine learning literature. The course will aim to provide an in-depth understanding for those who will work alongside experts who develop, build models, implement, and evaluate machine learning applications in health and clinical settings.

The textbook for the course will be: Hoyt, R. and Muenchen, R. (Eds.), 2019. Introduction to Biomedical Data Science, Lulu.com. The course syllabus provides further details on the topics to be covered.

The content of the course will be based on a combination of what faculty and students believe is most important for a course like this. Among the topics that be included are:

  • Data sources - electronic health records, registries (e.g., N3C, AllOfUs), patient-generated, social media, public health
  • Data preparation (wrangling) - cleaning, quality analysis, feature selection, de-biasing
  • Exploratory data analysis - summaries, correlations, visualizations
  • Machine learning approaches and models - supervised, unsupervised, reinforcement, deep learning
  • Software and tools available
  • Common pitfalls and misunderstandings of applying machine learning
  • Critical appraisal of clinical machine learning literature
  • Ethical issues and challenges

The 3-credit course will be taught in the OHSU spring academic quarter, which runs from late March to early June. The lead instructors will be Steven Chamberlin, ND and myself, with other department faculty contributing. As with all courses in the HCIN major, it will be mostly online and asynchronous, with some option synchronous activities (which will be recorded for those not able to attend). This course will be different from to complementary to other data science-related courses in the HCIN major, including:

  • BSTA 525 - Introduction to Biostatistics
  • BMI 540/640 - Computer Science and Programming for Clinical Informatics
  • BMI 544/644 - Databases
  • BMI 524/624 - Data Analytics for Healthcare
  • BMI 516/616 - Standards/Interoperability in Healthcare
  • BMI 537/637 - Healthcare Quality
  • BMI 525/625 - Principles and Practice of Data Visualization

I will be excited to see how this course is accepted and how it evolves based on feedback of students and others. I suspect there will be interest beyond our graduate program.

Friday, October 12, 2018

What are the Optimal Data Science and Machine Learning Competencies for Informatics Professionals?

Exactly 20 years ago, I organized a panel at the American Medical Informatics Association (AMIA) Annual Symposium that attracted so large an audience that the crowd spilled out of the room into the hallway. Entitled, What are the Optimal Computer Science Competencies for Medical Informatics Professionals?, the panel asked how much knowledge and skills in computer science were required to work professionally in informatics. In the early days of informatics, most informaticians had some programming skills and often contributed to the development of home-grown systems. Some educational programs, such as the one at Stanford University, had required courses in assembly language. (I took an assembler course myself during my informatics fellowship in the late 1980s.)

But as academic informatics systems grew in scope and complexity, they needed more engineering and hardening as they became mission-critical to organizations. At the same time, there was a recognized need for attention to people and organizational issues, especially in complex adaptive settings such as hospitals. Over time, most professional work in informatics has shifted from system building to implementing commercial systems.

With these changes, my evolving view has been that although few informatics professionals perform major computer programming, there is still value to understanding the concepts and thought process of computer science. While plenty of students enter our graduate program at Oregon Health & Science University with programming skills, our program will not turn those without programming skills into seasoned programmers. But I still believe it is important for all informatics professionals to understand the science of computing, even at the present time. This includes some programming to see computing concepts in action.

A couple decades later, I find myself asking a related question, which is, how much data science and machine learning is required of modern informatics professionals? Clearly data science, machine learning, artificial intelligence, etc. are very prominent now in the evolution healthcare and biomedical science. But not everyone needs to be a "deep diver" into data science and machine learning. I often point this out by referring to the data analytics workforce reports from a few years ago that note the need for a five- to ten-fold ring of people who identify the needs, put into practice, and communicate the results of the deep divers [1, 2]. I also note the observation of data analytics thought leader Tom Davenport, who has written the importance of the roles of "light quants" or "analytical translators" in data-driven organizations (such as healthcare)[3].

Thus to answer my question in the title of this post, competence in data science and machine learning may be analogous to the answer to the computer science question of a couple decades ago. Clearly, every informatician must have basic data science skills. These include knowing how to gather, wrangle, and carry out basic analysis of data. They should understand the different approaches to machine learning, even if they do not necessarily understand all of their deep mathematics. And of course they must critically know how to apply data science and machine learning in their everyday professional practice of informatics.

References

1. Manyika, J, Chui, M, et al. (2011). Big data: The next frontier for innovation, competition, and productivity, McKinsey Global Institute. http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation.
2. Anonymous (2014). IDC Reveals Worldwide Big Data and Analytics Predictions for 2015. Framingham, MA, International Data Corporation. http://bit.ly/IDCBigDataFutureScape2015.
3. Davenport, T (2015). In praise of “light quants” and “analytical translators”. Deloitte Insights. https://www2.deloitte.com/insights/us/en/topics/analytics/new-big-data-analytics-skills.html.

Wednesday, September 18, 2019

Closing the Loops on Data Science and Informatics

One of the most highly viewed posts of this blog is a 2015 posting, What is the Difference (If Any) Between Informatics and Data Science. One critique I have had of data science is the focus of most work on only showing prediction and not implementing prescription. In other words, how do we take the predictive output in an ever-increasing number of areas of biomedicine and turn it into programs that actually improve outcomes, whether better patient care, improved healthcare delivery, or more effective research? Some recent publications bring this issue to light and show that we have some loops to close before we attain the value of data science in biomedicine.

A couple of recent perspective pieces bring this closure into light. One is from colleagues Philip Payne, Elmer Bernstam, and Justin Starren [1]. In a Perspective last year in JAMIA Open, they put forth a model that delineates the loop that must be closed, from the development of data science (and informatics) models and systems to the real-world informatics that most who work in the field are familiar with of implementing and evaluating systems with real users and organizations. A more recent paper from Lenert et al. notes that as predictive models are put into place and impact outcomes, they will necessarily impact those models, which will need to be adjusted to the new reality of their use [2].

One aspect of this first loop to be closed is how we study data science and machine learning interventions in actual clinical practice. A pair of recently published papers demonstrate how models and systems can be built and validated, and then assessed in the clinical real world. A first paper by Barton et al. develops and evaluates a model for predicting sepsis from patient vital designs [3]. Sepsis is a medical problem of continued significance while vital sign data is readily available. A subsequent paper by Shimabukuro et al. implements a randomized controlled trial in two medical intensive care units, finding a decrease in length of stay in the units from 13.0 to 10.3 days and a 12.4% reduction in in-hospital mortality [4].

Another recent study assessed the application of machine learning to detecting colonic polyps during colonoscopy [5]. While the machine learning system worked effectively, it was mostly effective at recognizing polyps that were unlikely to progress to cancer quickly, such as small adenomas and hyperplastic polyps. Nonetheless, recognizing such polyps improves the overall quality of colonoscopy exam.

A second loop that will need to be closed to achieve the vision of widespread generalized application of data science will be the generation of standardized EHR data for use across the healthcare system. A group of colleagues and I wrote about this in 2013 [6], as have many others, but some recent work documents aspects of this problem are still not solved. Two recent analyses show variations in how physicians [7] and healthcare organizations [8] document patient care, which may lead to variation in data that is not due to underlying differences in patients.

The need to close these loops show we are still in the early days of machine learning and predictive algorithms. While their impact in medicine will likely be enormous in the long run, there is still much work that will need to be done to optimize their data and how they are most effectively used.

References

1. Payne P, Bernstam E, Starren J. Biomedical informatics meets data science: current state and future directions for interaction. JAMIA Open. 2018;1:136-41.
2. Lenert M, Matheny M, Walsh C. Prognostic models will be victims of their own success, unless. . . Journal of the American Medical Informatics Association. 2019; Epub ahead of print.
3. Barton C, Chettipally U, Zhou Y, Jiangce Z, Lynn-Palevsky A, Le S, et al. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Computers in Biology and Medicine. 2019;109:79-84.
4. Shimabukuro D, Barton C, Feldman M, Mataraso S, Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respiratory Research. 2019;4(1):e000234.
5. Wang P, Berzin T, Brown J, Bharadwa S, Becq A, Xiao X, et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut. 2019; Epub ahead of print.
6. Hersh W, Weiner M, Embi P, Logan J, Payne P, Bernstam E, et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical Care. 2013;51(Suppl 3):S30-S7.
7. Cohen G, Friedman C, Ryan A, Richardson C, Adler-Milstein J. Variation in physicians' electronic health record documentation and potential patient harm from that variation. Journal of General Internal Medicine. 2019; Epub ahead of print.
8. Glynn E, Hoffman M. Heterogeneity introduced by EHR system implementation in a de-identified data resource from 100 non-affiliated organizations. JAMIA Open. 2019; Epub ahead of print.

Tuesday, April 11, 2023

Learning Data Science and Machine Learning with Orange

I have written over the years that understanding data science (DS), machine learning (ML), artificial intelligence (AI), and related topics is essential for anyone who works in a profession where tools using them might be employed. This is certainly the case in biomedicine, both in clinical practice and research. In the same sense that all physicians should have basic knowledge of pharmacology, radiology, and other topics, even if they are not experts in the science and methods of those diverse disciplines, all in healthcare should be ready for ML and AI tools coming to their work.

Of course, learning to apply ML and AI historically has required a background in math and computer programming that most in healthcare do not have. Fortunately, there has been emergence in recent years of a number of "no-code" DS and ML tools. In some ways they are analogous to statistical packages, which allow anyone who knows how to use them load data and run statistical tests. Of course, that does not mean that the operator of statistical software will apply or interpret the results of their analyses correctly, which is especially important for large or complex data sets where appropriate statistical expertise is essential.

One no-code ML tool that I find particularly easy to understand and use is Orange Data Mining. Orange was developed by the University of Ljubljana in Slovenia and is an open-source project with a large community supporting it. It runs on Mac and Windows machines, for which binary packages can easily be downloaded and installed. Orange is programmed in Python and, as such, can be extended with Python code. It comes with sample data sets and allows easy loading of new ones. The original focus of Orange was for structured data and machine learning models for operating on them, including the main ML tasks of classification (e.g., logistic regression, kNN, and neural networks) and regression (e.g., linear regression, random forest, and neural networks). There are also methods for visualizing data and evaluating models. The latter includes techniques such as splitting training and test data and cross-fold evaluations are easy to configure. There are also various methods for imputation of missing data. In addition, modules have been developed for image analysis and text mining.

Orange is used by dragging widgets that represent data sets, visualization methods, models, and more on to a canvas called a workflow. Data flows between the widgets by connecting them via simple dragging between widgets. Once one becomes facile in dragging and connecting widgets, workflows can be built up quickly and easily. They can also be saved and reloaded later.

One can start with one of the built-in data sets or use the File widget to load one's own data. Most common rectangular data formats are accepted, such as CSV, tab-delimited, and Excel files. Included among the sample data sets is one for heart disease. The data itself is old (Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J., Sandhu, S., Guppy, K., Lee, S., & Froelicher, V. International application of a new probability algorithm for the diagnosis of coronary artery disease. American Journal of Cardiology, 1989, 64, 304-310.) and small (303 records with 13 variables), but the data set offers a good way to get started with Orange using clinical data. The outcome variable (called the target variable in Orange) is coronary artery diameter narrowing and the predictor variables include gender, chest pain, serum cholesterol, systolic blood pressure, and more.

The screenshot below shows a workflow I built using the heart disease data set. As in most data science analyses, one can begin with methods to explore and visualize the data. I connected a number of widgets that allow the data to be viewed in a data table and visualized in the form of feature statistics (also shown below), distributions, and box plots. I also created some models using the Test and Score widget connected to a number of different models appropriate for classification, in this case logistic regression, naive Bayes, and a neural network. One can view the classification accuracy, F1 score, and other evaluation variables (also shown below) and connect the results to widgets that compare ROC curves and generate confusion matrices for each model. (Perhaps not surprisingly, logistic regression works best!) Likewise, I created some models for regression with ST-segment elevation as the outcome variable, using linear regression, random forest, and a neural network, which can also be seen on the workflow.

It is fairly easy to get started using Orange. A set of video tutorials on the Orange YouTube channel covers the basics as well as a number of advanced topics. My colleague Dr. Robert Hoyt and his collaborator, Dr. David Pashtikoff, have been developing workshops and other learning materials using Orange, available at a Web site they have developed called, NoCode Data Science. They plan to publish a textbook, and setting up a login on the site allows access to a number of additional data sets that can be loaded into Orange.

Just as with statistical packages, just because one can load data into Orange and apply ML models does not mean that one is doing so correctly. On the other hand, my simple workflow just scratches the surface of what can be done with Orange, even for this tiny data set.

There are limitations of Orange as a data mining tool. I do not know how scalable it is as data sets get large (all data is kept in computer RAM), and of course any serious ML analysis requires more than the ability to just plug data into models. Nonetheless, Orange can be a great tool for learning machine learning without getting bogged down in coding, debugging, and the like. I see a great future for it in teaching ML as the reach of AI advances.

Friday, July 10, 2015

What is the Difference (If Any) Between Informatics and Data Science?

I am increasingly asked to describe the difference between data science and biomedical informatics. Distinguishing these disciplines takes on added importance with the recent publication of the NIH Advisory Committee to the Director, National Library of Medicine (NLM) Working Group, report on the future of the NLM, which calls for NLM to become a leader in data science at NIH. NLM has of course historically been a leader in research and training in biomedical informatics.

What is, if any, the difference between informatics and data science? Let me start with definitions. I have written my own definitions of biomedical informatics [1] but for the sake of the community, let me quote the latest consensus definition from our professional association, the American Medical Informatics Association (AMIA) [2]: "Biomedical informatics (BMI) is the interdisciplinary field that studies and pursues the effective uses of biomedical data, information, and knowledge for scientific inquiry, problem solving and decision making, motivated by efforts to improve human health."

How is data science defined? It is not as easy to find an "official" definition of data science, but a good starting point might be the definition from Wikipedia, which is the "extraction of knowledge from large volumes of data that are structured or unstructured." The Wikipedia article references that definition from a paper by Vasant Dhar [3] and a blog posting by Jeff Leek. A Google search also points out some highly-cited sources from O’Reilly & Associates Media and Forbes Magazine. The Forbes article quotes the famous information scientist Hal Varian, who has noted, "The ability to take data - to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it - that’s going to be a hugely important skill in the next decades." I myself have written that the core competencies of data science are statistics, especially machine learning; data-oriented computer programming, especially of querying databases; domain understanding where the analysis and interpretation will be applied; business processes; and communications.

I believe that whether data science is distinct from, partially overlapping with, or a subset of informatics is defined by how broadly one defines informatics. I tend to take a very broad definition of informatics, because I understand that its "sociotechnical" nature [4] covers many facets of data, information, and knowledge, including their technological as well as social context. Informatics recognizes that many aspects of data influence its use, aggregation, and interpretation. I have expressed concern in the past that data scientists need to understand research methodology, in particular how we distinguish cause-and-effect from correlation.

One possible way to answer the question of distinction between data science and informatics is to think about areas of informatics that are not ordinarily considered part of data science. I can think of (at least) several. One is usability. We know through the recent massive adoption of the electronic health record EHR) that there are significant usability challenges of current EHRs [5]. These not only adversely impact workflow, another important informatics topic, but may compromise safety. Another area of informatics we have come to recognize as critical is adherence to standards so that we may achieve better system interoperability. Finally, we also know that informatics is riddled with challenging "people and organizational issues" as information systems profoundly impact healthcare and individual health in many ways [6].

There is no question that what we can do with data is important for informatics, larger healthcare, and society as a whole. Informatics has recognized this for decades, but it also knows that there is much context beyond the data itself, and to this end, we are best served by viewing data science as a proper subset of informatics, certainly in the biomedical and health domain.

References

1. Hersh, W (2009). A stimulus to define informatics and health information technology. BMC Medical Informatics & Decision Making. 9: 24. http://www.biomedcentral.com/1472-6947/9/24/.
2. Kulikowski, CA, Shortliffe, EH, et al. (2012). AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. Journal of the American Medical Informatics Association. 19: 931-938.
3. Dhar, V (2013). Data science and prediction. Communications of the ACM. 56(12): 64-73.
4. Coiera, E (2007). Putting the technical back into socio-technical systems research. International Journal of Medical Informatics. 76(Supp 1): 98-103.
5. Zhang, J and Walji, M, Eds. (2014). Better EHR - Usability, workflow & cognitive support in electronic health records. Houston, TX, National Center for Cognitive Informatics & Decision Making in Healthcare.
6. Ash, JS, Berg, M, et al. (2004). Some unintended consequences of information technology in health care: the nature of patient care information system related errors. Journal of the American Medical Informatics Association. 11: 104-112.

Monday, May 22, 2023

It's All Informatics to Me: Data Analytics, Data Science, Digital Health, AI, and Whatever is Next

In the last decade or so, a number of new names have emerged for the kinds of activities historically studied, disseminated, and taught by those who work in biomedical and health informatics. Each of these areas has emerged as a "hot topic" in biomedicine and healthcare, with resulting academic funding opportunities, new courses or degree programs, and even academic centers or institutes bearing their names.

I suppose I have some skin in this game because I have spent my life's work among those developing the field that I consider to go by the over-arching term of this all, biomedical and health informatics. My concern for the new areas and their names has been their often ignoring the historical and current work of informatics, and with it the decades of research, development, implementation, and evaluation of methods and tools that aim to improve health and healthcare through the use of data and information.

Another ramification of this proliferation of terms is that those not directly working in the field, who may include the leaders of academic and other institutions, may not appreciate the historic role of informatics. Thus I write this post to provide my elaboration of the problem.

One early new term that started this trend about a decade ago was data analytics. As the growth of large amounts of digital data started with the growth of the Internet, the large-scale implementation of electronic health records, and the emergence of wearable and mobile devices, there were new sources of data that could be analyzed for interesting discoveries. From this started the proliferation of academic courses, certificate/degree programs, and centers/institutes devoted to healthcare (and related areas) data analytics.

With the proliferation of machine learning, deep learning, and other methods emerged the new discipline of data science. Again, this was followed by the proliferation of academic courses, certificate/degree programs, and centers/institutes focused on data science. One thoughtful perspective of the relationship between informatics and data science was a paper by Payne et al. in JAMIA Open in 2018, which showed there was not only substantial overlap but also aspects of informatics that go beyond data science. Infomatics includes implementation, evaluation, human-computer interaction, and workflow, to name a few. To use the language of the paper, these were the "efferent" processes of applying knowledge discovered from the "afferent" processes of data science (and informatics) to the real world.

In the meantime, especially with the growth of telehealth during the COVID-19 pandemic, there emerged a new term, digital health. Similar to data science, there was substantial overlap between digital health and informatics, with that overlap focused on many on the efferent processes of Payne et al. However, for many, the work of digital health is really no different than what informatics has historically done and continues to do.

And now there is the new term on the block, which is actually not a new term at all but a re-emergence of an area that was the focus of informatics a half-century ago. This term is of course artificial intelligence (AI). Clearly the advances of modern AI are very real and important. The new AI will likely have profound impacts on biomedicine and health, although we also need to see real-world implementation and evaluation to know what truly works.

My goal here is not to discount any of the important advances that have emerged under the guise of data analytics, data science, digital health, or AI. But rather, to express my concern that plenty of people tout these new disciplines sometimes with ignorance of the historic role that informatics has played in them all. I understand that sometimes funders, institutional leaders, and others want to put their resources into things that are "new." But I also am disappointed when these new areas re-invent the methods and tools of informatics.

The best of informatics imposes a rigor of thinking that these newer disciplines not always apply. Informatics is driven by problems in the domains of health, healthcare, public health, and clinical research. Health and clinical realism is imposed by informatics on the systems as they develop, implemented, and evaluate. I hope that these new areas and what emerges next will not ignore the lessons learned by informatics and incorporate them into their work.

Wednesday, September 12, 2018

Artificial Intelligence in Medicine: 21st Century Resurgence

I first entered the informatics field in the late 1980s, at the tail end of the first era of artificial intelligence (AI) in medicine. Initial systems focused on making medical diagnoses using symbolic processing, which was appropriate for a time of relatively little digital data, both for individual patients and healthcare as whole, and underpowered hardware. Systems like MYCIN [1], INTERNIST-1/QMR [2], and DXPLAIN [3] provided relatively accurate diagnostic performance, but were slow and difficult to use. They also provided a single likely diagnosis, which was not really what clinicians needed. Because of these shortcomings, they never achieved significant real-world adoption, and their "Greek Oracle" style of approach was abandoned. [4]. There was also some early enthusiasm for neural networks around that time [5], although in retrospect those systems were hampered by lack of data and computing power.

Into the 1990s, informatics moved on to other areas, such as information retrieval (search) from the newly evolving World Wide Web and more focused (rule-based) decision support. At the start of the new century, I started to wonder whether I should still even cover those early AI systems in my well-known introductory informatics course. I kept them included, mainly out of a sense of historical perspective, since those systems were a major focus of work in the field in its early days. However, the term "AI" almost seemed to disappear from informatics jargon.

In recent years, however, AI in medicine (and beyond) has re-emerged. Driven by much larger quantities of data (through electronic health records, curated data sets - mainly images, and personal tracking devices) and much more powerful hardware (mainly networked clusters of low-cost computers and hard disks as well as mobile devices), there has been a resurgence of AI, although with a somewhat different focus from the original era. There has also been a maturing of machine learning techniques, most prominently neural networks applied in complex formats known as deep learning [6, 7].

The most success for use of deep learning has come in image processing. The well-known researcher and author Dr. Eric Topol keeps an ever-growing list of systems for diagnosis and their comparison with humans (to which I have contributed a few, and to which I add studies that have only been published as preprints on bioArXiv.org):
  • Radiology - diagnosis comparable to radiologists for pneumonia [8] tuberculosis [9], intracranial hemorrhage [10]
  • Dermatology - detecting skin cancer from images [11-13]
  • Ophthalmology - detecting diabetic retinopathy from fundal images [14-15], predicting cardiovascular risk factors from retinal fundus photographs [16]; diagnosis of congenital cataract [17], age-related macular degeneration [18], plus disease [19]; and diagnoses of retinal diseases [20] and macular diseases [21]
  • Pathology - classifying various forms of cancer from histopathology images [22-25], detecting lymph node metastases [26]
  • Cardiology - cardiac arrhythmia detection comparable to cardiologists [27] and classification of views in echocardiography [28]
  • Gastroenterology - endocytoscope images for diagnose-and-leave strategy for diminutive, nonneoplastic, rectosigmoid polyps [29]
Organized medicine has taken notice of AI. Journal of the American Medical Association recently published two perspective pieces [30, 31] as well as editorial [32] on how AI and machine learning will impact medical practice. I have heard anecdotally that some of the most heavily attended sessions at radiology meetings are those devoted to AI. I am sure there is a mixture of intellectual excitement tinged with some fear of future livelihood.

The success of these systems and the technology underlying them are exciting, but I also would tell any thoughtful radiologist (or pathologist, dermatologist, or ophthalmologist) not to fear for his or her livelihood. Yes these tools will change practice, maybe sooner than we realize. However, I always think that high-tech medicine of the future will look like how it is used the doctors of Star Trek. Yes, those physicians have immense technology at their disposal, not only for diagnosis but also for treatment. But those tools do not remove the human element of caring for people. Explaining to patients their disease process, describing the prognosis as we know it, and shared decision-making among the diagnostic and treatment options are all important in applying advanced technology is medicine.

I also recognize we have a ways to go before this technology truly changes medicine. For several years running, I have expressed both my intellectual excitement at predictive data science while also noting that prediction is not enough, and we must demonstrate that what is predicted must be demonstrated to be able to be applied to improve the delivery of care and patient health.

This notion is best elaborated by some discussion of another deep learning paper focused on a non-image domain, namely the prediction of in-hospital mortality, 30-day unplanned readmission, prolonged length of stay, and the entirety of a patient’s final diagnoses [33]. The paper demonstrates the value of deep learning, the application of Fast Healthcare Interoperability Resources (FHIR) for data points, and efforts for the neural network to explain itself along its processing path. I do not doubt the veracity of what the authors have accomplished. Clearly, deep learning techniques will play a significant role as described above. These methods scale with large quantities of data and will likely improve over time with even better algorithms and better data.

But taking off my computer science hat and replacing it with my informatics one, I have a couple of concerns. My first and major concern is whether this prediction can be turned into information that can improve patient outcomes. Just because we can predict mortality or prolonged length of stay, does that mean we can do anything about it? Second, while there is value to predicting across the entire population of patients, it would be interesting to focus in on patients we know are more likely to need closer attention. Can we focus in and intervene for those patients who matter?

Dr. Topol recently co-authored an accompanying editorial describing a study that adheres to the kind of methods that are truly needed to evaluate modern AI in clinical settings [34]. The study itself is to be commended; it actually tests an application of an AI system for detection of diabetic retinopathy in primary care settings [35]. The system worked effectively, though it was not flawless, and other issues common to real-world medicine emerged, such as some patients being non-imageable and others having different eye diseases. Nonetheless, I agree with Dr. Topol that this study sets the bar for how AI needs to be evaluated before its widespread adoption in routine clinical practice.

All of this AI in medicine research is impressive. But its advocates will need to continue the perhaps more mundane research of how we make this data actionable and actually act on it in ways that improve patient outcomes. I personally find that kind of research more interesting and exciting anyways.

References

1. Miller, RA (2010). A history of the INTERNIST-1 and Quick Medical Reference (QMR) computer-assisted diagnosis projects, with lessons learned. Yearbook of Medical Informatics. Stuttgart, Germany: 121-136.
2. Shortliffe, EH, Davis, R, et al. (1975). Computer-based consultations in clinical therapeutics: explanation and rule acquisition capabilities of the MYCIN system. Computers and Biomedical Research. 8: 303-320.
3. Barnett, GO, Cimino, JJ, et al. (1987). DXplain: an evolving diagnostic decision-support system. Journal of the American Medical Association. 258: 67-74.
4. Miller, RA and Masarie, FE (1990). The demise of the "Greek Oracle" model for medical diagnostic systems. Methods of Information in Medicine. 29: 1-2.
5. Rumelhart, DE and McClelland, JL (1986). Parallel Distributed Processing: Foundations. Cambridge, MA, MIT Press.
6. Alpaydin, E (2016). Machine Learning: The New AI. Cambridge, MA, MIT Press.
7. Kelleher, JD and Tierney, B (2018). Data Science. Cambridge, MA, MIT Press.
8. Rajpurkar, P, Irvin, J, et al. (2017). CheXNet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv.org: arXiv:1711.05225. https://arxiv.org/abs/1711.05225.
9. Lakhani, P and Sundaram, B (2017). Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 284: 574-582.
10. Arbabshirani, MR, Fornwalt, BK, et al. (2018). Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. npj Digital Medicine. 1: 9. https://www.nature.com/articles/s41746-017-0015-z.
11. Esteva, A, Kuprel, B, et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature. 542: 115-118.
12. Haenssle, HA, Fink, C, et al. (2018). Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology. 29: 1836-1842.
13. Han, SS, Kim, MS, et al. (2018). Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. Journal of Investigative Dermatology. 138: 1529-1538.
14. Gulshan, V, Peng, L, et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Journal of the American Medical Association. 316: 2402-2410.
15. Ting, DSW, Cheung, CYL, et al. (2017). Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. Journal of the American Medical Association. 318: 2211-2223.
16. Poplin, R, Varadarajan, AV, et al. (2017). Predicting cardiovascular risk factors from retinal fundus photographs using deep learning. arXiv.org: arXiv:1708.09843. https://arxiv.org/abs/1708.09843.
17. Long, E, Lin, H, et al. (2017). An artificial intelligence platform for the multihospital collaborative management of congenital cataracts. Nature Biomedical Engineering. 1: 0024. https://www.nature.com/articles/s41551-016-0024.
18. Burlina, PM, Joshi, N, et al. (2017). Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmology. 135: 1170-1176.
19. Brown, JM, Campbell, JP, et al. (2018). Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmology. 136: 803-810.
20. DeFauw, J, Ledsam, JR, et al. (2018). Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine: Epub ahead of print. https://www.nature.com/articles/s41591-018-0107-6.
21. Kermany, DS, Goldbaum, M, et al. (2018). Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 172: 1122-1131.E1129.
22. Bejnordi, BE, Zuidhof, G, et al. (2017). Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. Journal of Medical Imaging. 4(4): 044504. https://www.spiedigitallibrary.org/journals/journal-of-medical-imaging/volume-4/issue-04/044504/Context-aware-stacked-convolutional-neural-networks-for-classification-of-breast/10.1117/1.JMI.4.4.044504.full?SSO=1.
23. Liu, Y, Gadepalli, K, et al. (2017). Detecting cancer metastases on gigapixel pathology images. arXiv.org: arXiv:1703.02442. https://arxiv.org/abs/1703.02442.
24. Yu, KH, Zhang, C, et al. (2017). Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nature Communications. 7: 12474. https://www.nature.com/articles/ncomms12474.
25. Capper, D, Jones, DTW, et al. (2018). DNA methylation-based classification of central nervous system tumours. Nature. 555: 469–474.
26. Bejnordi, BE, Veta, M, et al. (2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Journal of the American Medical Association. 318: 2199-2210.
27. Rajpurkar, P, Hannun, AY, et al. (2017). Cardiologist-level arrhythmia detection with convolutional neural networks. arXiv.org: arXiv:1707.01836. https://arxiv.org/abs/1707.01836.
28. Madani, A, Arnaout, R, et al. (2018). Fast and accurate view classification of echocardiograms using deep learning. npj Digital Medicine. 1: 6. https://www.nature.com/articles/s41746-017-0013-1.
29. Mori, Y, Kudo, SE, et al. (2018). Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Annals of Internal Medicine: Epub ahead of print.
30. Hinton, G (2018). Deep learning—a technology with the potential to transform health care. Journal of the American Medical Association: Epub ahead of print.
31. Naylor, CD (2018). On the prospects for a (deep) learning health care system. Journal of the American Medical Association: Epub ahead of print.
32. Stead, WW (2018). Clinical implications and challenges of artificial intelligence and deep learning. Journal of the American Medical Association: Epub ahead of print.
33. Rajkomar, A, Oren, E, et al. (2018). Scalable and accurate deep learning for electronic health records. npj Digital Medicine. 1: 18. https://www.nature.com/articles/s41746-018-0029-133.
34. Keane, PA and Topol, EJ (2018). With an eye to AI and autonomous diagnosis. npj Digital Medicine. 1: 40. https://www.nature.com/articles/s41746-018-0048-y.
35. Abràmoff, MD, Lavin, PT, et al. (2018). Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. npj Digital Medicine. 1: 39. https://www.nature.com/articles/s41746-018-0040-6.

Monday, December 19, 2022

Kudos for the Informatics Professor - 2022 Update

The year 2022 was a highly productive one for the Informatics Professor, and I am pleased to provide my periodic update of my recent accomplishments. This year had another big event in my decision to step down as the one and only Chair of the Oregon Health & Science University (OHSU) Department of Medical Informatics & Clinical Epidemiology. More will be posted about that when my successor is named. Meanwhile, here are my accomplishments for 2022.

I received two prominent awards this year. One was from the American Medical Informatics Association (AMIA), my main professional association, for the William Stead Award for Thought Leadership in Informatics. This award "acknowledges people who have influenced our thinking about informatics, especially improving health and health care in ways that are visionary and transformative." This year I was also elected to the second class of the ACM SIGIR Academy, which was established to "honor and recognize individuals who have made significant, cumulative contributions to the development of the field of information retrieval (IR)." I am thrilled to be honored by the two fields of which I work at the intersection between them.

This year I completed my term as the Second President of the International Academy of Health Sciences Informatics (IAHSI). I will continue for another two years as Past President.

In 2022, I was awarded several new grants, which will help sustain the research and teaching that I plan to continue as I reduce my administrative and leadership roles. Probably the most important of these was the OHSU Training Grant in Biomedical Informatics and Data Science, which was renewed for another five year cycle through 2027. This training grant from the National Library of Medicine (NLM) of the National Institutes of Health (NIH) funds predoctoral (PhD) and postdoctoral trainees in the OHSU Biomedical Informatics Graduate Program. This grant has been funded continuously at OHSU since 1992. This renewal marks the seventh consecutive five-year cycle of funding for OHSU. Its first 30 years make it the second-longest-running training grant at OHSU. The grant's total funding over that time of $21,912,538 make it the largest training grant ever at OHSU. I myself am a product of this training grant, completing my postdoctoral fellowship at Harvard University from 1987-1990.

Another grant from the NLM for which I am PI is an R25 grant to fund the OHSU Biomedical Informatics and Data Science College Undergraduate Summer Internship Program for five years. The grant is among 12 awards issued by the National Library of Medicine to prepare historically underrepresented students for biomedical informatics and data science graduate studies and research careers. The goal of the our program is to provide a rich, full-time experience to students to graduate study and careers in biomedical informatics and data science. In addition, the program aims to motivate students to pursue doctoral training in biomedical informatics and data science that leads to research careers in academia, government, healthcare, and industry. The funding from the R25 grant will be used to support summer experiences for students with backgrounds that have been historically underrepresented in biomedical informatics and data science.

A final new grant for 2022 was from the new NIH Bridge2AI Initiative. I am among a group of faculty from OHSU and Washington University who will be serving as the Skills and Workforce Development Module for one of the four funded data-acquisition projects. The project we are part of will focus on creating data sets that aim to use voice as a biomarker of health.

I also had a good year of publications in 2022, not the least of which was the release of the 8th edition of Health Informatics: Practical Guide, in which I serve as Editor and have authored about a third of the chapters. My journal publications for 2022 included the following:

In 2022, I was also a guest on a couple of podcasts. One was to provide an introduction to an episode on of the For Your Informatics podcast on the gender pay gap among physicians, including those in informatics. The other was to be interviewed for the IAHSI podcast on Exploring Healthcare Interoperability about my work in teaching about healthcare data standards and interoperability.

I also gave a number of invited talks in 2022:

  • Biomedical and Health Informatics: Impact, Challenges, and Opportunities, Nevada Institute of Personalized Medicine, University of Nevada Las Vegas (virtual), January 27, 2022 (slides and references)
  • Competencies and Curricula Across the Spectrum of Learners for Health Informatics, Keynote Talk, IEEE International Conference on Healthcare Informatics, First International Workshop on Health Informatics Education, Rochester, MN, June 11, 2022 (slides and references)
  • Translational Artificial Intelligence (AI): The Need to Translate from Basic Science to Clinical Value, University of Alabama Birmingham Informatics Institute (virtual), September 2, 2022 (slides and references and video)
  • Past, Present, and Future: A Discussion About DMICE Going Forward, OHSU Department of Medical Informatics & Clinical Epidemiology, October 6, 2022 (slides and video)
  • Applying Information Retrieval to the Electronic Health Record for Cohort Discovery and Rare Disease Detection, Brown University Center for Biomedical Informatics, Providence, RI, October 13, 2022 (slides and references)
  • Competencies and Curricula Across the Spectrum of Learners for Biomedical and Health Informatics, International Symposium on Achievements, Milestones and Challenges in Biomedical and Health Informatics, Athens, Greece, October 29, 2002 (slides)

In 2022 I also delivered my annual lecture to first-year OHSU medical students, Information is Different Now That You're a Doctor, on September 2, 2022. I also gave this talk to first-year medical students at Nova Southeastern University (virtual) on June 16, 2022.

I additionally attended my 37th consecutive AMIA Annual Symposium, where I participated in three activities:

  • Panel: Beyond Wrangling and Modeling: Data Science and Machine Learning Competencies and Curricula for The Rest of Us
  • Job Talk Panel for the AMIA Clinical Informatics Fellows (ACIF)
  • Debate: Extension of the Practice Pathway for ABMS Clinical Informatics Board Certification for Physicians in the United States

This year I also served as Co-Chair of the Scientific Program Committee of the conference hosted by Brown University, The Patient and The Practitioner in the Age of Technology: Promoting Healing Relationships.

In 2022, I also maintained my various scientific citation rankings. In the list of Best Computer Science Scientists, I now rank 694th in the US and 1183rd in the world based on a discipline-specific H-index they calculate. I also maintain a presence in the latest iteration of a database of the top 100,000 cited scientists in the world for 22 scientific fields and 176 subfields, where I rank 22,553rd based on a composite formula of citation and publishing variables. In the medical informatics subfield, I rank 13th. Of the 290 individuals affiliated with OHSU, I rank 50th. I also maintain my presence on ranking lists based on the Google Scholar h-index for biomedical informatics and information retrieval.

Finally, this blog is mentioned in various lists of best blogs, including the following:

Thursday, April 11, 2019

Beyond Images and Waves: How Will Deep Learning Benefit Health or Healthcare?


Scarcely a week goes by without another study published of a deep learning algorithm that achieves accuracy comparable to or sometimes better than human experts. The vast majority of these studies focus on some application to diagnostic imaging or waveform interpretation. There is no question that the practice of medicine will be highly impacted by these systems, especially for practitioners in fields that directly use them, such as radiologists, pathologists, dermatologists, and so forth.

What about deep learning applied to other areas of medicine? One group of studies has applied deep learning to retrospective electronic health record (EHR) data. A number of studies have shown impressive abilities to use EHR data to predict or diagnose:
  • Several dozen diseases [1]
  • Length of stay, mortality, readmission, and diagnosis at two large medical centers [2]
  • Prognosis in palliative care [3]
  • 30-day readmission in heart failure [4]
  • Patient mortality from coronary artery disease more accurately than traditional cardiovascular risk models [5]
  • Early risk of chronic kidney disease in patients with diabetes [6]
  • Many pediatric diagnoses at a major referral center [7]
  • Clinical outcomes in rheumatoid arthritis [8]
One can see obvious use cases for these types of systems, such as being able to intervene when patients might fare poorly when hospitalized or are at risk for readmission or more serious outcomes after they are discharged. But making a faster or more accurate diagnosis from an imaging or waveform study is a different matter than trying to determine the best use of an algorithm that tells a clinical team that a patient may be heading toward a bad outcome. When do we apply the results of the system? Once we intervene, does this change the nature of future recommendations? These are fascinating research questions, but also big unknowns in terms of how to apply such data in the clinical setting.

I wrote back in 2014, and more recently in 2017, that these systems must go beyond their ability to predict and actually be used in the context of prescriptive systems that results in better outcomes for the patient and/or the healthcare system. I hope to see studies going forward where these sorts of systems become part of the patient care team, and lead to demonstration of Friedman’s Fundamental Theorem of Informatics [9], which is that humans aided by machines do better than humans or machines alone. A nice roadmap for such studies was recently published that notes the need for studies that include meaningful endpoints, appropriate benchmarks, transportable to other settings and systems, and including legal and ethical monitoring [10].

This sentiment is echoed in the excellent new book by Dr. Eric Topol, Deep Medicine [11]. Dr. Topol raises the notion of deep learning benefitting patient care in even additional ways. One could solve a problem that has vexed healthcare with the widespread adoption of EHRs, which is the introduction of a third entity into the patient-physician encounter, namely the computer. Clinicians now unfortunately spend too much time “feeding the beast,” especially when that beast distracts from the patient and has additional compliance and billing burdens, contributing heavily to our epidemic of burnout in clinicians [12].

Perhaps one area where deep learning might help reduce the clinician burden is in data entry. An intriguing new study was recently published by one of the same authors who contributed some of the EHR suites above, automatically charting symptoms from patient-physician conversations [13]. One also wonders whether the level of a patient visit for billing purposes, currently determined by the presence of various elements document in the medical record, could be replaced by other data easy to collect in the modern medical office, such as time spent with the patient, time with other aspects of care, and other requirements of care. While we may never be able to achieve a “computer-free” patient examination room, we will hopefully find ways to reduce its impact and burden.

Dr. Topol suggests a second major area of benefit for deep learning, which is monitoring patients in much more depth than our current approach to providing episodic healthcare. Of course, this must be done in ways that provide actionable information presented in ways that do not further overburden clinicians. But it is not beyond the pale to envision his view of algorithms the coalescence of -omics, personal sensors, physiological measurements, healthcare, and public health data coming together to give diagnostic, therapeutic, and prognostic advice. There are still many issues around cost, ethics, practicality, and so forth, so the benefits will need to outweigh the risks.

Thus, while we will likely see machine assistance in diagnosis from images and wave forms in the near future, the impact of what deep learning may do with patient data in the EHR and with the patient will likely take longer. I see exciting opportunities for research and development focus on how to prospectively determine how such systems fit into the workflow of patient interaction within and outside the healthcare system. A nice overview

References
1. Miotto, R, Li, L, et al. (2016). Deep Patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific Reports. 2016: 26094. https://www.nature.com/articles/srep26094. 
2. Rajkomar, A, Oren, E, et al. (2018). Scalable and accurate deep learning for electronic health records. npj Digital Medicine. 1: 18. https://www.nature.com/articles/s41746-018-0029-1. 
3. Avati, A, Jung, K, et al. (2018). Improving palliative care with deep learning. BMC Medical Informatics & Decision Making. 18: 122. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-018-0677-8. 
4. Golas, SB, Shibahara, T, et al. (2018). A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: a retrospective analysis of electronic medical records data. BMC Medical Informatics & Decision Making. 18: 44. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-018-0620-z. 
5. Steele, AJ, Denaxas, SC, et al. (2018). Machine learning models in electronic health records can outperform conventional survival models for predicting patient mortality in coronary artery disease. PLoS ONE. 13(8): e0202344. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0202344. 
6. Ravizza, S, Huschto, T, et al. (2019). Predicting the early risk of chronic kidney disease in patients with diabetes using real-world data. Nature Medicine. 25: 57-59.
7. Liang, H, Tsui, BY, et al. (2019). Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nature Medicine. 25: 433-438.
8. Norgeot, B, Glicksberg, BS, et al. (2018). Assessment of a deep learning model based on electronic health record data to forecast clinical outcomes in patients with rheumatoid arthritis. JAMA Network Open. 2(3): e190606. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2728001. 
9. Friedman, CP (2009). A 'fundamental theorem' of biomedical informatics. Journal of the American Medical Informatics Association. 16: 169-170.
10. Parikh, RB, Obermeyer, Z, et al. (2019). Regulation of predictive analytics in medicine. Science. 363: 810-812.
11. Topol, E (2019). Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. New York, NY, Basic Books.
12. Gardner, RL, Cooper, E, et al. (2019). Physician stress and burnout: the impact of health information technology. Journal of the American Medical Informatics Association. 26: 106-114.
13. Rajkomar, A, Kannan, A, et al. (2019). Automatically charting symptoms from patient-physician conversations using machine learning. JAMA Internal Medicine. Epub ahead of print.

Sunday, February 24, 2013

Data Mining Systems Improve Cost and Quality of Healthcare - Or Do They?

Several email lists I am on were abuzz last week about the publication of a paper that was described in a press release from Indiana University to demonstrate that "machine learning -- the same computer science discipline that helped create voice recognition systems, self-driving cars and credit card fraud detection systems -- can drastically improve both the cost and quality of health care in the United States." The press release referred to a study published by an Indiana faculty member in the journal, Artificial Intelligence in Medicine [1].

While I am a proponent of computer applications that aim to improve the quality and cost of healthcare, I also believe we must be careful about the claims being made for them, especially those derived from results from scientific research.

After reading and analyzing the paper, I am skeptical of the claims made not only by the press release but also by the authors themselves. My concern is less about their research methods, although I have some serious qualms about them I will describe below, but more so with the press release that was issued by their university public relations office. Furthermore, as always seems to happen when technology is hyped, the press release was picked up and echoed across the Internet, followed by the inevitable conflation of its findings. Sure enough, one high-profile blogger wrote, "physicians who used an AI framework to make patient care decisions had patient outcomes that were 50 percent better than physicians who did not use AI." It is clear from the paper that physicians did not actually use such a framework, which was only applied retrospectively to clinical data.

What exactly did the study show? Basically, the researchers obtained a small data set for one clinical condition in one institution's electronic health record and applied some complex data mining techniques to show that lower cost and better outcomes could be achieved by following the options suggested by the machine learning algorithm instead of what the clinicians actually did. The claim, therefore, is that if the data mining were followed by the clinicians instead of their own decision-making, then better and cheaper care would ensue.

As done in many scientific papers about technology, the paper goes into exquisite detail about the data mining algorithms and the experiments comparing them. But the paper unfortunately provides very little description about the clinical data itself. There is a reference to another paper from a conference that appears to describe the data set [2], but it is still not clear how the data was applied to evaluate the algorithms.

I have a number of methodological problems with the paper. First is the paucity of clinical details about the data. The authors refer to a metric called the "outcomes rating scale" of the "client-directed outcome informed (CDOI) assessment." No details are provided as to exactly what this scale measures or how differences in measurement correlate with improved clinical outcome. Furthermore, the variables of the details of care for the patient that the data mining algorithm supposedly outperforms are not described either. Therefore anyone hoping to understand the clinical value that this approach is claimed to have improved is not able to do so.

A second problem is that there is no discussion about the cost data or what cost perspective (e.g., system, clinician, societal, etc.) is taken. This is a common problem that plagues many studies in healthcare that attempt to measure costs [3]. Given the relatively modest amounts of money spent on the care that is reported in their results, amounting only to a few hundred dollars per patient, it is unlikely that the data includes the full amount of the costs of treatment for each patient, or over an appropriate time period. If my interpretation of the low value of the cost data is correct (which is difficult to discern from reading the paper due, again due to lack of details), the data do not include the cost of clinician time, facilities, or longer-term costs beyond the time frame of the data set. If that is indeed the case, then it would be particularly problematic for a machine learning system, since such systems make inferences limited only to the data that is provided to the model. Therefore if poor data is provided to the model, its "conclusions" are suspect. (This raises a side issue as to whether there is truly "artificial intelligence" here, since the only intelligence applied by the system is the models developed by their human creators.)

A third concern is that this is a modeling study. As every evaluation methodologist knows, modeling studies are limited in their ability to assign cause and effect. There is certainly a role in informatics science for modeling studies, although we saw recently that such studies have their limits, especially when revisited over the long run. In this study, there may have been reasons for the clinicians following the more expensive path or confounding reasons why such patients had worse outcomes, but they cannot be captured by the approach used in this study.

This is related to the final and most serious problem of the work, which is that the modeling evaluation is a very weak form of evidence to demonstrate the value of an intervention. If the authors truly wanted to show the benefits of the system and approach they developed, they should have performed a randomized controlled trial that compared their intervention with an appropriate control group. This would have led to the type of study that the blogger mentioned above erroneously described this to be. Such a study design would assess some of the more vexing problems we face in informatics, such as whether the advice coming from a computer will change clinician behavior. Or, when such systems are introduced into the "real world," whether the "advice" provided will prospectively lead to better outcomes.

I do believe that the kind of work addressed by this paper is important, especially as we move into the area of personalized medicine. As eloquently described by Stead and colleagues, healthcare will soon be reaching the point where the number of data points required for clinical decisions will exceed the bounds of human cognition [4]. (It probably already has.) Therefore clinicians will require aids to their cognition provided by information systems, perhaps one like that described in the study.

But such aids require, like everything else in medicine, robust evaluative research to demonstrate their value. The methods used in this paper may indeed be the methods to provide this value, but the implementation and evaluation described miss the mark. That miss is further exacerbated by the hype and conflation the ensued after the paper was published.

What can we learn from this paper and its ensuing hype? First, bold claims require bold evidence to back them up. In the case of showing value for an approach in healthcare - be it test, treatment, or informatics application - we must use evaluation methods that provide best evidence for the claim. That is not always a randomized controlled trial, but in this situation, it would be, and the modeling techniques used are really just preliminary data that (might) justify an actual clinical trial. Second, when we perform technology evaluation, we need to describe, and ideally release, all of the clinical data so that others can analyze and even replicate the results. Finally, while we all want to disseminate the results of our research to the widest possible audience, we need to be realistic in explaining what we accomplished and what are its larger implications.

References

[1] Bennett, C. and K. Hauser (2013). Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach. Artificial Intelligence in Medicine. Epub ahead of print.
[2] Bennett, C., T. Doub, A. Bragg, J. Luellen, C. VanRegenmorter, J. Lockman and R. Reiserer (2011). Data mining session-based patient reported outcomes (PROs) in a mental health setting: toward data-driven clinical decision support and personalized treatment. 2011 First IEEE International Conference on Healthcare Informatics, Imaging and Systems Biology (HISB 2011), San Jose, CA. 229-236.
[3] Drummond, M. and M. Sculpher (2005). Common methodological flaws in economic evaluations. Medical Care. 43(7 Suppl): 5-14.
[4] Stead, W., J. Searle, H. Fessler, J. Smith and E. Shortliffe (2011). Biomedical informatics: changing what physicians need to know and how they learn. Academic Medicine. 86: 429-434.

Wednesday, July 14, 2021

Translational Artificial Intelligence: A Grand Challenge for AI

The potential for artificial intelligence (AI) to transform biomedicine and health is immense, yet at this time that potential remains largely unfulfilled. As I have noted in this blog over the years, there have been many impressive achievements in applying AI to biomedical and health data using clean and well-curated data sets, yet its use in every day medical care or health pursuits is modest. I have noted the reasons for this gap over the years in various postings, namely in the need to move capabilities beyond prediction to the ability to take action on them and the importance of showing actual clinical value, whether in leading to better health outcomes or care delivery. Failure to achieve these will relegate AI to the same fate as its first generation in the last century.

The premiere biomedical research agency in the US, the National Institutes of Health (NIH), has recognized the need to advance the science of AI in biomedicine. As such, the NIH has launched a new initiative, Bridge to Artificial Intelligence (Bridge2AI), which aims to "propel biomedical research forward by setting the stage for widespread adoption of artificial intelligence (AI) that tackles complex biomedical challenges beyond human intuition."

I applaud this program, and hope it will address the larger picture of translating current advances in AI into algorithms and systems that truly lead to improved health outcomes and care delivery. Along the way, it will hopefully address other issues related to AI, such as deploying AI ethically, focusing on solving important problems, and developing human expertise, not only of those who implement and evaluate AI systems but also the clinicians who employ them in their professional practice and patients who understand the benefits and limitations.

The Bridge2AI program recently hosted a virtual workshop series. One of these was devoted to enumerating grand challenges for AI in biomedicine. Participants were allowed to suggest grand challenges related to any aspects of the problems or solutions. I submitted a grand challenge focused on what I perceive is the need for translational AI, i.e., building on the successes in the "basic science" of showing the predictive value of algorithms to studying and ultimately deploying evidence-based AI in the real world. My grand challenge was one of 25 accepted by the meeting organizers for presentation to workshop attendees. We were allowed to use one slide, which is shown below.

To reiterate from my slide and presentation, I noted that most AI and machine learning research is still at the "basic science" stage of the biomedical research pipeline. Advances have been made with clean and well-curated data sets in simulated settings. Just as drugs and devices must progress from the lab to patient care, AI advances must do the same. In other words, moving from prediction to prescription and providing value. There are all sorts of potential ethical issues that must be addressed in the translational activities, such as balancing privacy protection vs. the public good, identifying and eliminating bias in data and algorithms, and ensuring the resulting actions do not exacerbate groups that have historically been discriminated against in health and healthcare. It is also critical that the "people perspectives" are addressed to develop the expertise in scientific as well as sociotechnical issues, including patients/consumers, clinicians, and researchers. In particular, appropriate education must be provided for clinical users and leaders to maximize value and minimize harm from these approaches. I am hardly the only person to hold these views, and a number of others have articulated the importance of AI providing real-world value in its application in biomedicine and health.[1-3]

One concern I have for Bridge2AI is that the first round of funding opportunities is focused on data generation projects. Generating high-quality, ethically-sourced, and relevant data is of course necessary, but is not sufficient. In fact, I have a hard time aligning what is proposed in my grand challenge with this initial funding opportunity. I hope that Bridge2AI will broad the focus of research and usher in translational research and ultimately demonstrates its true potential to improve human health.

References

1. Allen, B., Agarwal, S., Kalpathy-Cramer, J., Dreyer, K., 2019. Democratizing AI. Journal of the American College of Radiology 16, 961–963. https://doi.org/10.1016/j.jacr.2019.04.023
2. Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V.X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., Ossorio, P.N., Thadaney-Israni, S., Goldenberg, A., 2019. Do no harm: a roadmap for responsible machine learning for health care. Nat. Med. 25, 1337–1340. https://doi.org/10.1038/s41591-019-0548-6
3. Faes, L., Liu, X., Wagner, S.K., Fu, D.J., Balaskas, K., Sim, D.A., Bachmann, L.M., Keane, P.A., Denniston, A.K., 2020. A Clinician’s Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies. Transl Vis Sci Technol 9, 7. https://doi.org/10.1167/tvst.9.2.7

Monday, June 6, 2016

Generalizability and Reproducibility of Scientific Literature and the Limits to Machine Learning

A couple years ago, some colleagues and I wrote a paper raising a number of caveats about the enthusiasm for leveraging the growing volume of patient data in electronic health records and other clinical information systems for so-called re-use of secondary use, such as clinical research, quality improvement, and public health [1]. While we shared that enthusiasm for that type of use, we also recognized some major challenges for trying to extract knowledge from these sources of data, and advocated a disciplined approach [2].

Now that the world’s knowledge is increasingly available in electronic form in online scientific papers and other resources, a growing number of researchers, companies, and others are calling for the same type of approach that will allow computers to process the world’s scientific literature to answer questions, give advice, and perform other tasks. Extracting knowledge from scientific literature may be easier than from medical records. After all, scientific literature is written in a way to report findings and conclusions in a relatively unambiguous manner. In addition, scientific writing is usually subject to copy-editing that decreases the likelihood of grammatical or spelling errors, both of which often make processing medical records more difficult.

There is no question that machine processing of literature can help answer many questions we have [3]. Google does an excellent job of answering questions I have about the time in various geographic locations, the status of current airplane flights, and calories or fat in a given food. But for more complex questions, such as the best treatment for a complex patient, we still have a ways to go.

Perhaps the system with the most hype around this sort of functionality is IBM’s Watson from. Recently, one of the early leaders of artificial intelligence research, Dr. Roger Schank, took IBM to task for its excessive claims (really marketing hype) around Watson [4]. Among the concerns Schank raised were IBM claims that Watson can “out-think” cancer. I too have written about Watson, in a posting to this blog now four years ago, in which I lamented the lack of published research describing its benefits (as opposed to hype pieces extolling its “graduation” of medical school) [5]. While there have been some conference abstracts presented on Watson’s work, we have yet to see any major contributions toward improving the diagnosis or treatment of cancer [6]. Like Schank, I find Watson’s technology interesting, but claims of its value in helping clinicians to treat cancer or other diseases need scientific verification as much as the underlying treatments being used.

We also, however, need to do some thought experiments as to how likely computers can carry out machine learning in this manner. In fact, there are many reasons why the published scientific literature must be approached with care. It has become clear in recent years that what is reported in the scientific literature may not reflect the totality of knowledge, but instead representing the “winner’s curse” of results that have been positive and thus more likely to be published [7,8]. In reality, however, “publication bias” pervades all of science [9].

In addition, further problems plague the scientific literature. It has been discovered in recent years that a good deal of scientific experiments are not reproducible. This was found to be quite prevalent in preclinical studies analyzed by pharmaceutical companies looking for promising drugs that might be candidates for commercial development [10]. It has also been demonstrated in psychology [11]. In a recent survey of scientists, over half agreed with the statement there is a “reproducibility crisis” in science, with 50-80% (depending on the field) unable to reduce an experiment yet very few trying or able to publish about it [12].

Even on the clinical side we know there are many problems with randomized controlled trials (RCTs). Some recent analyses have documented that RCTs do not always reflect the larger population from which the sampling is intended to represent [13,14], something we can now document with that growing quantity of EHR data [15].  Additional recent work has questioned the use of surrogate outcomes in cancer drugs, questioning their validity as indicators of efficacy of the drugs [16,17]. Indeed, it has been shown in many areas of medicine that initial studies are overturned with later, usually larger, studies [18-20].

In addition, the problem is not limited to published literature. A recent study was published that documented significant amounts of inaccuracy in drug compendia that are commonly used by clinicians [21].

My argument has always been that informatics interventions must prove their scientific mettle no differently than other interventions that claim to improve clinical practice, patient health, or other tasks for which we develop them. A few years back some colleagues and I raised some caveats about clinical data. For different reasons, there are also challenges with scientific literature as well. Thus we should be wary of system that claim to “ingest” scientific literature and perform machine learning from it. While it is important to continue this important area of research, we must resist efforts to over-hype it and also must carry out research to validate its success.

References

1. Hersh, WR, Weiner, MG, et al. (2013). Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical Care. 51(Suppl 3): S30-S37.
2. Hersh, WR, Cimino, JJ, et al. (2013). Recommendations for the use of operational electronic health record data in comparative effectiveness research. eGEMs (Generating Evidence & Methods to improve patient outcomes). 1: 14. http://repository.academyhealth.org/egems/vol1/iss1/14/.
3. Wright, A (2016). Reimagining search. Communications of the ACM. 59(6): 17-19.
4. Schank, R (2016). The fraudulent claims made by IBM about Watson and AI. They are not doing "cognitive computing" no matter how many times they say they are. Roger Schank. http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-and-AI.
5. Hersh, W (2013). What is a Thinking Informatician to Think of IBM's Watson? Informatics Professor. http://informaticsprofessor.blogspot.com/2013/06/what-is-thinking-informatician-to-think.html.
6. Kim, C (2015). How much has IBM’s Watson improved? Abstracts at 2015 ASCO. Health + Digital. http://healthplusdigital.chiweon.com/?p=83.
7. Ioannidis, JP (2005). Why most published research findings are false. PLoS Medicine. 2(8): e124. http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124.
8. Young, NS, Ioannidis, JP, et al. (2008). Why current publication practices may distort science. PLoS Medicine. 5(10): e201. http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050201.
9. Dwan, K, Gamble, C, et al. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS ONE. 8(7): e66844. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0066844.
10. Begley, CG and Ellis, LM (2012). Raise standards for preclinical cancer research. Nature. 483: 531-533.
11. Anonymous (2015). Estimating the reproducibility of psychological science. Science. 349: aac4716. http://science.sciencemag.org/content/349/6251/aac4716.
12. Baker, M (2016). 1,500 scientists lift the lid on reproducibility. Nature. 533: 452-454.
13. Prieto-Centurion, V, Rolle, AJ, et al. (2014). Multicenter study comparing case definitions used to identify patients with chronic obstructive pulmonary disease. American Journal of Respiratory and Critical Care Medicine. 190: 989-995.
14. Geifman, N and Butte, AJ (2016). Do cancer clinical trial populations truly represent cancer patients? A comparison of open clinical trials to the Cancer Genome Atlas. Pacific Symposium on Biocomputing, 309-320. http://www.worldscientific.com/doi/10.1142/9789814749411_0029.
15. Weng, C, Li, Y, et al. (2014). A distribution-based method for assessing the differences between clinical trial target populations and patient populations in electronic health records. Applied Clinical Informatics. 5: 463-479.
16. Prasad, V, Kim, C, et al. (2015). The strength of association between surrogate end points and survival in oncology: a systematic review of trial-level meta-analyses. JAMA Internal Medicine. 175: 1389-1398.
17. Kim, C and Prasad, V (2015). Strength of validation for surrogate end points used in the US Food and Drug Administration's approval of oncology drugs. Mayo Clinic Proceedings. Epub ahead of print.
18. Ioannidis, JP (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association. 294: 218-228.
19. Prasad, V, Vandross, A, et al. (2013). A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clinic Proceedings. 88: 790-798.
20. Prasad, VK and Cifu, AS (2015). Ending Medical Reversal: Improving Outcomes, Saving Lives. Baltimore, MD, Johns Hopkins University Press.
21. Randhawa, AS, Babalola, O, et al. (2016). A collaborative assessment among 11 pharmaceutical companies of misinformation in commonly used online drug information compendia. Annals of Pharmacotherapy. 50: 352-359.
22. Malin, JL (2013). Envisioning Watson as a rapid-learning system for oncology. Journal of Oncology Practice. 9: 155-157.