Stanford Webinar - Creating Fair, Useful, and Reliable AI in Healthcare

Bringing AI into Clinical Use

• Dr. Nigam Shaw is a professor of medicine at Stanford University and chief data scientist for Stanford Healthcare, with research focused on bringing AI into clinical use safely, ethically, and cost-effectively (14s).
• Dr. Shaw has an extensive background, including being an inventor on patents, authoring over 300 scientific publications, co-founding three companies, and being inducted into the American College of Medical Informatics and the American Society for Clinical Investigation (27s).
• The quality of AI and machine learning models in healthcare is heavily dependent on the quality of the data they are trained on, with data being collected from patient timelines (1m33s).
• Patient timelines are visualized as a series of data points collected over time, including ECGs, blood pressure, respiratory rate, cardiac output, medication orders, lab tests, and reports (2m7s).

Data Quality and Patient Timelines

• In a typical healthcare setting, not all data is collected at every point in time, and there is often a lack of longitudinal coverage for individual patients (2m46s).
• The manipulation and processing of patient timeline data have a significant impact on the performance of AI and machine learning models (3m22s).
• With large amounts of patient timeline data, models can be built to support decision-making in healthcare, including whether to treat a patient and how to treat a patient (4m1s).

Classification vs. Prediction vs. Recommendation

• The decision of whether to treat a patient can be broken down into classification or diagnosis tasks, or prediction tasks, such as prognosis (4m18s).
• The terms "prediction" and "classification" are often misused, with classification being the correct term when analyzing an image to determine if it contains a specific object or condition, such as pneumonia or a dog, as the outcome is already present (4m46s).
• In medicine, many things that claim to be predictions are actually classifications, such as sepsis predictors, which are actually figuring out if a patient has sepsis, and this distinction is important as it affects how the information is used (5m13s).
• The distinction between prediction and classification is crucial, as predicting an outcome may lead to attempts to prevent it, while classifying and diagnosing a condition leads to treatment, not prevention (5m30s).
• Recommendation is the hardest task, given the data's limitations and biases, and it has been a 40-year journey in medicine to figure out reliable recommendation (5m58s).
• There are three things that can be done with AI in healthcare: classification, prediction, and recommendation, and it's essential to consider whether these technical exercises are advancing the science of medicine, the practice of medicine, or the delivery of medical care (6m13s).

Advancing Science, Practice, and Delivery of Medical Care

• An example of advancing the science of medicine is the discovery of three subtypes of heart failure with preserved ejection fraction, which would be a classification (6m42s).
• Advancing the practice of medicine would involve developing a test to determine the subtype of heart failure and having a treatment available to target the specific subtype (6m58s).
• Advancing the delivery of medical care would involve implementing the test and treatment over time, resulting in improved patient outcomes, such as longer life, lower costs, and better quality of life (7m28s).

The Green Button Project and On-Demand Data Analysis

• The "Green Button Project" is an example of advancing the practice of medicine, where a simple query system was developed to help clinicians make decisions at the bedside by analyzing similar patient cases (8m1s).
• A bedside consultation service was created to provide written reports with recommendations for patient care, utilizing aggregated data from millions of patients to make better decisions (8m16s).
• Research has shown that medical evidence can be unreliable, with physicians often making decisions without prior published data, highlighting the need for on-demand data analysis (8m47s).
• A project was conducted to analyze data on demand, which was later scaled and shared through a company called Atropos Health, reducing the time it takes to conduct bedside studies from a day or two to under 24 hours (9m36s).
• The use of generative AI has further reduced the time it takes to conduct studies, allowing for on-demand analysis in a matter of minutes (10m9s).

Predictive AI and Cost Savings

• An example of simple AI being used to predict which patients will become medically costly in the future, and taking proactive action to enroll them in management programs, resulted in an estimated 10-15% cost savings without sacrificing quality (10m54s).
• AI and machine learning can be used to make various predictions, including operational, biological, and delivery-related predictions, such as predicting no-shows, classifying images, and deciding who to put on an air ambulance (11m53s).
• The AI model provides a risk estimate value, but the actual value comes from taking responsive action based on that estimate, such as early intervention or Advanced Care planning, depending on the specific case (12m28s).
• A three-star logo is used to remember the interplay between the model's risk estimate, the work capacity to follow through, and the action taken, with the goal of achieving net benefit (13m10s).
• Research has shown that about 25 papers and five to six faculty members have worked on studying this interplay, leading to the key insight that the focus should be on what can be achieved given work capacity (13m36s).
• A plot is used to illustrate the relationship between the rank-ordered cases based on the probability of an event happening and the cumulative benefit of taking action, with the goal of determining how far down the list action can be taken before diminishing returns are seen (13m53s).

Fair, Useful, Reliable (FIR) Models

• The approach is called Fair, Useful, Reliable (FIR) models, which is a multi-step process that involves usefulness simulations, financial projections, ethical considerations, and prospective evaluation (14m59s).
• The current way of doing things in AI is unsustainable, with 220 atomic pieces of guidance on how to do good AI, but half of them focus only on building the model, with little emphasis on workflow analysis and implementation (16m22s).
• The FIR approach is a packaged process that has been developed over 57 years of work and is used on a routine basis in the campus healthcare system (15m43s).

Unsustainable AI Practices and the FIR Approach

• The current organization of medical research is unsustainable, with an example of a model taking 10 years and $28 million to be tested and validated at multiple sites, highlighting the need for more efficient processes like form assessment in healthcare (16m47s).
• To create fair, useful, and reliable AI in healthcare, a three-step approach can be taken: Discovery (solving for the science), Development (validating the intent), and Dissemination (scaling) (17m43s).
• In the Discovery stage, the process is often too slow and costly, while in the Development stage, the focus should be on achievable benefit and financial sustainability, which may require changes to business models (18m15s).
• The firm assessment is a tool used to evaluate the development of AI models, with a link to the assessment available at fm.stan.edu (18m49s).

Firm Assessment and Workflow Definition

• The first step in the firm assessment is to define the workflow, including what actions will be taken and by whom, with an example workflow provided for a classifier that identifies undiagnosed pediphil arter disease (19m8s).
• Clarity on responsible action is key when building AI models, including defining the policy and workflow, and determining the threshold for action (19m42s).
• An Ethics assessment is also conducted as part of the firm assessment, considering factors such as equity, reliability, governance, and autonomy in decision-making (20m6s).

Form Assessment and Capacity Planning

• A form assessment process is used to evaluate the impact of AI projects in healthcare, considering factors such as the number of patients affected, sustainability, and potential ethical problems, with the goal of identifying good projects to pursue (20m46s).
• The assessment process involves analyzing six cases, with the first row of the table showing an example where 1,400 patients are impacted, and the project is deemed sustainable with no ethics problems (20m57s).
• Capacity planning is necessary for responsible AI in healthcare, as launching multiple projects simultaneously can be challenging with limited personnel, and operational engineering work is done to determine the number of concurrent assessments needed to achieve a certain throughput (21m33s).
• Little's law, a basic operations engineering principle, is used to calculate the required team size, indicating that a team that can handle two assessments at the same time is needed to complete at least one assessment per month (22m12s).
• Good governance is essential to ensure that everything needed is done, and a life cycle is established to make sure that the form assessment is integrated into the workflow, with governance, IT support, and standard work (22m35s).
• The governance process involves making decisions, assigning responsibility, and conducting analyses to produce numbers and inform decision-making, with the goal of ensuring that AI projects are fair, useful, and reliable (23m12s).
• The four key components of the process are standard work, IT support, governance, and form assessment, which work together to ensure that AI projects are well-planned and executed (23m34s).

Language Models and Patient Timelines

• The process was developed to ensure that machine learning projects are fair, useful, and reliable, but the emergence of large language models (LLMs) in 2022 has introduced new challenges and uncertainties (23m49s).
• A language can be viewed as a sequence of tokens from a finite vocabulary, and with this lens, a patient timeline can be seen as a language, consisting of tokens such as ICD codes, CPT codes, and LOINC codes (24m46s).
• There are two ways to build language models: the classical way using natural language, which can be used for chat and summarization, and using patient timelines to forecast what will happen, a unique way of using language models in healthcare (25m17s).

Evaluating Language Models in Healthcare

• A study was conducted to verify the effectiveness of language models in healthcare, using GPT 3.5 and GPT 4 to answer questions from a bedside service, and the results showed that agreement with reference answers increased and disagreement decreased from GPT 3.5 to GPT 4, but around 40-50% of the time, physicians couldn't decide if the answers were correct (26m19s).
• Another study, MedAlign, was conducted to evaluate the alignment of language model outputs with medical needs, and the results showed a 35% error rate in answering medical prompts, even in the best-case setup (27m57s).
• Research is being conducted to train models that can forecast patient outcomes, using a forecasting classifier or predictor, and the results show that the number of positive examples used to train the model affects its performance (28m22s).
• The performance of different models, including grade and boosted models, logistic regression, random forest, and timeline-trained language models, was compared using a receiver operator curve, with the timeline-trained language model (dark blue line) consistently showing higher accuracy (28m39s).
• The timeline-trained language model achieved an accuracy of around 78%, outperforming the highest accuracy of classical methods (red, orange, or light blue dots) while using 95% less training data and training eight times faster (29m11s).

Climber and Motor: Open-Source Language Models

• The models, called Climber and Motor, have been publicly released and can be found on GitHub (29m50s).
• The focus should be on verifying the benefits of language models and generative AI, as there are many tech companies building models that cost millions of dollars, which academic sites cannot afford (30m7s).
• It is essential to ask hard questions about whether these models actually work as advertised, and there is a need to develop a worldview for generative AI (30m36s).
• The development and dissemination of this worldview are uncertain, and there is a need to focus on verifying benefits, as seen in conflicting articles about whether AI is better than doctors (30m50s).

Building Fair, Useful, and Reliable Models

• To build fair, useful, and reliable models, data engineering and data science work must be done collaboratively, with data engineers and data scientists working side by side (31m56s).
• The data science team should have more data engineers than data scientists, and they should work together to extract, clean, and make decisions about the data, as these decisions will affect the kind of science that can be done (32m7s).

Efficiency in AI Model Development

• The time it takes to develop and refine AI models in healthcare decreases with each replication, as the team's maturity and established platforms and procedures contribute to increased efficiency and reliability (32m45s).
• The first end-to-end development of an AI model in healthcare took around 5-7 years, but subsequent replications took significantly less time, with the goal of reducing the time by 50% with each replication (33m14s).

Verification of EHR Data Accuracy

• Electronic Health Record (EHR) data is inherently noisy and prone to errors, so it's essential to verify conclusions by looking for multiple lines of corroboration, such as independent pieces of information confirming a diagnosis (34m2s).
• To establish the accuracy of EHR data, it's crucial to look beyond a single code or piece of information and instead consider multiple factors, such as lab results, medications, and other relevant data points (34m18s).

Stanford's Approach to LLMs

• Stanford is not training its own Large Language Model (LLM) for medical analysis, instead opting to use an open-source model with no copyright issues and fine-tuning it for their specific needs (34m37s).
• Training an LLM from scratch is expensive and not considered an efficient approach, especially given the rapid pace of advancements in the field (35m1s).

Promising Applications of Machine Learning in Medicine

• The most promising applications of machine learning in medicine include both clinical and non-clinical areas, such as operational applications like transcription and billing, as well as clinical applications like disease diagnosis and management (35m10s).
• The choice of application and prioritization of AI development in healthcare depends on the specific environment and structural issues of the healthcare system, with different priorities in resource-rich versus resource-poor settings (35m31s).
• In resource-poor settings, AI may be used for clinical care, such as retinal scanning algorithms for diabetic patients, due to the lack of alternative options and the potential for AI to provide better-than-no-care solutions (36m27s).
• The development and implementation of AI in healthcare must consider the local context and prioritize applications that address the most pressing needs and challenges in that specific environment (36m50s).

Real-world Applications and Generative AI in Behavioral Health

• Academic work has been successfully applied to solve real-life problems in the healthcare industry, such as predicting mortality for improving Advanced Care planning, which has improved the care of over 6,000 patients at Stanford Healthcare.
• Generative AI has potential applications in Behavioral Health, but there are concerns about its reliability, as seen in incidents like Gemini providing harmful advice to a high schooler, making it uncertain if it's the best use of the technology today.

Patient Involvement and Stakeholder Collaboration

• Patient involvement in implementing AI models, such as the FEMR model, is ensured through a patient family advisory council, which reviews the workflow and actions taken by the algorithm to ensure patient comfort and consent.
• Multiple stakeholders, including clinicians, patients, administrators, and developers, should be involved in the development and deployment of AI algorithms to ensure their appropriateness and effectiveness.

Machine Learning in Laboratories

• Machine learning has practical applications in laboratories, such as in histology, cytology, or flow cytometry, with examples including deep neural nets that can assist pathologists in reading slides and identifying areas of interest.
• AI is widely used in pathology, ranging from cell sorters that use lasers and calculations to count white blood cells, to more advanced tools that help pathologists read slides and augment their work.

Data Gaps in EHR Systems

• Critical data gaps exist in current Electronic Health Record (EHR) systems, with the biggest issue being the presence of too many systems, often ranging from 500 to 1,000, which can include EHR systems like Epic or Cerner, as well as specialized systems for various departments, resulting in medical data being scattered across hundreds of systems (41m7s).
• It is a myth to believe that all medical data is in the EHR, and instead, there is an opportunity to combine and put this data in one place (41m46s).

Addressing Bias in Predictions

• When addressing bias in predictions made by models, it is essential to distinguish between two interpretations of bias: systematic differences in model performance for people belonging to different subgroups, and systematic differences in the actual benefit or reward resulting from the model's output (42m9s).
• The latter type of bias is more concerning, as it can result in unequal benefits for people belonging to different subgroups, and addressing this requires focusing on policies and workflows driven by the model's output rather than just removing model-side differences (42m48s).
• Algorithmic fixes may not always be effective in addressing bias, and it is crucial to consider other factors that can impact the actual benefit or reward, such as policies and workflows (43m16s).

Traceability, Explainability, and Trust

• Traceability and explainability requirements can slow down the progress of AI model advancements, but it depends on the purpose of these requirements, which can include debugging, mitigating outcomes, or establishing trust (43m50s).
• Different scenarios require different types of interpretability, including engineers' interpretability for debugging, causal interpretability for mitigating outcomes, and transparency for establishing trust (44m25s).
• Providing the wrong type of explanation can be counterproductive, and it is essential to understand the purpose of the request for explainability or interpretability to provide the most effective response (45m10s).
• Establishing trust in AI models can be done through prospective studies, similar to how trust is established in medications, even if the exact mechanism of action is not fully understood (45m20s).

AI Assistance and Medical Errors

• Studies have compared error rates in medical diagnosis and treatment with and without AI assistance, with one recent study by Jonathan Chen finding that doctors sometimes make mistakes when using AI, possibly due to suboptimal use (45m59s).
• The study by Jonathan Chen involved giving case vignettes to physicians with and without access to AI, and found that AI alone performed better than doctors with AI assistance (46m6s).
• More research is needed to understand how AI can be effectively used in practice, rather than just debating whether it should be used (46m56s).

EHRs as Raw Material for AI Training

• Electronic Health Records (EHRs) can be considered one of the raw materials for teaching AI, but should not be the only source, and should be supplemented with information from textbooks and online sources (47m15s).
• Medical Imaging AI has many practical uses, with the FDA approving around 1,000 image-based models, mostly in radiology and cardiology (48m4s).

Medical Imaging AI and Resources

• The Stanford Center for Health Education offers resources for learning more about machine learning and medicine, including a program and online content (49m1s).