Generative AI ‘drift’ and ‘nondeterminism’ inconsistences are important considerations in health care applications

Samuel (Sandy) Aronson, ALM, MA, executive director of IT and AI Solutions for Mass General Brigham Personalized Medicine and senior director of IT and AI Solutions for the Accelerator for Clinical Transformation, is the corresponding author of a paper published in NEJM AI that looked at whether generative AI could hold promise for improving scientific literature review of variants in clinical genetic testing. Their findings could have a wide impact beyond this use case.

How would you summarize your study for a lay audience?

We tested whether generative AI can be used to identify whether scientific articles contain information that can help geneticists determine whether genetic variants are harmful to patients. While testing this work, we identified inconsistencies in generative AI that could present a risk for patients if not adequately addressed. We suggest forms of testing and monitoring that could improve safety.

What question were you investigating?

We investigated whether generative AI can be used to determine: 1) whether a scientific article contains evidence about a variant that could help a geneticist’s assessment of a genetic variant and 2) whether any evidence found about the variant supports a benign, pathogenic, intermediate or inconclusive conclusion.

What methods or approach did you use?

We tested a generative AI strategy based on GPT-4 using a labeled dataset of 72 articles and compared generative AI to assessments from expert geneticists.

What did you find?

Generative AI performed relatively well, but more improvement is needed for most use cases. However, as we ran our tests repeatedly, we observed a phenomenon we deemed important: running the same test dataset repeatedly produced different results. Through repeated running of the test set over time, we characterized the variability. We found that both drift (changes in model performance over time) and nondeterminism (inconsistency between consecutive runs) were present. We developed visualizations that demonstrate the nature of these problems.

What are the implications?

If a clinical tool developer is not aware that large language models can exhibit significant drift and nondeterminism, they may run their test set once and use the results to determine whether their tool can be introduced into practice. This could be unsafe.

What are the next steps?

Our results show that it could be important to run a test set multiple times to demonstrate the degree of variability (nondeterminism) present. Our results also show that it is important to monitor for changes in performance (drift) over time.