Leveraging simulation to provide a practical framework for estimating the novel scope of risk of large language models in healthcare.

BACKGROUND

Large language models (LLMs) are rapidly entering clinical and consumer use, yet their probabilistic outputs have delivered a variety of unsafe user responses. Difficulties in quantifying and mitigating risks posed by LLMs threaten to stall regulatory evaluation and clinical deployment of LLM-based software as a medical device (LLM-SaMD). Practical approaches are needed to extend existing medical-device regulations to LLM-SaMDs.

OBJECTIVE

To demonstrate how simulation can extend existing medical-device risk management frameworks for addressing LLM-SaMD-specific risks.

METHODS

We implement a simulation-based methodology for estimating LLM-SaMD risk. Fourteen open-source models were evaluated on three safety-classification tasks: suicidal-ideation, therapy-request and therapy-like interaction detection. Synthetic datasets were generated by Gemini 2.5 Pro and evaluated by psychiatrists. Model false-negative rates informed estimates of P1, the likelihood that a hazard progresses to a hazardous situation, and P2, the likelihood that that situation results in harm.

FINDINGS: LLM success at generating synthetic datasets varied by task, with strong performance for neutral and non-therapeutic content but frequent errors in suicidal-ideation and therapy-like interactions. Performance generally improved with model size. Estimated P1 values ranged from 1.1×10⁻⁸ to 1.6×10⁻⁴ and P2 from 4.9×10⁻⁵ to 5.1×10⁻³, spanning four orders of magnitude.

CONCLUSIONS

By linking model failure modes to structured pathways to harm, simulation can extend existing medical-device risk frameworks to help address the probabilistic and context-dependent risks of LLM-SaMDs.

CLINICAL IMPLICATIONS: Simulation-based risk estimation offers a practical way to characterise the risk landscape for specific LLM-SaMD, patient population and clinical context combinations.