Annals of internal medicine · 2026
Overuse and False Restraint in Large Language Model Recommendations for Choosing Wisely Scenarios.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Ariel Dean, Endelman Stav, Laban Guy
Аннотация
BACKGROUND: The Choosing Wisely initiative made recommendations where screening, testing, or treatment may be overused, with caveats around appropriate use. Clinicians are increasingly using large language models (LLMs).
OBJECTIVE: To evaluate whether commercial LLMs appropriately call for restraint or action from scenarios based on Choosing Wisely recommendations.
DESIGN: Benchmark evaluation. Fifteen Choosing Wisely recommendations were used to develop 225 scenarios (per recommendation, 10 in which restraint was appropriate and 5 in which action was appropriate). Six commercial LLMs (3 runs each) generated 4050 responses to the 225 scenarios, using a prompt from the clinician perspective.
SETTING: Production application programming interfaces of OpenAI, Anthropic, and Google from February to March 2026.
PARTICIPANTS: No human participants were involved.
MEASUREMENTS: Co-primary outcomes were the overuse rate (endorsing action when restraint was appropriate) and the false restraint rate (withholding an appropriate action).
RESULTS: The rate of LLM-recommended overuse was 5.4% (95% CI, 0.6% to 12.2%); LLMs did not recommend overuse in 8 of 15 recommendations. Two recommendations accounted for most LLM-recommended overuse: thyroid ultrasound for an incidental subcentimeter nodule (overuse recommended in 42.8% of LLM responses) and routine electrocardiography in low-risk adults (overuse recommended in 28.3% of LLM responses). The LLMs recommended overuse in 0.6% to 3.3% of responses for the remaining 5 recommendations. False restraint was rare (0.6% [CI, 0.1% to 1.2%]), was absent for 11 of 15 recommendations, and was highest for prostate-specific antigen screening (3.3%).
LIMITATIONS: Single prompts without follow-up; a benchmark rather than real-world use; responses classified by LLM judge with physician adjudication. Findings may not extend to consumer-facing use.
CONCLUSION: The LLMs rarely withheld recommendations for treatment when it was appropriate. Overuse was concentrated in 2 recommendations.
PRIMARY FUNDING SOURCE: None.
OBJECTIVE: To evaluate whether commercial LLMs appropriately call for restraint or action from scenarios based on Choosing Wisely recommendations.
DESIGN: Benchmark evaluation. Fifteen Choosing Wisely recommendations were used to develop 225 scenarios (per recommendation, 10 in which restraint was appropriate and 5 in which action was appropriate). Six commercial LLMs (3 runs each) generated 4050 responses to the 225 scenarios, using a prompt from the clinician perspective.
SETTING: Production application programming interfaces of OpenAI, Anthropic, and Google from February to March 2026.
PARTICIPANTS: No human participants were involved.
MEASUREMENTS: Co-primary outcomes were the overuse rate (endorsing action when restraint was appropriate) and the false restraint rate (withholding an appropriate action).
RESULTS: The rate of LLM-recommended overuse was 5.4% (95% CI, 0.6% to 12.2%); LLMs did not recommend overuse in 8 of 15 recommendations. Two recommendations accounted for most LLM-recommended overuse: thyroid ultrasound for an incidental subcentimeter nodule (overuse recommended in 42.8% of LLM responses) and routine electrocardiography in low-risk adults (overuse recommended in 28.3% of LLM responses). The LLMs recommended overuse in 0.6% to 3.3% of responses for the remaining 5 recommendations. False restraint was rare (0.6% [CI, 0.1% to 1.2%]), was absent for 11 of 15 recommendations, and was highest for prostate-specific antigen screening (3.3%).
LIMITATIONS: Single prompts without follow-up; a benchmark rather than real-world use; responses classified by LLM judge with physician adjudication. Findings may not extend to consumer-facing use.
CONCLUSION: The LLMs rarely withheld recommendations for treatment when it was appropriate. Overuse was concentrated in 2 recommendations.
PRIMARY FUNDING SOURCE: None.