Overuse and False Restraint in Large Language Model Recommendations for Choosing Wisely Scenarios.

BACKGROUND

The Choosing Wisely initiative made recommendations where screening, testing, or treatment may be overused, with caveats around appropriate use. Clinicians are increasingly using large language models (LLMs).

OBJECTIVE

To evaluate whether commercial LLMs appropriately call for restraint or action from scenarios based on Choosing Wisely recommendations.

DESIGN

Benchmark evaluation. Fifteen Choosing Wisely recommendations were used to develop 225 scenarios (per recommendation, 10 in which restraint was appropriate and 5 in which action was appropriate). Six commercial LLMs (3 runs each) generated 4050 responses to the 225 scenarios, using a prompt from the clinician perspective.

SETTING

Production application programming interfaces of OpenAI, Anthropic, and Google from February to March 2026.

PARTICIPANTS

No human participants were involved.

MEASUREMENTS: Co-primary outcomes were the overuse rate (endorsing action when restraint was appropriate) and the false restraint rate (withholding an appropriate action).

RESULTS

The rate of LLM-recommended overuse was 5.4% (95% CI, 0.6% to 12.2%); LLMs did not recommend overuse in 8 of 15 recommendations. Two recommendations accounted for most LLM-recommended overuse: thyroid ultrasound for an incidental subcentimeter nodule (overuse recommended in 42.8% of LLM responses) and routine electrocardiography in low-risk adults (overuse recommended in 28.3% of LLM responses). The LLMs recommended overuse in 0.6% to 3.3% of responses for the remaining 5 recommendations. False restraint was rare (0.6% [CI, 0.1% to 1.2%]), was absent for 11 of 15 recommendations, and was highest for prostate-specific antigen screening (3.3%).

LIMITATIONS: Single prompts without follow-up; a benchmark rather than real-world use; responses classified by LLM judge with physician adjudication. Findings may not extend to consumer-facing use.

CONCLUSION

The LLMs rarely withheld recommendations for treatment when it was appropriate. Overuse was concentrated in 2 recommendations.

PRIMARY FUNDING SOURCE: None.