Streamlining the Abstract Selection Process for the American Society of Breast Surgeons' Annual Meeting by Utilizing Large Language Models.

PURPOSE: The American Society of Breast Surgeons (ASBrS) receives an increasing number of abstracts for its annual meeting, resulting in increasing time and effort for peer-review. We aimed to compare artificial intelligence (AI) large language models (LLMs) to human-generated scores to explore if AI-review is a plausible abstract screening process.

METHODS

Abstracts published from the 2025 ASBrS meeting were assessed with three LLMs: OpenAI's GPT4-o (GPT), Meta's Llama-3.1-405b (Llama) and DeepSeek's DeepSeek-V3 (DeepSeek). Abstracts were analyzed using zero shot (ZS), which included the 2025 ASBrS scoring rubric and few shot (FS) with examples of high- and low-scoring abstracts. Large language model scores were compared with the committee's scores. Model inference was evaluated by using Spearman rank correlation, quartile accuracy, and mean absolute error; 95% CIs were calculated by using bootstrap resampling.

RESULTS

A total of 378 published accepted abstracts were included. Human-generated median abstract score was 21.2 (rubric ranged 0-35). Median scores were higher amongst all LLMs for both the ZS and FS approaches (27.0-29.7, p < 0.001). Spearman values with the ZS approach for GPT, DeepSeek, and Llama were 0.24, 0.30, and 0.31, respectively, and for the FS approach were 0.37, 0.35, and 0.27, respectively (p < 0.001 for all). DeepSeek ranked the first quartile correctly with 53.5% accuracy while Llama ranked the fourth quartile correctly with 42.9% accuracy. The FS approach modestly improved correlation and first quartile accuracy, with greatest improvement seen in GPT.

CONCLUSIONS

Large language models showed statistically significant rank correlation with human ASBrS abstract ranking. Large language models may be useful as a preliminary screening tool to prioritize reviewer effort toward higher-quality abstracts.