Clinical microbiology and infection : the official publication of the European Society of Clinical Microbiology and Infectious Diseases · 2026
Evaluation of six generative large language models answering pre-analytical questions in medical microbiology.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Bressan Michelle, Meijer Eline Nadieh, Hinic Vladimira, Ellendorff Tilia, Goldman Jean-Philippe, Kew Tannon, Nolte Oliver, Schneider Gerold, Egli Adrian, LLM Evaluation Consortium
Аннотация
OBJECTIVES: Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in pre-analytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common pre-analytical questions in the domain of medical microbiology. Next, we propose a blueprint to evaluate LLMs in medical microbiology.
METHODS: We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domain-enhanced versions (customised GPT-4 agent and Llama2-7B Medical) in a hybrid evaluation framework. First, we quantified the response length (word count), then assessed the similarity between LLM-generated responses and expert-reviewed reference answers (BLEU/ROUGE scores, BERTScore). Finally, thirty-three medical professionals rated the accuracy and completeness of the LLM-generated responses on a 1-7-point scale.
RESULTS: Responses of OpenAI models showed greater similarity to the reference set and significantly outperformed Meta AI-based models in accuracy and completeness. The customized generative pre-trained transformer (GPT) matched performance of OpenAI's general-purpose models, while generating significantly shorter responses (p<0.001). Quality ratings varied markedly between professionals, as medical microbiologists and experienced infectious disease specialists assigned significantly lower scores compared to infectious disease trainees and medical students (p<0.001).
CONCLUSIONS: We observed considerable variation in LLM performance and highlight differences in perceived response quality across professional groups, emphasizing the need for critical appraisal when consulting LLMs for pre-analytical guidance.
METHODS: We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domain-enhanced versions (customised GPT-4 agent and Llama2-7B Medical) in a hybrid evaluation framework. First, we quantified the response length (word count), then assessed the similarity between LLM-generated responses and expert-reviewed reference answers (BLEU/ROUGE scores, BERTScore). Finally, thirty-three medical professionals rated the accuracy and completeness of the LLM-generated responses on a 1-7-point scale.
RESULTS: Responses of OpenAI models showed greater similarity to the reference set and significantly outperformed Meta AI-based models in accuracy and completeness. The customized generative pre-trained transformer (GPT) matched performance of OpenAI's general-purpose models, while generating significantly shorter responses (p<0.001). Quality ratings varied markedly between professionals, as medical microbiologists and experienced infectious disease specialists assigned significantly lower scores compared to infectious disease trainees and medical students (p<0.001).
CONCLUSIONS: We observed considerable variation in LLM performance and highlight differences in perceived response quality across professional groups, emphasizing the need for critical appraisal when consulting LLMs for pre-analytical guidance.