Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology · 2025
Can Chatbots Please Both Patients and Experts? Benchmarking AI and Clinical Guidelines for Hearing Loss.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Hack Sholem, Gvili Ben, Tessler Idit, Yogev David, Wolfowitz Amit, Rozendorn Noa
Аннотация
OBJECTIVE: To evaluate the reliability, accuracy, and clarity of responses generated by 7 contemporary artificial intelligence chatbots in answering patient-focused questions about age-related hearing loss and sudden sensorineural hearing loss, and to compare these outputs to expert-authored guideline responses as well as layperson ratings.
STUDY DESIGN: Cross-sectional.
SETTING: Academic medical center.
PATIENTS: Not applicable. Ten independent layperson raters, all over the age of 18, recruited from personal networks, assessed a subset of chatbot and expert responses.
INTERVENTIONS: Patient-centered questions, derived from official clinical practice guidelines for hearing loss, were submitted to 7 artificial intelligence chatbots from 3 major development groups. Responses were rated by a blinded panel of 5 otolaryngologists for accuracy, extensiveness, misleading content, quality of cited references, and overall reliability. A panel of 10 independent layperson raters, all over the age of 18, recruited from personal networks, assessed a subset of chatbot and expert responses.
MAIN OUTCOME MEASURES: Proportion of chatbot answers rated fully accurate by expert panel; mean layperson clarity and trustworthiness scores; frequency of misleading information and high-quality references.
RESULTS: The most advanced chatbots achieved full guideline-concordant accuracy for up to 50% of questions, while earlier models ranged from 25% to 37.5%. All models performed highly for extensiveness and reference quality. Layperson ratings were highest for gold-standard expert answers, the latest chatbots approached these levels for both clarity and trustworthiness (mean scores: 4.7 to 4.8 out of 5; 95% CI: 4.67-4.85), and differences between models were of moderate-to-large effect size (η 2 =0.29 to 0.30). Misleading content was rare and typically not clinically significant.
CONCLUSIONS: Modern artificial intelligence chatbots can provide clear and generally reliable patient education for hearing loss, but full guideline concordance remains inconsistent. Expert oversight is advised to ensure clinical accuracy.
STUDY DESIGN: Cross-sectional.
SETTING: Academic medical center.
PATIENTS: Not applicable. Ten independent layperson raters, all over the age of 18, recruited from personal networks, assessed a subset of chatbot and expert responses.
INTERVENTIONS: Patient-centered questions, derived from official clinical practice guidelines for hearing loss, were submitted to 7 artificial intelligence chatbots from 3 major development groups. Responses were rated by a blinded panel of 5 otolaryngologists for accuracy, extensiveness, misleading content, quality of cited references, and overall reliability. A panel of 10 independent layperson raters, all over the age of 18, recruited from personal networks, assessed a subset of chatbot and expert responses.
MAIN OUTCOME MEASURES: Proportion of chatbot answers rated fully accurate by expert panel; mean layperson clarity and trustworthiness scores; frequency of misleading information and high-quality references.
RESULTS: The most advanced chatbots achieved full guideline-concordant accuracy for up to 50% of questions, while earlier models ranged from 25% to 37.5%. All models performed highly for extensiveness and reference quality. Layperson ratings were highest for gold-standard expert answers, the latest chatbots approached these levels for both clarity and trustworthiness (mean scores: 4.7 to 4.8 out of 5; 95% CI: 4.67-4.85), and differences between models were of moderate-to-large effect size (η 2 =0.29 to 0.30). Misleading content was rare and typically not clinically significant.
CONCLUSIONS: Modern artificial intelligence chatbots can provide clear and generally reliable patient education for hearing loss, but full guideline concordance remains inconsistent. Expert oversight is advised to ensure clinical accuracy.