BMJ digital health & AI · 2025
Clinical AI Scribes in primary care: accuracy, error severity and implications for clinical practice.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Draper Thomas C, Cox Timothy, Lamb-Riddell Kathryn, Moretti Luigi Andrea, McCormick John, Trowell Stephen, Kiely Janice, Luxton Richard
Аннотация
OBJECTIVES: To investigate the performance of commercially available Clinical Artificial Intelligence Scribes (CAISs), assessing their accuracy, potential clinical impact of errors, and documentation quality, given growing concerns around errors and safety.
METHODS AND ANALYSIS: Seven CAIS products were investigated, using eight standardised clinical consultation scenarios recorded as audio. CAIS-generated summaries were assessed against a human-validated transcript and evaluated for errors (omissions, factual inaccuracies and hallucinations). Error severity was rated by medical doctors, generating a novel severity-weighted mpact Score (linear and exponential variants), to quantify potential clinical impact. Further analysis using the Physician Documentation Quality Instrument (PDQI-10) (a validated clinical note quality score) reinforced the findings.
RESULTS: Omissions dominated error counts (83.8%, p<<0.001), with CAISs varying widely in error frequency and severity, and a median of 1-6 omissions per consultation (depending on CAIS). Although less frequent, hallucinations and factual inaccuracies were more often clinically serious. No tested CAIS produced error-free summaries. The Impact Score highlighted clinical severity, notably amplifying the significance of less frequent but high-severity errors. PDQI-10 analysis indicated summaries were weakest in succinctness and organisation, but strong in consistency and clinical usefulness.
CONCLUSIONS: The CAISs demonstrate high levels of summarisation accuracy. However, there is great disparity between the currently available CAIS products and, while some perform well, none are perfect. Clinicians should therefore maintain vigilance, particularly checking omitted psychosocial details and medications, and scrutinising plausible-sounding insertions. Purchasers and regulators should be aware of the significant performance disparities identified, reinforcing the need for careful evaluation and selection of CAIS products.
METHODS AND ANALYSIS: Seven CAIS products were investigated, using eight standardised clinical consultation scenarios recorded as audio. CAIS-generated summaries were assessed against a human-validated transcript and evaluated for errors (omissions, factual inaccuracies and hallucinations). Error severity was rated by medical doctors, generating a novel severity-weighted mpact Score (linear and exponential variants), to quantify potential clinical impact. Further analysis using the Physician Documentation Quality Instrument (PDQI-10) (a validated clinical note quality score) reinforced the findings.
RESULTS: Omissions dominated error counts (83.8%, p<<0.001), with CAISs varying widely in error frequency and severity, and a median of 1-6 omissions per consultation (depending on CAIS). Although less frequent, hallucinations and factual inaccuracies were more often clinically serious. No tested CAIS produced error-free summaries. The Impact Score highlighted clinical severity, notably amplifying the significance of less frequent but high-severity errors. PDQI-10 analysis indicated summaries were weakest in succinctness and organisation, but strong in consistency and clinical usefulness.
CONCLUSIONS: The CAISs demonstrate high levels of summarisation accuracy. However, there is great disparity between the currently available CAIS products and, while some perform well, none are perfect. Clinicians should therefore maintain vigilance, particularly checking omitted psychosocial details and medications, and scrutinising plausible-sounding insertions. Purchasers and regulators should be aware of the significant performance disparities identified, reinforcing the need for careful evaluation and selection of CAIS products.