Frontiers in public health · 2026
Benchmarking public large language model responses to patient-facing varicose veins questions: informational quality, verifiability indicators, and readability.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Zhong Wei, Zheng Guoxue, Li Qin, Huang Yu, Zhao Jinjie
Аннотация
OBJECTIVES: To benchmark the informational quality, verifiability indicators, and readability of publicly accessible large language model (LLM) responses to standardized, patient-facing varicose veins (VV) questions.
METHODS: Twenty single-intent VV questions were derived from PubMed-indexed VV and chronic venous disease guidelines and consensus statements (search date: February 10, 2026). The question set was designed as a decision-critical benchmark across the care pathway rather than a prevalence-weighted sample of real-world patient queries. Five publicly accessible LLMs (ChatGPT 5.2, DeepSeek-V3.2, Gemini 3 Pro, Grok 4.1, and Qwen3-Max) were queried through their official web interfaces from February 10 to 12, 2026, under default settings, generating 100 responses. Each prompt was entered in a new privacy-mode session, and refusals or other non-responsive outputs were retained as returned. Two blinded clinicians independently rated DISCERN (16-80), EQIP (0%-100%), GQS (1-5), and the JAMA benchmark (0-4). In this study, JAMA was used as a structured measure of visible attribution and verifiability-related features rather than as a comprehensive measure of transparency for conversational AI. Readability was assessed using six standard indices. Interrater reliability was evaluated with ICC(A,1) and weighted Cohen's κ. Between-model differences were tested using Friedman tests with Kendall's W and Holm adjustment.
RESULTS: Interrater reliability was high [DISCERN ICC(A,1) = 0.913; EQIP ICC(A,1) = 0.859; GQS κ = 0.883; JAMA κ = 0.864]. Informational-quality scores were broadly similar across models (DISCERN means, 46.50-50.75; EQIP means, 71.50-74.25; GQS medians, 4.0). JAMA scores were uniformly low (means, 0.00-0.25; medians, 0), indicating sparse visible attribution and limited verifiability cues in default outputs. Between-model differences in the primary informational-quality outcomes were small and were not significant after Holm adjustment. Readability differences were more pronounced, and all models exceeded commonly recommended sixth-grade readability thresholds.
CONCLUSIONS: Under default public-user settings, publicly accessible LLMs generated fluent VV responses with limited visible verifiability indicators and suboptimal readability. Differences in the primary informational-quality outcomes were modest and should be interpreted cautiously. This benchmark evaluates communication-related performance rather than claim-level clinical accuracy or safety. These findings support efforts to improve auditability, provenance reporting, and uncertainty communication, but these dimensions do not substitute for formal assessment of factual accuracy, guideline concordance, and clinical safety.
METHODS: Twenty single-intent VV questions were derived from PubMed-indexed VV and chronic venous disease guidelines and consensus statements (search date: February 10, 2026). The question set was designed as a decision-critical benchmark across the care pathway rather than a prevalence-weighted sample of real-world patient queries. Five publicly accessible LLMs (ChatGPT 5.2, DeepSeek-V3.2, Gemini 3 Pro, Grok 4.1, and Qwen3-Max) were queried through their official web interfaces from February 10 to 12, 2026, under default settings, generating 100 responses. Each prompt was entered in a new privacy-mode session, and refusals or other non-responsive outputs were retained as returned. Two blinded clinicians independently rated DISCERN (16-80), EQIP (0%-100%), GQS (1-5), and the JAMA benchmark (0-4). In this study, JAMA was used as a structured measure of visible attribution and verifiability-related features rather than as a comprehensive measure of transparency for conversational AI. Readability was assessed using six standard indices. Interrater reliability was evaluated with ICC(A,1) and weighted Cohen's κ. Between-model differences were tested using Friedman tests with Kendall's W and Holm adjustment.
RESULTS: Interrater reliability was high [DISCERN ICC(A,1) = 0.913; EQIP ICC(A,1) = 0.859; GQS κ = 0.883; JAMA κ = 0.864]. Informational-quality scores were broadly similar across models (DISCERN means, 46.50-50.75; EQIP means, 71.50-74.25; GQS medians, 4.0). JAMA scores were uniformly low (means, 0.00-0.25; medians, 0), indicating sparse visible attribution and limited verifiability cues in default outputs. Between-model differences in the primary informational-quality outcomes were small and were not significant after Holm adjustment. Readability differences were more pronounced, and all models exceeded commonly recommended sixth-grade readability thresholds.
CONCLUSIONS: Under default public-user settings, publicly accessible LLMs generated fluent VV responses with limited visible verifiability indicators and suboptimal readability. Differences in the primary informational-quality outcomes were modest and should be interpreted cautiously. This benchmark evaluates communication-related performance rather than claim-level clinical accuracy or safety. These findings support efforts to improve auditability, provenance reporting, and uncertainty communication, but these dimensions do not substitute for formal assessment of factual accuracy, guideline concordance, and clinical safety.