A comparative evaluation of large language models in aligning with European Respiratory Society (ERS) guidelines for high-flow nasal cannula in acute respiratory failure.

BACKGROUND

The potential for large language models (LLMs) to standardize clinical decision-making in ARF, particularly in optimizing high-flow nasal cannula (HFNC) therapy application, is gaining interest due to the condition's complex etiology. This study aimed to evaluate the clinical utility of artificial intelligence (AI) models in aligning with the European Respiratory Society (ERS) clinical practice guidelines for the use of HFNC in acute respiratory failure (ARF).

METHODS

Eight advanced LLMs were assessed across eight representative clinical scenarios related to HFNC use in ARF. Each model's responses were independently scored by three independent reviewers across four domains: accuracy, overconclusiveness, supplementary value, and completeness, using a 5-point Likert scale. Inter-rater reliability was measured by Fleiss' Kappa. Readability metrics, including Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL), were also analyzed.

RESULTS

No differences were detected in accuracy or overconclusiveness (both P>0.99). Significant differences emerged in supplementary value among models (P<0.0001). DeepSeek-V3.1 ranked highest, significantly outperforming Claude and ChatGPT models (P<0.05). For completeness, ChatGPT-4o scored lower than both DeepSeek-V3.1 (P=0.006) and DeepSeek-R1 (P=0.01). Readability analyses revealed that DeepSeek-V3.1 achieved the highest FRE score, while DeepSeek-R1 had the lowest FKGL, reflecting better textual accessibility. Inter-rater reliability was substantial (κ=0.781).

CONCLUSIONS

Evaluations of LLMs for HFNC guideline interpretation show that while all are accurate, DeepSeek-V3.1 and R1 excel in completeness, supplemental detail, and readability, marking them as more suitable for ARF clinical decision support. Their integration into critical workflows still demands expert supervision.