Evaluation of the performance of large language models in the management of hip and knee osteoarthritis: analysis of EULAR 2023 recommendations.

BACKGROUND

Clinical practice guidelines play a pivotal role in the management of complex and chronic conditions such as osteoarthritis (OA). However, the systematic evaluation of large language models (LLMs) against established guidelines-particularly in the context of knee and hip OA-remains limited. This study aimed to compare the responses generated by multiple LLMs to open-ended questions derived from the 2023 European League Against Rheumatism (EULAR) recommendations.

METHODS

This study transformed eight recommendations from the 2023 EULAR guideline on non-pharmacological management of knee and hip OA into clinically framed open-ended questions. Each question was posed to DeepSeek-R1, Gemini 2.5 Flash, and ChatGPT-5. The generated responses were evaluated in terms of informational quality, reliability, usability, readability, as well as semantic and lexical (surface-level) similarity to the reference recommendations.

RESULTS

DeepSeek-R1 achieved significantly higher scores in informational quality, reliability, usability, and readability compared with Gemini 2.5 Flash and ChatGPT-5 (p < 0.05). Lexical overlap with the reference recommendations was significantly lower for DeepSeek-R1 than for the other models (p = 0.001), indicating reduced verbatim similarity. No statistically significant differences were observed among the models with respect to semantic similarity (p = 0.409).

CONCLUSION

All three models produced generally acceptable outputs; however, DeepSeek-R1 consistently outperformed Gemini 2.5 and ChatGPT-5. Although these models show promising informational quality, reliability, and usability, limited readability restricts their accessibility. LLMs are not yet appropriate for independent patient use and should currently function as supportive tools under the supervision of qualified healthcare professionals.