Performance of large language models in interpreting evidence-based clinical guidelines for lumbar disc herniation with radiculopathy.

OBJECTIVE

Large language models (LLMs) are increasingly used as clinical information tools; however, their ability to accurately interpret evidence-based spine guidelines remains unclear. This study compared the performance of ChatGPT-5.1, Gemini, and Perplexity in interpreting the North American Spine Society (NASS) guideline for lumbar disc herniation with radiculopathy.

METHODS

Nineteen open-ended clinical questions derived from the NASS guideline were submitted to each LLM under standardized conditions. Responses were evaluated by two blinded clinicians using validated Likert scales for clinical accuracy (1-5), reliability, and usability (1-7). Semantic similarity to guideline-based answers was assessed using the Universal Sentence Encoder, surface-level textual similarity using ROUGE-L F1 scores, and readability using multiple established readability indices. Reference reliability was analyzed using the Reference Hallucination Score.

RESULTS

Perplexity demonstrated significantly higher clinical accuracy (3.95 ± 0.70) compared with ChatGPT-5.1 (3.45 ± 0.68) and Gemini (3.50 ± 0.65) (p = 0.018). Reliability and usability scores were also highest for Perplexity (4.85 ± 1.05 and 4.75 ± 0.95, respectively; both p < 0.01). Semantic similarity scores were greater for Perplexity (0.71 ± 0.06) than for ChatGPT-5.1 (0.64 ± 0.07) (p < 0.001), whereas Gemini achieved the highest ROUGE-L F1 scores (0.14 ± 0.04; p < 0.001). Readability indices were comparable across models, indicating similar levels of textual complexity. ChatGPT-5.1 exhibited the highest reference hallucination (8.10 ± 2.85), while Perplexity showed the lowest (4.15 ± 2.70) (p < 0.001).

CONCLUSIONS

LLMs show significant variability in guideline-based clinical reasoning. Although none should be used as independent decision-making tools, reference-oriented models may provide more reliable adjunctive support for evidence-based spine practice.