Comparison of responses from google and large language models to the top frequently asked questions on lumbar spinal stenosis: an evaluation of accuracy and completeness.

PURPOSE: Lumbar spinal stenosis (LSS) is a common degenerative spinal condition and a leading cause of pain and disability in adults. With increasing use of artificial intelligence (AI) for medical information, this study evaluated and compared the accuracy and completeness of responses generated by large language models (LLMs) and Google Search to standardized patient-oriented questions about LSS, aiming to characterize AI performance in patient education.

METHODS

Eight frequently asked questions regarding LSS were identified using NHS hospital websites, NASS clinical guidelines, Google "People Also Ask," Reddit discussions, and common ChatGPT prompts. Each question was submitted to Google Search, ChatGPT, Gemini, and Perplexity under standardized conditions. Orthopedic spine surgeons independently rated responses for accuracy and completeness using 5-point Likert scales. Parametric (one-way ANOVA) and non-parametric (Kruskal-Wallis) tests evaluated overall differences. A minimal clinically important difference (MCID) was established as a ≥ 1.0-point delta, with post-hoc pairwise comparisons evaluated using Tukey's Honestly Significant Difference.

RESULTS

Gemini achieved the highest completeness (4.47 ± 0.67) and accuracy (4.34 ± 0.65), followed by Perplexity (completeness 4.03 ± 0.82; accuracy 4.00 ± 0.95). Google and ChatGPT demonstrated similar completeness (3.75 ± 1.04 vs. 3.78 ± 0.71), though Google showed slightly higher accuracy than ChatGPT (3.78 ± 0.91 vs. 3.63 ± 0.75). Global performance variations were highly significant across both omnibus parametric testing (completeness: p = 0.002; accuracy: p = 0.004) and non-parametric Kruskal-Wallis testing (completeness: p = 0.003; accuracy: p = 0.004). Post-hoc pairwise testing via Tukey's HSD confirmed that this statistical significance was driven solely by Gemini, which outperformed both ChatGPT (completeness p = 0.006; accuracy p = 0.004) and Google Search (completeness p = 0.004; accuracy p = 0.036). No other isolated pairwise combinations achieved statistical significance (p > 0.05). Despite these isolated statistical boundaries, no platform pairing cleared the predefined ≥ 1.0-point threshold for clinical significance, with maximum mean score deltas reaching only 0.72 for completeness and 0.71 for accuracy.

CONCLUSIONS

Although newer-generation and search-augmented LLMs demonstrate clear statistical superiority over traditional web results in synthesizing structured spine information, these differences do not yet translate into clinically meaningful quality shifts under concise prompt constraints. Substantial inter-rater variability and the lack of threshold-clearing clinical significance reinforce that these emerging AI platforms should serve as supplements to, rather than substitutes for, physician-guided counseling in degenerative spine care.