Assessment of OpenEvidence responses to American Academy of Orthopaedic Surgeons clinical practice guidelines for anterior cruciate ligament reconstruction.

INTRODUCTION/

OBJECTIVES

General-purpose large language models (LLMs) have substantial limitations, including fabricated references and inconsistent concordance with evidence-based clinical practice guidelines (CPGs). OpenEvidence (OE) (Cambridge, MA) is a retrieval-augmented generation (RAG) platform designed to provide accurate, evidence-based, citation-supported answers. This study assessed the concordance, quality, and citation characteristics of OE responses to the 2022 American Academy of Orthopaedic Surgeons (AAOS) CPG for anterior cruciate ligament reconstruction (ACLR). It was hypothesized that OE would deliver relevant, accurate, clear, complete, and evidence-based responses, with low rates of hallucination and irrelevant citations.

METHODS

Fifteen prompts reflecting the 2022 AAOS ACLR CPG were curated and categorized into three domains: (1) preoperative evaluation and diagnosis, (2) surgical timing and technique, and (3) rehabilitation and prevention. Each prompt was entered into OE in a new session to minimize context bias. Two orthopedic surgeons rated response quality on a 5-point Likert scale for relevance, accuracy, clarity, completeness, and evidence-based content. Inter-rater reliability was assessed using intra-class correlation coefficients (ICCs). All citations were extracted and classified as relevant, irrelevant, or hallucinated, and journal sources and publication years were recorded.

RESULTS

Inter-rater reliability was high across all domains (overall ICC, 0.90; 95% confidence interval, 0.82-0.96). Median scores exceeded 4.0 for all domains, with perfect scores for evidence-based (5.0 [5.0-5.0]) and the lowest scores for clarity (5.0 [4.0-5.0]). Across 15 prompts, OE generated 127 PubMed-indexed citations (mean, 8.5 per response; range 6-11) with zero hallucinated and only 2.4% irrelevant citations. Citations spanned 29 journals from 1997 to 2025.

CONCLUSION

OE produced highly relevant, accurate, clear, complete, and evidence-based responses to AAOS ACLR CPG prompts, with trustworthy PubMed-indexed citations and no hallucinations. Its RAG architecture may offer a credible, citation-oriented adjunct to evidence-based decision support in orthopedic surgery.

LEVEL OF EVIDENCE: Level V, descriptive study.