Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews.

BACKGROUND

Systematic reviews and meta-analyses represent the highest level of evidence in clinical research, but the process of article retrieval and screening is labor-intensive. Large language models, such as ChatGPT-5, may offer an efficient alternative, yet their performance in full systematic review workflows remains untested. This study compares ChatGPT-5's Deep Research and Agent Modes with human researchers in replicating gold standard systematic reviews in total joint arthroplasty.

METHODS

Five published systematic reviews were selected as reference articles. Three groups: orthopaedic research fellows, ChatGPT-5 Deep Research Mode, and ChatGPT-5 Agent Mode, independently identified eligible articles using standardized search terms and inclusion/exclusion criteria. Artificial intelligence (AI) searches were repeated 3 times for reproducibility. Extracted articles were evaluated against the gold standard for recall, precision, false positives/negatives, and time efficiency. Newly identified eligible studies were also assessed.

RESULTS

The research fellows dedicated 268 hours to screening 9101 articles, achieving 85.2% recall of gold standard articles. Deep Research and Agent Modes averaged 12-14 minutes per search, identifying 47.5% and 40.9% of gold standard articles, respectively. Fellows had fewer false negatives (n = 5) compared with Deep Research (n = 19) and Agent Mode (n = 12). AI models retrieved several additional eligible studies not captured by humans, demonstrating complementary potential.

CONCLUSIONS

Human reviewers remain superior to current AI models in replicating systematic review article selection, particularly for nuanced inclusion/exclusion criteria. However, ChatGPT-5 significantly reduces search time and can identify additional relevant studies, suggesting its role as a valuable adjunct in systematic review workflows with expert oversight.