The hunt for the last relevant paper: blending the best of humans and AI.

Background

The exponential growth of research literature makes it increasingly difficult to identify all relevant studies for systematic reviews and meta-analyses. While traditional search methods are labour-intensive, modern AI-aided approaches have the potential to act as a powerful 'super-assistant' during both the searching and screening phases.

Objective

This paper evaluates how a combined, open-source approach - merging traditional and AI-aided search and screening methods - can help identify all relevant literature up to the 'last relevant paper' for a systematic review on post-traumatic stress symptom (PTSS) trajectories after traumatic events.Method: We applied eight search strategies, including database searches, snowballing, full-text retrieval, and semantic search via OpenAlex. All records were screened using a combination of human reviewers, active learning, and large language models (LLMs) for quality control.

Results

On top of replicating the original 6,701 search results, we identified an additional 3,822 records using AI-aided methods. The combination of AI tools and human screening led to 126 relevant studies, with each method uncovering papers the others missed. Notably, machine-aided techniques helped find studies with missing keywords, unusual phrasing, or limited indexing. Across all AI-assisted strategies, 10 additional studies were identified, and while the overall yield was modest, these papers were unique and relevant and would likely have been missed using traditional methods.

Conclusions

Our findings demonstrate that even when returns are low, AI-aided approaches can meaningfully enhance coverage and offer a scalable path forward when combined with screening prioritisation. A transparent, hybrid workflow where AI serves as a 'super-assistant' can meaningfully extend the reach of systematic reviews and increase the quality of the findings, but is not ready to replace humans fully.