Use of Artificial Intelligence-Based Platforms to Support Searches in Evidence Synthesis Studies: A Study of the EAACI Methodology Committee.

OBJECTIVE

To demonstrate an evaluation of the use of artificial intelligence (AI) in supporting tasks in evidence synthesis (particularly search for primary studies) in the allergy or respirology fields.

METHODS

We queried three AI-based platforms specialised on identifying scientific publications using several strategies (i.e., prompting strategies and search approaches) in November 2024 to identify primary studies related to health-related case studies in the allergy or respirology field. We compared how many eligible primary studies we identified using AI-based platforms versus in a systematic search in multiple electronic bibliographic databases. We also compared meta-analytical results obtained with the primary studies identified by querying AI-based platforms versus in the context of a systematic review. Finally, we developed a structured methodological framework and a reporting checklist for using these AI-based platforms.

RESULTS

In our main case study, a systematic review of randomised controlled trials (RCTs) informing the Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines, a strategy involving searching multiple AI-based platforms identified 85.7% of all full articles with DOI, but failed to identify unpublished trials registered in trial databases, resulting in an overall identification of 56.3% eligible RCTs. Meta-analytical estimates were similar when considering only the primary studies identified using AI-based strategies versus those identified by the systematic review. We observed a lower performance in the case study of systematic reviews of observational studies.

CONCLUSIONS

We provide an example on how it is possible to evaluate AI-based platforms in the support of evidence synthesis, with a focus on the allergy and respirology fields.