Evaluating the impact of methodological trade-offs on meta-analyses and certainty of the evidence according to GRADE: part 5 of a meta-research study of rapid reviews.

OBJECTIVES

To determine whether methodological trade-offs used in rapid reviews result in changes in treatment effects or Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) certainty of the evidence (COE) compared to systematic reviews.

STUDY DESIGN AND

SETTING

We used 3 systematic reviews on acupuncture, education, and transcutaneous electrical nerve stimulation (TENS) for the management of chronic low back pain (LBP) conducted to inform the World Health Organization (WHO) guideline on nonsurgical management of chronic primary LBP and simulated the conduct of 3 rapid reviews. We assessed the impact of 3 commonly used trade-offs: restricted database selection, single reviewer screening, and single reviewer risk of bias assessment. We meta-analyzed studies included for each rapid review and compared them to the results of the WHO systematic reviews and identified when treatment effects would change recommendations. We used evidence profiles to update any changes to GRADE criteria and resulting COE.

RESULTS

Compared to the WHO systematic reviews, our rapid reviews resulted in no changes to recommendations based on the meta-analyses in the acupuncture or TENS reviews and 15.4% in the education review. Further, between 2.9% and 51.6% of descriptive syntheses were not performed because the single study in the analysis were excluded from the rapid review, 25.8% of descriptive syntheses became meta-analyses because of false positives from single reviewer screening, and 6.45%-13.33% of meta-analyses became descriptive syntheses because the review was reduced to a single study. The COE was affected in 13.3% in the TENS review and 51.6% in the education review due to descriptive syntheses that were not performed (exclusion of the single study in the analysis). In the acupuncture review, the COE changed in 23.5% of analyses.

CONCLUSION

Rapid review methodological trade-offs led to few changes in treatment effects and COE. However, it is difficult to predict how they will impact quantitative and descriptive syntheses and COE because multiple factors are involved, including the breadth of the research question, volume and quality of studies, and review topic. When possible, review authors should consider few comparators and outcomes, well defined eligibility criteria and using fewer methodological trade-offs.

PLAIN LANGUAGE SUMMARY: Rapid reviews synthesize evidence more efficiently than systematic reviews, but they use trade-offs, including searching fewer databases or using 1 instead of 2 reviewers. Our study investigated how these trade-offs affect the findings of rapid reviews on 3 treatments for chronic low back pain (acupuncture, education, and transcutaneous electrical nerve stimulation [TENS]). We reanalyzed 3 systematic reviews using 3 commonly used trade-offs, including limiting the database search, having 1 reviewer select studies instead of 2, and having 1 reviewer assess study quality. We then evaluated how these changes affected the validity of results and our confidence in the review conclusions. We found that these trade-offs changed few results. Up to 15.4% of the estimates from meta-analyses resulted in a change in recommendations for the education review, but none for the acupuncture or TENS reviews. In some cases, analyses could no longer be done because the study was missed, while others had to be described instead of analyzed statistically. The confidence in the evidence changed in 13% to 52% of analyses. In the acupuncture review, confidence in the evidence increased or decreased. In the education and TENS reviews, confidence in the evidence changed (to not applicable) in the analyses that could not be replicated due to missing studies. In summary, our findings suggest that rapid review trade-offs do not change many recommendations or the confidence in the evidence compared to full systematic reviews, except when there are few studies included in the analysis. The impact of using trade-offs depend on the research question, available evidence, and the quality of the studies. To improve the trustworthiness of rapid reviews, researchers should focus on a clearly defined research question, clear criteria for including studies, and limit the number of trade-offs.