DOI 10.1016/j.jclinepi.2026.112441
Single-reviewer assessments of certainty of evidence in network meta-analyses probably do not mislead decision-makers: a case study.
Аннотация
BACKGROUND
Systematic reviews with network meta-analyses (NMA) are the most comprehensive form of evidence synthesis for informing evidence-based practice recommendations. Rating the certainty of evidence (CoE) is a crucial step for drawing appropriate NMA's conclusions, but it can be time-consuming.
OBJECTIVES
To evaluate the disagreements of assessments by a single reviewer (novice or expert) v consensus among two reviewers on CoE ratings of NMA used to support practice recommendations.
METHODS
We conducted a methodological case study using the grading of recommendations, assessment, development, and evaluations (GRADE) assessments for each comparison and outcome of interest from the COVID-19 living NMA, which informed the World Health Organization recommendations. Expert and novice reviewers assessed CoE independently and resolved disagreements by consensus. We defined an "important disagreement" in CoE ratings as any disagreement that would lead to an increase from high/moderate to low/very low or vice versa. We calculated the proportion of outcomes with important disagreements comparing the CoE rating from a single vs two reviewers. We performed a generalized linear mixed model analysis to compare the relative odds of disagreement between expert and novice reviewers.
RESULTS
Six experts and five novice reviewers rated 6652 comparisons. Compared with consensus ratings, novice reviewers exhibited higher disagreement rates across all GRADE domains, including assessments of direct, indirect, and network evidence. The disagreement rates for the overall CoE rating assessed by novice reviewers were higher than those of experts (10.6 vs 2.3%), but the important disagreement rate was similar (1.2% vs 1.0%). Novice reviewers were more likely to make important disagreements, but the association was not statistically significant (odds ratio, 1.3; 95% confidence interval 0.9-1.8).
CONCLUSION
Novice reviewers disagreed with the consensus more often than expert reviewers when assessing the CoE from NMA. However, having a single reviewer was associated with relatively few disagreements likely to have meaningful implications for decision-makers.
Подготовка и перевод публикации
Обработка статьи · этапы, ошибки и управление
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Тип публикации указан по данным источника. Уровень доказательности автоматически не присваивается.