Quality and methodological heterogeneity of COVID-19 vaccine safety studies focusing on the myocarditis safety signal: A systematic review, meta-analysis and meta-regression.

BACKGROUND

The risk of myocarditis following COVID-19 vaccines has been widely investigated, yielding highly variable effect estimates. It remains unclear how much of this variation can be explained by methodological differences and study quality. This study aimed to evaluate the methodology of observational studies on myocarditis risk post-COVID-19 vaccination and identify sources of heterogeneity.

METHODS

We systematically searched PubMed and EMBASE for observational studies reporting relative risks of myocarditis after COVID-19 vaccination published between December 2020 and October 2023. Risk of Bias (RoB) was assessed using the ROBINS-I tool. Meta-analysis and multivariable meta-regression were conducted for studies addressing comparable population-intervention-comparator-outcome (PICO) questions.

RESULTS

We included 30 studies, comprising 35 design-specific analyses. We identified considerable variability in design elements including risk window length (7-288 days), reference period timing (three different ways), and control for COVID-19 disease (five different approaches). Thirteen (37%) design-specific analyses had serious or critical overall RoB. We observed strongest heterogeneity between studies in general population for dose 2 of BNT162b2 and mRNA-1273 compared to unvaccinated individuals/reference period (I2 = 96%, prediction interval (PI) of relative risk 0.40-20.20; I2 = 92%, PI 1.23-88.63, respectively). Meta-regression for the former PICO indicated that after adjusting for age and sex, effect estimates of samples with 28-day risk window were 0.56 times (95% CI 0.43-0.72) lower than those with 7-day risk window, but its overall effect was not significant. The effect of study design, outcome definition, approaches to handle COVID-19 infection and overall RoB were not significant in the meta-regression.

CONCLUSIONS

There is substantial variation in study design specifications and corresponding heterogeneity in reported effect estimates. Design choices like risk window length may explain some of this heterogeneity, although evidence remains inconclusive. Future vaccine safety studies should include sensitivity analyses to explore the effect of design choices on their findings.