Psychometric properties of instruments for assessing the methodological quality and risk of bias in non-randomized studies of interventions: a systematic review and meta-analysis in accordance with the COSMIN guideline.

OBJECTIVES: Nonrandomized studies of interventions (NRSIs) provide valuable evidence in health sciences but are prone to biases that affect validity. Multiple instruments have been developed for critical appraisal, although their measurement properties remain insufficiently investigated. These instruments typically aim to evaluate two key theoretical constructs: methodological quality (MQ) and risk of bias (RoB), which reflect different but complementary dimensions of study rigor and internal validity. The COnsensus-basedStandards for the selection of health Measurement INstruments (COSMIN) framework offers internationally recognized standards for evaluating instrument robustness, thus enhancing transparency and comparability in their selection. This systematic review specifically focused on identifying and critically evaluating studies that have empirically tested the measurement properties of instruments developed to assess MQ and/or RoB in NRSIs.

STUDY DESIGN: A systematic review and meta-analysis was conducted in accordance with the COSMIN initiative, which is structured in three parts: (1) a literature search reported according to the Preferred Reporting Items for Systematic Reviews andMeta-Analyses (PRISMA) and the PRISMA-COSMIN search extension; (2) an assessment of methodological quality (MQ-COSMIN) and measurement properties using the COSMIN Risk of Bias tool, together with an evaluation of the certainty of evidence (CoE) following the Grading of Recommendations Assessment,Development, and Evaluation(GRADE), and (3) a meta-analysis of measurement properties when sufficient quantitative data were available across validation studies for a given instrument.

RESULTS: Eleven instruments for critical appraisal of NRSIs were identified. None were evaluated for all COSMIN measurement properties; instrument development, content validity, and reliability were most consistently reported. The Methodological Index forNon-RandomizedStudies (MINORS), the Modified Medical Education Research Study Quality Instrument (MMERSQI), and the AcomprehensSive tool tosupport rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes (ASSESS) showed the highest quality evidence for MQ assesment, while the Risk of Bias inNon-randomisedStudies of InterventionsI (ROBINS-I) provided the strongest evidence for RoB assessment. For instruments with sufficient comparable data, such as ROBINS-I, a meta-analysis of inter-rater reliability coefficients was conducted, showing moderate agreement for selection and exposure domains, with substantial heterogeneity across studies.

CONCLUSION: MINORS emerged as the most robust instrument for critical appraisal of MQ, whereas ROBINS-I stood out for RoB. ASSESS and MMERSQI provided adequate and sufficient evidence of content validity. Further assessment of different psychometric properties is needed, highlighting that only a small subset of available tools for NRSIs have undergone formal psychometric validation. Further research is needed to strengthen the measurement evidence base for these instruments.