An exploratory descriptive survey on the use of GRADE and CINeMA: time-consuming, process transparency and subjectivity vs high-speed, practical challenges, and poor understanding.

OBJECTIVES: To explore assessors' experience regarding the use of two approaches that evaluate the certainty of evidence from Network Meta-Analysis (NMA): Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence In Network Meta-Analysis (CINeMA).

STUDY DESIGN AND SETTING: Thirteen assessors trained in NMA methodology and evaluation methods were randomly assigned to four groups to apply GRADE and CINeMA to two networks each to compare the approaches' concordance, inter-rater reliability, and application time (Minozzi et al, 2025). Assessors' experiences were collected through an online questionnaire including open-ended questions. The questionnaire explored usability, strengths, weaknesses, and perceived accuracy of each approach. Each assessor was surveyed twice: once after GRADE use and once after CINeMA use. Thematic analysis was used to analyze their responses.

RESULTS: We identified seven themes: instruments' challenges, knowledge need, process execution, positive aspects, interpretative uncertainty, implementation, and level of confidence. GRADE assessment was considered long because of the extensive material to manage and at high risk of subjectivity but with a transparent and clear process. CINeMA automatization allowed a quick and easy use but impacted negatively on the process methodological understanding. Technical challenges related to website and files management were identified. Overall, assessors highlighted the need for further training on tool usage.

CONCLUSION: GRADE was considered time-consuming and at high risk of subjectivity but excelled in transparency and clarity. CINeMA was considered high-speed but showed some practical and output-interpretation challenges, particularly where specialized knowledge of NMA methods was required. Additional training courses were suggested for both tools.

PLAIN LANGUAGE SUMMARY: Researchers use NMA to compare multiple medical treatments. To rate how trustworthy the results are, they can choose between two tools: Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence In Network Meta-Analysis (CINeMA). We asked thirteen researchers who had used both tools to describe their experiences in a survey. Their feedback highlighted a key trade-off. GRADE was praised for being transparent and educational, as its manual process helped assessors understanding each step. However, it was consistently described as very time-consuming and somewhat subjective. CINeMA was appreciated for its speed and ease of use, thanks to its automated, web-based system. Yet, this automation made the process feel like a "black box," reducing assessors' understanding on how conclusions were reached. Practical issues, like file upload problems, were also a hurdle. A common theme was that both tools require solid training and background knowledge to be used correctly. In short, GRADE offers depth and clarity at the cost of time, whereas CINeMA offers efficiency at the cost of transparency. These insights can help guide future training and improve these important research tools.