Inter-rater reliability during live pediatric formative Objective Structured Clinical Examination (OSCE) in a French University.

BACKGROUND AND

OBJECTIVE

In France, OSCE (Objective Structured Clinical Examination) have recently been incorporated as an official assessment method for the final National Ranking Examination of undergraduate medical students. Given these high stakes, it is of paramount importance to provide high-quality OSCE training in every discipline. Our pediatrics department has developed an OSCE training program covering all areas of pediatric specialties. Students are evaluated by two independent assessors for each station. As part of a quality assessment process, this study investigated inter-rater reliability in OSCE grading between the two assessors.

MATERIALS AND

METHODS

This prospective observational study was conducted from March to June 2023 in the Department of Pediatrics of Sorbonne University (Paris, France). Undergraduate medical students were evaluated during OSCE training sessions. Each student had to complete two consecutive stations. Each station was evaluated by two independent assessors. The scoring checklist was divided into two parts: "clinical skills" and "general skills," assessing the student's behavior and relational aptitudes. Evaluation scales were dichotomous for all items.

RESULTS

About 206 fifth-year medical students participated in four distinct OSCE sessions. A total of 411 stations were evaluated by 20 pairs of assessors, resulting in 822 different scores. The intraclass correlation coefficients of "total score," "clinical skills," and "general skills" scores between both assessors were 0.83 [0.64-0.93], 0.85 [0.68-0.94], and 0.50 [0.14-0.75], respectively. Inter-rater reliability was influenced by overall OSCE score and varied significantly between OSCE scenarios. Variability was greater in pediatric tuberculosis and hemolytic uremic syndrome scenarios, compared to bronchiolitis.

CONCLUSIONS

During live pediatric training OSCE, inter-rater reliability was good regarding clinical skills, and moderate for general skills. Inter-rater variability might be more pronounced for students with lower overall OSCE score. Our study emphasizes the importance of a careful drafting of OSCE cases with precise and clear-cut scoring checklist, especially in stations focusing on very specialized clinical situations.