Construct validity and reliability of structured assessment of endoVascular expertise in a simulated setting.

OBJECTIVES

To study the construct validity and reliability of a novel endovascular global rating scale, Structured Assessment of endoVascular Expertise (SAVE).

DESIGN

A Clinical, experimental study.

MATERIALS: Twenty physicians with endovascular experiences ranging from complete novices to highly experienced operators performed a video-recorded simulated contra-lateral iliac-artery-stenting procedure. The virtual-patient case was a novel technically challenging procedure presenting the distal arteries below the knee.

METHODS

Three experts assessed the performances blinded to operator identity. Validity was analysed by correlating experience with performance results. Reliability was analysed according to generalisability theory.

RESULTS

The mean score on the 29 items of the SAVE scale correlated well with clinical experience (R = 0.84, P < 0.01) and was found discriminative even among the more experienced participants having performed up to 500 endovascular procedures in total. Only the most experienced participants (>5000 procedures) obtained maximum scores. The inter-rater reliability was high (G = 0.94 and G = 0.95). The procedure time (median 69 min, range 32-86) correlated moderately with clinical experience (R = -0.53, P < 0.05), whereas the fluoroscopy time and amount of contrast fluid did not correlate.

CONCLUSIONS

The construct validity and reliability of assessment with the SAVE scale was high when applied to performances in a simulation setting with advanced realism. No ceiling effect was present in the assessment situation.