Associations between an open-response situational judgment test and performance on OSCEs and fieldwork: implications for admissions decisions and matriculant diversity in an occupational therapy program.

BACKGROUND: Casper, an online open-response situational judgement test that assesses social intelligence and professionalism [1], is used in admissions to health professions programs.

METHOD: This study (1) explored the incremental validity of Casper over grade point average (GPA) for predicting student performance on objective structured clinical examinations (OSCEs) and fieldwork placements within an occupational therapy program, (2) examined optimal weighting of Casper in GPA in admissions decisions using non-linear optimization and regression tree analysis to find the weights associated with the highest average competency scores, and (3) investigated whether Casper could be used to impact the diversity of a cohort selected for admission to the program.

RESULTS: Multiple regression analysis results indicate that Casper improves the prediction of OSCE and fieldwork performance over and above GPA (change in Adj. R2 = 3.2%). Non-linear optimization and regression tree analysis indicate the optimal weights of GPA and Casper for predicting performance across fieldwork placements are 0.16 and 0.84, respectively. Furthermore, the findings suggest that students with a slightly lower GPA (e.g., 3.5-3.6) could be successful in the program as assessed by fieldwork, which is considered to be the strongest indicator of success as an entry-level clinician. In terms of diversity, no statistically significant differences were found between those actually admitted and those who would have been admitted using Casper.

CONCLUSION: These results constitute preliminary validity evidence supporting the integration of Casper into applicant selection in an occupational therapy graduate program.