Can Graduate Students Rate Personality Reliably in Psychoanalytic Treatments Using the Shedler-Westen Assessment Procedure (SWAP-200)?

Introduction: Psychotherapy researchers often recruit students to code psychotherapy process and outcome variables on individual therapy sessions. To do so, graduate students read verbatim psychotherapy transcripts, listen to audio or watch video of psychotherapy sessions, and then rate process and outcome measures on these sessions. Although prior studies have investigated the reliability and validity of graduate student ratings of personality dysfunction, no current study has investigated the reliability and validity of these ratings made on the basis of psychotherapy transcripts and audio. In this study, we evaluated the degree to which graduate students can reliably and validly code observer-rated personality assessments on the Shedler-Westen Assessment Procedure-200 (SWAP-200). Methods: We related graduate student and experienced clinician-researcher's SWAP-200 scores in an existing dataset of 27 patients undergoing psychoanalytic psychotherapy at early and late phases of treatment. Results: Using truth and bias multilevel models, we found that graduate students tended to underestimate personality pathology at the early phase of treatment and overestimate pathology at the late phase in treatment. These responses resulted in smaller effect sizes in the graduate student ratings, such that patients' personality functioning did not appear to change following treatment. By contrast, the experienced clinician's scores supported moderate to large effect sizes in personality change. Discussion: These deviations from expert judgment may have significant ramifications for evaluating pre and post effect sizes in psychotherapy process studies using graduate student coding. Suggestions for research and practice are discussed.