When does machine learning outperform clinicians? A comparison of prediction accuracy for PTSD treatment outcomes.

BACKGROUND

Machine learning (ML) models show promise in predicting post-traumatic stress disorder (PTSD) treatment outcomes, but it is unknown how their predictions compare to those of clinicians. This study directly compared the accuracy of clinicians' predictions of patient treatment outcomes with those of three ML models.

METHODS

Twenty clinicians providing cognitive processing therapy repeatedly predicted outcomes for 194 veterans. We compared their accuracy against three ML models on two key endpoints: clinically meaningful symptom reduction (≥10-point PCL-5 decrease) and posttreatment severity (final PCL-5 < 33). Clinician predictions were compared against a recurrent neural network, a mixed-effects random forest, and a generalized linear mixed-effects model. We analyzed prediction accuracy and the association between clinician confidence and accuracy using logistic mixed-effects models.

RESULTS

ML models were significantly more accurate than clinicians at predicting whether a patient's posttreatment PCL-5 score would be below 33 (p < .001). However, no significant difference in accuracy was found for predicting a ≥10-point symptom reduction (p = .734). Clinician confidence increased throughout treatment and was significantly associated with greater prediction accuracy for both outcomes (ORs = 1.06, ps < .001).

CONCLUSIONS

ML models can outperform clinicians in predicting posttreatment symptom severity, particularly early in treatment, suggesting they could be a useful tool for identifying patients at risk for suboptimal outcomes. However, ML models were not superior in predicting symptom reduction, where clinicians also performed at a high level. Findings support the selective use of ML to enhance, rather than replace, clinical judgment in PTSD treatment.