Evaluating the validity and performance of a modifiable prediction model to predict new long-term opioid use in prescription drug monitoring programs.

OBJECTIVE

Prescription drug monitoring programs (PDMPs) offer a promising data source for the development of clinical decision support (CDS) systems to guide opioid prescribing, but most PDMP agencies lack capacity for in-house model development-instead relying on third-party tools. This study aims to externally validate a PDMP-based model, developed using 2018 California PDMP records, to predict new long-term use among Kentucky opioid-naïve patients and compare its performance to Kentucky-specific machine learning (ML) models.

MATERIALS AND

METHODS

We identified opioid-naïve Kentucky residents who filled an index pill opioid prescription between 2021 and 2023 (development cohort: 2021-2022; validation cohort: 2023). The California model was updated via logistic calibration and evaluated against 5 newly developed Kentucky ML models (least absolute shrinkage and selection operator [LASSO], elastic net, deep neural network, random forest, XGBoost) using discrimination and calibration metrics.

RESULTS

Among 985 317 opioid-naïve patients, 33 298 (3.4%) transitioned to new long-term opioid use. The updated California model achieved excellent discrimination (area under the receiver operating characteristic curve [AUROC]: 0.919 [0.915-0.923]) and was calibrated up to the highest risk levels (calibration intercept: 0.010, [-0.021, 0.041]; slope: 0.997, [0.979, 1.014]), with negative predictive values exceeding 0.99. Performance was comparable to Kentucky-specific ML models, including the highest performer XGBoost (AUROC: 0.930 [0.926-0.933]). All models struggled with low positive predictive values (0.15-0.19).

CONCLUSION

A California PDMP-derived prediction model transported successfully to Kentucky with minimal updating. Its strong comparative performance provides a feasible foundation for PDMP agencies to develop transparent, in-house CDS systems with potential clinical implications for safer and more individualized opioid prescribing.