External validation of GDM risk prediction models using a machine learning reciprocal model-exchange framework.

BACKGROUND

Although many risk prediction models have been developed, very few undergo external validation, primarily due to issues with data access. Therefore, we implemented a reciprocal model-exchange approach to facilitate external validation and demonstrate its use with gestational diabetes mellitus (GDM) prediction models.

OBJECTIVE

To assess the robustness and generalisability of two independently developed GDM risk prediction models using a reciprocal model-exchange framework.

METHODS

Two independently developed GDM risk prediction models were externally validated using a reciprocal model-exchange. The saved model's corresponding variable types and data pre-processor were exchanged. The Monash CatBoost model was validated using Irish data at Dublin City University (DCU), and the DCU logistic-regression GDM model was validated using Australian data at Monash University. Performance was assessed using discrimination, calibration and decision curve analysis. Model fairness was assessed.

RESULTS

The prevalence of GDM was 21.1% in the Australian cohort and 11.7% in the Irish cohort. The Monash model's AUC dropped from 0.93 to 0.77, while the DCU model's AUC fell from 0.82 to 0.69. Calibration estimates confirmed systematic risk misestimation; each model tends to over or under-predict GDM probabilities outside its training domain, with calibration-in-the-large of -0.573 for the Monash model and 0.17 for the DCU model; slopes were 1.278 and 0.55 respectively. Both models showed performance variability across ethnic groups, with lower performance for Southeast/Northeast Asians and both performed better with increasing parity and among women without a prior GDM diagnosis.

CONCLUSIONS

Each model's performance decreased upon external validation, and the fairness evaluations on the different sub-categories (ethnicities; parity and previous GDM) provided evidence on the areas to be addressed in model recalibration/updating before deployment can be progressed. This reciprocal model-exchange approach provides a solution to facilitating external validations, which are notably lacking in the current literature but are necessary to advance the risk prediction field.