Agreement between heuristic shrinkage factor and optimal shrinkage factors in logistic regression for risk prediction: a simulation study across different sample sizes and settings.

INTRODUCTION: The heuristic shrinkage factor of Van Houwelingen and Le Cessie (𝑆𝑉𝐻) is a commonly used closed-form solution to adjust for overfitting in unpenalised logistic regression models for risk prediction. It is also the basis of widely-adopted minimum sample size criteria for developing clinical prediction models. However, current evidence is lacking regarding the bias of 𝑆𝑉𝐻 compared to the optimal shrinkage factor (𝑆𝑜𝑝𝑡). Here, we examine this issue and also assess the bias of an alternative bootstrap-derived shrinkage factor (𝑆𝑏𝑜𝑜𝑡).

METHODS: We undertook two simulation studies. The first examined the bias of 𝑆𝑉𝐻 and 𝑆𝑏𝑜𝑜𝑡 as estimators of 𝑆𝑜𝑝𝑡 across a range of different scenarios defined by 𝐶𝑝𝑜𝑝, the C-statistic of the model developed in a population sized dataset. The second examined the bias of 𝑆𝑜𝑝𝑡 when using development sample sizes targeting a shrinkage of 0.9, based on a sample size calculation defined by 𝑆𝑉𝐻 itself (𝑁𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙) or by an adapted simulation-based approach (𝑁𝑠𝑖𝑚).

RESULTS: For high C-statistics, 𝑆𝑉𝐻 overestimated 𝑆𝑜𝑝𝑡, whereas for low C-statistics 𝑆𝑉𝐻 underestimates 𝑆𝑜𝑝𝑡. For example, across scenarios when 0.8≤𝐶𝑝𝑜𝑝<0.85, the 95-percentile range in the bias was (0.005,0.387), compared to (−0.580,−0.007) across scenarios when 0.6≤𝐶𝑝𝑜𝑝<0.65. The magnitude of bias increased as 𝐶𝑝𝑜𝑝 tended to either 0.5 or 1. As sample size increased and 𝑆𝑜𝑝𝑡→1, the magnitude of the bias in either direction reduced. 𝑆𝑏𝑜𝑜𝑡 was less biased than 𝑆𝑉𝐻, with a median magnitude of bias across all scenarios of 0.007, compared to 0.032 for 𝑆𝑉𝐻. Developing models on datasets of size 𝑁𝑠𝑖𝑚 gave 𝑚𝑒𝑎𝑛(𝑆𝑜𝑝𝑡) closer to 0.9 (mean magnitude of bias across all scenarios 0.004) than 𝑁𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 (mean magnitude of bias 0.041).

CONCLUSIONS: 𝑆𝑉𝐻 is often a poor estimator of the optimal global shrinkage factor. If global shrinkage is needed, we recommend using the bootstrap shrinkage estimate. The bootstrap estimate shows minimal bias in most scenarios, though in small samples the variability is large so provides no guarantees to address overfitting in a single dataset. A sample size calculation based on simulation is often preferable over formula dependent on targeting 𝑆𝑉𝐻.

SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s41512-026-00222-1.