Decomposing performance inflation in radiomics: factorial quantification of analytical leakage.

PURPOSE: To quantify the cumulative performance inflation produced by analytical decisions leaked outside nested cross-validation in radiomics, and to decompose it by decision type.

METHODS

We compared six pipeline configurations with progressively leaked analytical decisions using Monte Carlo simulation (3 sample sizes x 3 feature dimensions x 2 scenarios, 100 repetitions each) and validated findings on real radiomic features from The Cancer Imaging Archive (TCIA) NSCLC-Radiomics dataset (n = 726, 111 features, 30 permutation-null repetitions). The primary outcome was AUC inflation (fully leaked pipeline minus properly nested pipeline) with bootstrap 95% confidence intervals. Secondary outcomes included false positive rate, additive decomposition by decision type, and concordance between synthetic and real-data profiles (Spearman rho, ICC).

RESULTS

Inflation was statistically significant across all 19 tested conditions (all p < 0.001). At n = 50 and p = 1,000, the fully leaked pipeline returned a mean AUC of 1.000 [95% CI: 1.000-1.000] on pure noise, versus 0.493 [0.472-0.514] for the nested reference, a difference of 0.507 [0.486-0.528]. Feature selection leakage accounted for the bulk of this effect (60-92% across null and weak-signal conditions), outweighing hyperparameter tuning by a factor of 4 to 8. The same staircase pattern appeared on real radiomic features (Spearman rho = 0.943, p = 0.005). Every leaked-pipeline experiment yielded an AUC above 0.60, in every synthetic condition, even though no signal existed.

CONCLUSION

Leaked analytical decisions inflate radiomics performance cumulatively and severely, with feature selection leakage alone driving most of the effect. On real data, leaked feature selection paradoxically lowered generalization performance. All analytical decisions must be enclosed within the validation loop. Studies should report their full analytical search space, and reviewers should verify that feature selection, hyperparameter tuning, and model comparison were all nested.