Whole population cohorts vs sampled comparators designs for evaluating health and educational outcomes of children with inborn rare conditions: a simulation study.

OBJECTIVES

Linked administrative data covering whole populations are fundamental resources for longitudinal studies of children with rare conditions (cases) and unaffected peers (comparators). Data minimization regulations sometimes limit the number of sampled comparators per case (hereafter abbreviated SCn) for research, with unknown impact on study findings.

STUDY DESIGN AND

SETTING

Using Monte-Carlo draws, we simulated data from 100,000 children with and without an exemplar condition, congenital hypothyroidism (CHT), with sex and comorbidity as covariates. Three illustrative outcomes (Y: Maths tests standardized marks at age 11 years [z-score]; L: achieving expected Maths attainment [binary]; and T: months to neurodevelopmental disorder diagnosis) were modeled as linear combinations of CHT, sex, and comorbidity. Varying parameters (comorbidity prevalence, comorbidity-CHT association, and CHT effect on Y, L, and T) factorially produced 36 data-generating mechanisms (DGMs). Three sizes of CHT effect were explored: small (z-score ≤|0.03|, log odds/hazard ratios ≤|0.41|); medium (z-score = |0.3|, log odds/hazard ratios ≥|0.74|); and large (z-score ≥|0.6|, log odds/hazard ratios ≥|1.6|). Summary measures of regression coefficients, SEs, and P values from 1000 simulations were used to evaluate statistical power (percentage of P values ≤0.05), precision (relative variance), and bias (mean distance from true effect), comparing various SCn (where n= 5, 10, 15, 25, 50, and 100) vs full cohort (FC). Percentage differences of ≤2% and 2% to 5% (from FC value) were considered noninferior and satisfactory, respectively.

RESULTS

The mean P values for SC25 generally deviated ≤5% vs FC with medium effects, and ≤2% for large effects across all DGMs. For Y, L, and T, none of the SCn nor FC had power ≥80% with small or medium effects, while all SCn had sufficient power with large effects. Compared with FC, precision for SC25 decreased by 1.3%-5.0% for Y, 5.5%-7.4% for L, and 8.6%-10.6% for T in one DGM (comorbidity prevalence = 0.001 and CHT was 10× more common in those with comorbidity vs those without). No association between SCn and bias was discernible throughout.

CONCLUSION

SC25 provided comparable performance as FC for rare disease studies under several typical scenarios, but small effects posed challenges, notwithstanding sampling. This approach facilitates cost-effective recommendations for study design where whole population data are not available.

PLAIN LANGUAGE SUMMARY: Studies investigating long-term outcomes of children with rare conditions are often most reliable when they use data from the entire population. However, data protection regulations sometimes limit how many children without the condition can be used for comparison. We do not exactly know how these limits affect study results. We used computer simulations to generate many sets of fictional data containing information on 100,000 children, including their sex, whether they had a rare condition called congenital hypothyroidism (CHT) and some example outcomes: Maths test marks at age 11 years, whether they achieved the expected standard in Maths, and how long it took for children to be diagnosed with neurodevelopmental conditions. We explored several scenarios such as how common other health conditions were, how strongly these were linked to CHT, and how outcomes differed between children with and without CHT. We compared results using all children and when we randomly selected between 5 and 100 comparison children (without CHT) per child with CHT. We assessed how likely each sample could detect true differences between the two groups, how spread out the estimated differences were, and whether estimates were consistently too high or low (bias). We found that generally using 25 comparison children per CHT child produced results that were very similar to those using all children (less than 5% deviation), especially when group differences were considerable. Reducing the number of comparison children did not lead to noticeable bias. However, when the true differences were small, all samples, including the full population, struggled to detect group differences. Overall, we found that when full population data are not available, using at least 25 comparison children per CHT child can still provide acceptable results. This offers a systematic and cost-effective way to design studies while complying with data protection requirements.