Optimizing hard cutoff quality control metrics for IBD segment inference and genotyping accuracy in low-coverage sequencing data.

Effective quality control (QC) is critical for reliable identity-by-descent (IBD) inference from low-pass whole-genome sequencing (LPWGS), where genotype uncertainty under low coverage can bias relatedness estimates. We conducted a comprehensive grid search to evaluate genotype posterior probability (GP) and Bayes factor (BF) thresholds across multiple coverage levels, assessing performance by false positive rate, false negative rate, and mean squared error (MSE) of total IBD sharing. At moderate-to-high coverage (> 0.10 ×), GP > 0.99 maintained stable performance and additional BF filtering provided little benefit; at low coverage (≤ 0.10 ×), GP-only filtering substantially increased false positive IBD inference. Incorporating BF mitigated this effect by contrasting posterior genotype support against population allele frequency priors. Increasing the BF threshold from 0.3 to 100 reduced false positive rate by approximately 35%, whereas higher thresholds (≥ 200) provided minimal additional reduction in false positive rate while increasing false negative rate. BF > 100 balanced false positive reduction with acceptable marker retention. Overall, increasing GP thresholds alone did not consistently improve IBD inference in LPWGS. Integrating BF with GP improved inference stability under low coverage conditions and supported a practical unified QC strategy for low-to-moderate coverage data.