Machine Learning-Based Adjudication of Acute Infection for Diagnostic Classifier Development with Silver-Standard Labels.

To produce training labels for diagnostic classifiers of acute infection for sepsis, an adjudication process involving manual chart review by one or more clinicians is current standard practice. This process is costly and varies from clinician to clinician. Acute infections can be difficult to definitively diagnose with resulting 'silver-standard' adjudications depending on clinician's subjective judgement. We leverage an international, multi-cohort dataset of multi-clinician-adjudicated acute infections to develop and evaluate a machine learning (ML)-based system for automated adjudication. We tested the performance and robustness to missing data of several off-the-shelf ML models in a benchmark study, tested the calibration of their output probabilities, and assessed what minimal set of features would be required to maintain performance in future studies. The resulting system, based on a logistic regression model, is accurate and well-calibrated enabling low-cost, reproducible and highly scalable adjudication of acute infection for downstream diagnostic classifier development.Clinical relevance-This work aims to automate the resource-intensive and variable process of chart review for adjudication of acute infection. Our approach makes adjudication of challenging, 'silver standard' acute infections reproducible and scalable to future studies, enabling more rapid, cost-effective incorporation of these studies in downstream development of diagnostic classifiers for sepsis. In addition, automating the chart review process could allow clinicians to spend more time on patient care.