Clinician Performance in Training Data Curation for an Arrhythmia Machine Learning Model: Is Anyone Qualified?

BACKGROUND: Arrhythmia identification in the intensive care unit (ICU) is important to prevent ICU morbidity and mortality. Timely arrhythmia detection relies on bedside providers' telemetry interpretation. Machine learning models can function as clinical support tools to facilitate diagnoses. Machine learning model development requires well-curated training data. The differential performance between labelers of different roles and experience is currently unknown.

METHODS: This was a prospective observational study with frontline providers. A total of 300 (200 original and 100 duplicate) telemetry tracings were labeled as sinus rhythm, second-/third-degree atrioventricular block, junctional ectopic tachycardia, ectopic atrial tachycardia, and reentrant supraventricular tachycardia. Inter-rater reliability was calculated against the ground truth label (read by an electrophysiologist, AB) as the primary performance measure (intrarater reliability for consistency using duplicate labels).

RESULTS: A total of 11 participants completed the study: 1 cardiology fellow, 4 pediatric ICU fellows, 2 pediatric cardiac ICU fellows, 3 pediatric cardiac ICU nurse practitioners, and 1 pediatrics resident. The highest level of agreement was moderate (κ = 0.68, P < 0.001), with the majority poor to moderate. Performance varied by rhythm type (median κ): sinus (0.61), atrioventricular block (0.68), junctional (0.47), ectopic atrial tachycardia (0.25), and supraventricular tachycardia (0.49). There was good intrarater reliability (κ = 0.71 [median], P < 0.001).

CONCLUSIONS: Overall, frontline provider performance was suboptimal, especially for complex arrhythmia classes. These findings highlight the need for thoughtful consideration in labeler training and validate the need for a clinical decision support tool in arrhythmia detection.