Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness.

OBJECTIVES

Reliable measurement of advance care planning (ACP) is essential for evaluating both communication practices and quality of care in pediatric critical illness. However, most ACP documentation is embedded in free-text clinical notes and cannot be captured using administrative data, while manual chart review is time-intensive and difficult to scale. We sought to validate a rule-based natural language processing (NLP) approach to identify ACP documentation in the electronic health record (EHR) compared with manual chart review.

DESIGN

Retrospective cohort study.

SETTING

Single-center, quaternary PICU.

PATIENTS: Children younger than 21 years admitted to the PICU for greater than 24 hours following out-of-hospital cardiac arrest (OHCA) from 2012 to 2024.

INTERVENTIONS: None.

MEASUREMENTS AND MAIN

RESULTS

A rule-based NLP approach incorporating semi-automated chart review was developed through iterative refinement of previously validated keyword libraries to identify ACP documentation across prespecified domains in the EHR. In the validation cohort (n = 95), NLP-assisted chart review showed complete agreement with manual chart review (F1 = 1.0) while substantially reducing abstraction time, enabling analysis of over 31,479 clinical notes in 3 days compared with 6 months for manual review of 43,179 notes. Among 125 children with OHCA, 65% had documented ACP discussions (n = 81). Goals-of-care conversations were most common (99%), followed by limitations of life-sustaining treatment (78%), subspecialty palliative care involvement (28%), preferred location of death (9%), and hospice discussions (7%).

CONCLUSIONS

NLP-assisted chart review can efficiently and accurately identify ACP documentation in pediatric critical illness for retrospective research and quality improvement efforts. This semi-automated workflow substantially reduced manual abstraction burden while maintaining excellent agreement with manual chart review, enabling effective abstraction of communication processes not captured in structured data.