Extracting adverse event nature, severity, timelines and resulting interventions from clinical notes of patients receiving CAR-T cell therapy using large language models.

Introduction

Chimeric Antigen Receptor T-cell (CAR-T) therapy represents a significant advancement in the field of oncology, offering a treatment modality that can induce high response rates and durable remissions in refractory hematologic cancers, with active innovation aimed at expanding targets, improving safety, and overcoming barriers in solid tumors [ 1 – 4 ]. This therapeutic approach involves the genetic modification of a patient’s T-cells to express a CAR that targets specific antigens on tumor cells, thereby enhancing the immune system’s ability to recognize and destroy cancer cells [ 5 ]. In 2023, six CAR T-cell therapies were available on the US market for the treatment of hematologic malignancies, including lymphomas, some forms of leukemia, and multiple myeloma [ 6 ]. These Food and Drug Administration (FDA)-approved CAR T-cell therapies target one of two antigens on B cells, CD19 or BCMA. Despite their considerable therapeutic potential, CAR-T cell therapies are associated with frequent and clinically significant adverse events (AEs) [ 1 – 4 ], including Cytokine Release Syndrome (CRS) [ 7 – 9 ] and Immune Effector Cell-Associated Neurotoxicity Syndrome (ICANS) [ 10 ] which are among the most important toxicities observed after CAR-T therapy.

CRS is a potentially severe and potentially life-threatening AE characterized by an excessive immune response triggered by the activated CAR-T cells releasing large amounts of cytokines into the bloodstream [ 11 ]. Symptoms of CRS can range from mild constitutional symptoms to severe inflammatory responses, which can lead to multi-organ dysfunction or even death if not managed promptly and effectively [ 12 ]. The severity of CRS is often correlated with the disease burden and treatment response, which complicates the clinical management of affected patients [ 13 ]. In addition to CRS, other AEs can be detrimental to patients, including ICANS - a neurologic toxicity of CAR‑T and other immune effector therapies characterized by symptoms such as encephalopathy, confusion, aphasia, tremor, seizures, and, rarely, cerebral edema - that is thought to arise from cytokine‑driven endothelial activation and blood–brain barrier disruption [ 8 , 10 , 14 ]. Given the serious nature of CRS, ICANS, and other associated AEs, continuous and meticulous monitoring is imperative for the successful implementation of CAR-T therapy, the early detection of AEs, and the optimization of treatment protocols to reduce patient risks and ensure the benefits of CAR-T therapy [ 15 ].

Electronic health records (EHRs) provide a rich data source for monitoring outcomes in patients receiving CAR-T therapy; however, much of the clinically relevant information required is difficult to capture from structured data alone. Structured EHR data (diagnosis codes, medication administrations, lab values, procedure codes) are often not sufficiently accurate or detailed. As a result, they rarely capture adverse events with sufficient precision: the timelines, how symptoms change, what interventions occur, and the differences among AE grades. Prior studies have shown that relying exclusively on structured data can lead to underreporting or misclassification of adverse events in oncology cohorts, as coding practices vary across institutions and key clinical details are frequently documented only in clinical notes [ 16 , 17 ]. In contrast, unstructured data—including provider progress notes, nursing documentation, pathology reports, and consult notes—contain the granular qualitative information necessary to identify event onset, progression, and severity. Extracting such information at scale has historically required intensive manual chart review, which is labor-intensive and not feasible for large real-world datasets [ 18 ].

The advent of large language models (LLMs), such as the GPT (Generative Pre-trained Transformer) series developed by OpenAI, allows for innovative application of artificial intelligence in various fields, including healthcare [ 19 ]. LLMs are a type of artificial intelligence that utilizes deep learning techniques to understand and generate human-like text based on the training received from a vast corpus of text data [ 20 ]. These models are not only capable of analyzing context and generating text but also of performing tasks like translation, summarization, and, importantly for healthcare science, information extraction [ 21 ]. LLMs can be used to automate and enhance various data-intensive tasks that traditionally require extensive human labor, such as extracting patient information from unstructured clinical notes or assisting in real-time patient monitoring [ 22 – 24 ]. Leveraging natural language processing and large language models enables automated extraction of clinically relevant features from unstructured text, bridging the gap between codified data and nuanced clinical phenomena, and ultimately supporting more accurate AE monitoring in CAR-T populations [ 25 – 27 ].

The aim of this study was to evaluate the capability of a large language model to extract information about AEs, including their dates and associated clinical interventions, and to identify CRS and ICANS grades from the unstructured text of clinical progress notes following CAR-T therapy.

Methods

In this cross-sectional study, we identified all patients visiting the University of California, San Francisco (UCSF) hospital between January 1, 2012 and October 21, 2023 and receiving any of the six following CAR-T cell therapies: axicabtagene ciloleucel, idecabtagene vicleucel, ciltacabtagene autoleucel, lisocabtagene maraleucel, tisagenlecleucel, and brexucabtagene autoleucel ( Fig 1 ). From reference papers and expert opinions we identified a list of 25 adverse events that are recognized as CAR-T adverse events [ 6 – 8 , 14 ]. We defined these as the prespecified adverse events, which included: “Cytokine Release Syndrome (CRS)”, “Immune Effector Cell-Associated Neurotoxicity Syndrome (ICANS)”, “Hypogammaglobulinemia”, “Neutropenic Fever”, “Macrophage Activation Syndrome (MAS)”, “Hemophagocytic Lymphohistiocytosis (HLH)”, “Tumor Lysis Syndrome”, “Movement disorder”, “Cognitive impairment”, “Personality changes”, “B-cell aplasia”, “thrombocytopenia”, “anemia”, “neutropenia”, “leukopenia”, “CD4 lymphopenia”, “Cranial Nerve Palsy”, “Bacterial infection”, “Viral Infection”, “Zoster Infection”, “Fungal infection”, “Aspergillus infection”, “Pneumocystis Jirovecii Pneumonia infection”, “Secondary myeloid malignancies”, “Graft-versus-host disease (GVHD)”.

We retrieved all clinical notes from this cohort. We extracted all progress notes that occurred in the 30 days following CAR-T therapy infusion. If a patient received multiple CAR-T infusions at UCSF, each infusion was counted as a separate, independent episode. However, the primary description of the cohort and AE frequencies were aggregated at the patient level. These clinical notes have been previously de-identified, externally certified, and made available for research as previously described [ 28 ]. We tokenized all candidate notes using tiktoken and compared token counts to the GPT 4 0314 context window (8k tokens). To ensure the full note plus prompt (described below) fit and allow headroom for model output, we set an operational cap of 6,000 input tokens per note. Notes with fewer than 71 tokens (bottom quartile) were excluded due to limited clinical content, and notes with more than 6,000 tokens were excluded to avoid exceeding model context. No truncation or chunking was performed in the primary analysis; each retained note was submitted in full.

We restricted the analysis to progress notes because these documents consistently synthesize interval symptoms, assessment/plans, and bedside interventions relevant to AE onset, grade rationales, and management. To reduce heterogeneity in narrative structure and to ensure cost-effective analysis and evaluation, we excluded other types of notes (e.g., nursing documentation, consult notes, procedure notes, imaging reports, discharge summaries). This design choice prioritized meaningful clinical extraction for zero-shot prompting.

Accessing the application programming interface (API) via the Health Insurance Portability and Accountability Act (HIPAA)–compliant UCSF Secure Azure OpenAI environment, we utilized the GPT-4 model (OpenAI model = “gpt-4-0314”; role = “user”; temperature = 0; all other settings at default values). For each retained note, we constructed the user message as ‘Clinical note: “”“”“”’ + task and computed a conservative output-token budget to keep the combined input+output within the 8,192-token context window. Because the 6,000-token input cap was enforced upstream with tiktoken, input side overflows were not expected. If the code returned a token-budget error, we reduced the max tokens parameter and retried. We used four distinct prompts in a zero-shot manner ( S1 Table ), i.e., providing no additional training context, examples, or guidelines to the model. The first prompt asked the model to return prespecified AEs expected after any CAR-T infusion along with their dates and any associated clinical interventions documented within the clinical notes. The second prompt asked the model to detect the presence of any AEs related to CAR-T that are not among the prespecified AEs along with their dates and any clinical interventions. Third, we prompted the LLM to identify the presence, date, and grade of CRS along with any associated clinical interventions. Lastly, we prompted the LLM to identify the presence, date, and grade of ICANS, and any associated clinical interventions. Clinicians frequently document ‘CRS grade 0’ and ‘ICANS grade 0’ in the clinical notes during daily monitoring to indicate no active toxicity. We therefore instructed the model to extract ‘grade 0’ when explicitly documented, even though it reflects absence of the event. In all analyses, grade 0 was treated as an explicitly documented grade category and interpreted as absence of active CRS/ICANS, rather than as an adverse event. Each of the four prompts was executed independently on the same set of progress notes. AE and Clinical interventions counts were derived separately per prompt without cross‑prompt reconciliation. Therefore, AE and clinical intervention counts are prompt‑specific and may differ across prompts.

A sample of notes was randomly selected for manual evaluation of the LLM performance. It was designed as a feasibility evaluation to assess whether a zero‑shot LLM could recover key adverse event elements from routine notes. We selected 50 progress notes at random from the eligible set. These sampled notes represented 47 different patients and enabled independent double evaluation by two clinical evaluators within practical time constraints. While each note generated multiple element-level assessments across the four prompts (AE presence, date, grade for CRS/ICANS, and interventions), these observations were not fully independent, and therefore do not eliminate the uncertainty associated with the modest note-level sample size. Our intent was not to produce narrow confidence bounds but to obtain point estimates and to identify tasks where model behavior appeared reliable enough at the zero-shot step. No prompt engineering was performed on the validation notes. LLM outputs were manually reviewed and evaluated as True or False for AEs, grades (for the 3rd and 4th prompts), dates of occurrence, and associated clinical interventions by Jordan Guillot (JG): a PharmD, PhD, with 5 years of postgraduate training in pharmacoepidemiology research and Arvind Suresh (AS): an MD, an oncologist with 3 years of postgraduate general medical training, to compare the performance between the LLM models and human annotations. We evaluated performance separately for each element: AE presence, grade, AE date, and clinical intervention. For each element, a true positive was counted when the system correctly extracted the element as documented in the note. A false positive was counted when the system extracted an element that was not documented or extracted an incorrect element. A true negative was counted when the element was not documented in the note and the LLM made no extraction. A false negative was counted when the element was documented but the LLM failed to extract it. An annotation guide was written by JG and provided to AS to ensure that both evaluators followed a similar evaluation process ( S1 Annotation Guide ). After reviewing the guide, the reviewers independently assessed the veracity of the model outputs. The reviewers remained independent and did not share information or formally resolve disagreements for the IRR analysis, because the goal was to quantify independent agreement rather than generate consensus labels. Performance was evaluated using accuracy, precision, recall, and F1 score, computed using their standard definitions [ 29 ]. The average performance was calculated by macro-averaging the results from the two evaluators. Inter-rater reliability (IRR) between the two reviewers was assessed by Cohen’s kappa [ 30 ]. Among the prespecified adverse events, four could be identified from either or both free text or blood count laboratory values: anemia, thrombocytopenia, neutropenia, and leukopenia. Given the complexity and variability in how these are reported in clinical notes, we conducted a sensitivity analysis and calculated IRR for the prespecified adverse events after excluding these four.

This study utilized data drawn from the UCSF Health system, where deidentification of the data has been completed to enable clinical research projects in accordance with guidance from UCSF institutional review boards (IRBs), privacy, and compliance standards. The UCSF IRBs determined that this use of de-identified structured and clinical text data in the UCSF Information Commons is not considered human participant research and that the study was therefore exempt from further approval and the need for informed consent ( Fig 1 ).

Results

A total of 293 patients receiving CAR-T cell therapy at UCSF were included in the study. Among them, 97 (33.1%) patients received Axicabtagene ciloleucel, 64 (21.8%) received Ciltacabtagene autoleucel, 43 (14.7%) received Tisagenlecleucel, 37 (12.6%) received Lisocabtagene maraleucel, 36 (12.3%) received Idecabtagene vicleucel and 16 (5.4%) received Brexucabtagene autoleucel. Among these patients, only one received CAR-T therapy twice, in two separate instances approximately 10 months apart. Due to the rarity of this situation and the time difference between instances, we expect any impact on the analysis descriptive findings to be minimal. Median age was 65.6 years [IQR: 54.8 – 71.7]. There were 176 (60.0%) male and 117 (40.0%) female. Among all patients 170 (58.0%) were White, 69 (23.5%) Hispanic, 34 (11.6%) Asian, 11 (3.8%) Black and ≤ 10 (≤3.4%) other than these race/ethnicity categories. Among these patients, the most common disease codes in UCSF EHR that matched the approved indications for CAR-T therapy were diffuse large B-cell lymphoma (DLBCL), coded for 150 patients (51.2%); multiple myeloma, for 131 patients (44.7%); and acute lymphoblastic leukemia (ALL), for 36 patients (12.3%). Among included patients, 25,974 clinical notes documented within 30 days after CAR‑T administration were identified. Specifically, after filtering by note length (i.e., retaining only notes with more than 71 tokens and fewer than 6000 tokens), 18,808 notes remained, of which 4,762 (25.3%) were progress notes retained for analysis ( Table 1 ). The other types of notes not retained for the analysis were 2503 (13.3%) imaging notes, 2212 (11.8%) consults, 1986 (10.6%) plans of care, 1475 (7.8%) registered nurse notes, 1359 (7.2%) interdisciplinary notes, 588 (3.1%) procedure notes and 3923 (20.9%) other type of notes ( S2 Table ).

Point estimates for performance in our evaluation sample varied across the four prompts ( Fig 2 , S3 Table ). Averaging across the two evaluators, prespecified AEs showed mean accuracies of 0.62 (nature), 0.76 (date), and 0.69 (clinical intervention), with average F1 scores ranging from 0.74 (nature) to 0.78 (date). For non‑prespecified AEs, average accuracies were 0.76 (nature), 0.78 (date), and 0.84 (clinical intervention), with average F1 scores from 0.62 (date) to 0.75 (clinical intervention). For CRS-specific prompts, point estimates were consistently high: average accuracies ranged from 0.86 (date) to 0.97 (grade), with average F1 scores from 0.89 (date) to 0.97 (grade). For ICANS-specific prompts, average accuracies ranged from 0.69 (date) to 0.91 (grade), with average F1 scores from 0.56 (date) to 0.80 (grade).

Inter-rater reliability (IRR), assessed using Cohen’s kappa, between evaluators varied by output category, using the bands weak (0.40–0.60), moderate (0.60–0.80), strong (0.80–0.90), and near-perfect (>0.90). For prespecified AEs, IRR was weak (nature 0.40; date 0.55; clinical intervention 0.45). For non‑prespecified AEs, IRR was moderate (nature 0.61; date 0.63; clinical intervention 0.68). For CRS prompts, IRR was strong for presence (0.86), near-perfect for grade (0.96), and moderate for date (0.78) and clinical intervention (0.77). For ICANS prompts, IRR was moderate for presence (0.79) and clinical intervention (0.78), strong for grade (0.86), and near-perfect for date (0.92) ( Fig 2 ). Of the 264 inter-reviewer discrepancies, 16 (6.1%) involved thrombocytopenia, 14 (5.3%) involved anemia within the prespecified AE prompt. False negative model outputs — defined as instances in which the presence of an AE, date, or clinical intervention was identified by an evaluator but not the LLM — were also common discrepancies (3.8% each for AE presence and date in the non-prespecified event prompt; 2.3% each for AE presence and date in the prespecified event prompt). Disagreements on CRS/ICANS presence were infrequent (CRS yes: 1.5%; ICANS yes: 1.1%), consistent with the stronger kappas and higher point estimates we observed for these tasks. Date-related disagreements included many single-date mismatches at low frequency. The sensitivity analysis excluding lab-based AEs (anemia, thrombocytopenia, neutropenia, leukopenia) in the prespecified event prompt showed a moderate IRR for nature (0.63) and lower agreement for date (0.56) and clinical intervention (0.51).

The most common prespecified AEs among all patients included CRS, observed in 235 patients (80.2%), with a median onset of 4 days (IQR: 2–7); neutropenia, affecting 205 patients (70.0%), with a median onset at 16 days (IQR: 9–22); and neutropenic fever, reported in 190 patients (64.8%), with a median onset of 4 days (IQR: 2–7). Other hematologic toxicities, such as thrombocytopenia (155, 52.9%) and anemia (148, 50.5%) were also common and had median onset timelines of 19 days (IQR: 8–24) and 15 days (IQR: 6–22), respectively. Infectious complications were also prevalent, with bacterial infections documented in 84 patients (28.7%) and viral infections in 29 patients (9.9%), both occurring at a median timeline of 7–9 days post-treatment. ICANS was reported in 102 (34.8%) patients ( Fig 3 , Table 2 ). The most common information on clinical interventions for prespecified AEs extracted by the LLM included “no transfusions needed today” (reported for 48 patients, 16.4%), tocilizumab administration (46, 15.7%), and “Transfuse for Hb<7,plt<10k” (25, 8.5%) ( Fig 3 , S4 Table ).

Non-prespecified AEs were less frequent than prespecified AEs and more heterogeneous in type because not classified as categories. The most prevalent included fever (4.1%), nausea (3.8%), hypotension (≤3.4%), and constipation (≤3.4%), with median onset timelines ranging from 4 to 7.5 days post-CAR-T therapy. Due to the heterogeneity of non-prespecified AEs extracted, we grouped them for descriptive purposes ( S5 Table ). When grouped by medical domains, The most prevalent included neurological issues and dizziness (20.1%), gastrointestinal Issues (18.8%), and cardiovascular issues (16.7%), although they might be symptoms of prespecified AEs that would have been wrongly extracted in the prompt for non-prespecified AEs ( Fig 4 , Table 3 ). Clinical interventions for non-prespecified AEs were also heterogeneous and included continuous telemetry monitoring (CTM) (≤3.4%) and bicarbonate supplementation (≤3.4%) ( Fig 4 , S6 Table ).

The prompt aimed at extracting the presence of CRS and its grade identified that CRS impacted 239 (81.6%) of patients. Because each prompt was run independently and outputs were not reconciled across prompts, the patient-level CRS count from the CRS-specific prompt differed from the CRS count extracted by the prespecified AE prompt. Among those, Grade 1 CRS was observed in 142 patients (59.4%), making it the most common grade reported. Grade 0 CRS occurred in 71 patients (29.7%), Grade 1–2 CRS in ≤10 patients (≤4.2%), Grade 2 CRS in 76 patients (31.8%), and Grade 2–3 CRS in ≤10 patients (≤4.2%). Grade 3 CRS in ≤10 patients (≤4.2%), and Grade 4 CRS in ≤10 patients (≤4.2%). CRS grade was identified as unknown for 148 patients (61.9%), highlighting limitations in information available in clinical notes, e.g., physician mentioning CRS without stating the grades even though they might give characteristics of the CRS. Because multiple CRS grades could be documented for the same patient across notes or time points, the grade-specific percentages are not mutually exclusive and should not be summed to 100%; rather, they reflect how frequently each grade was identified overall. ( Fig 5 , Table 3 ). The timeline for CRS onset varied across grades, with median onset days ranging from 2 to 7 days post-CAR-T therapy. Patients with Grade 1 CRS had a median onset of 4 days, whereas Grade 2 CRS had a median onset of 3 days. For Grade 3 CRS, the median timeline was 6 days, and Grade 4 CRS had a median onset of 5 days ( Fig 5 , Table 4 ). The most frequent clinical interventions were, for Grade 0 CRS, daily assessment of CRS. For Grade 1 CRS, administration of tocilizumab, for Grade 2 CRS, dexamethasone 10 mg IV Q6H, for Grade 3 CRS, administration of dexamethasone 10 mg IV Q6H, for Grade 4 CRS, the use of methylprednisone pulse therapy ( S1 Fig , S7 – S13 Tables ).

The prompt designed to identify ICANS grades reported that ICANS occurred in 132 (45.1%) patients. Because each prompt was run independently and outputs were not reconciled across prompts, the patient-level ICANS count from the ICANS-specific prompt differed from the ICANS count extracted by the prespecified AE prompt. Among those, Grade 0 which correspond to absence of ICANS was the most commonly extracted information, observed in 41.7% of the 132 cases, followed by Grade 2 (28.0%) and Grade 1 (25.0%). Higher grades of ICANS, including Grade 3 (9.1%), Grade 3–4 (≤7.6%), and Grade 4 (≤7.6%), were less frequent. Grade Unknown was reported in 56.1% of cases, potentially reflecting again incomplete documentation. Because more than one ICANS grade could be recorded for an individual patient across different notes or time points, the grade-specific percentages are not mutually exclusive and should not be added to total 100%; instead, they indicate the overall frequency with which each grade was identified. ( Fig 6 , Table 4 ). ICANS median timeline was 5–7 days post-CAR-T therapy. ( Fig 6 , Table 4 ). The most frequent clinical intervention was seizure prophylaxis with Keppra for Grade 0 ICANS, no intervention for Grade 1, dexamethasone 10mg IV q6h for Grades 2 and 3, and Solumedrol 1 gram IV daily for 3 days for Grade 4 ( S2 Fig , S14 – S21 Tables ).

Discussion

Adverse events (AEs) after Chimeric Antigen Receptor T-cell (CAR-T) therapy are common and often severe, making timely detection and intervention essential for patient safety. However, much of the clinical detail needed to understand AE onset, severity, and management is buried in unstructured notes that are difficult to analyze at scale. Leveraging large language models (LLMs) to extract these patterns offers a promising way to better characterize AE trajectories and ultimately improve monitoring and outcomes for CAR-T recipients.

In our study, the LLM demonstrated high point estimate performance for CRS- and ICANS-specific prompts, particularly for presence and grade. For broader prespecified AE prompts, point estimates were moderate and showed evaluator-dependent variability. Although the validation sample was not powered for an accurate estimate of LLM performance, this likely reflects the ambiguity and variability of AE documentation in unstructured clinical notes and the inherent challenges of using an LLM to identify multiple categories of adverse events. In contrast, agreement was higher for CRS and ICANS presence and grading, likely due to their more standardized clinical definitions and documentation. When using the LLM in our data, trends in AE occurrence highlighted CRS as the most frequently reported event, followed by neutropenia in prevalence, with each displaying characteristic timing patterns post-treatment. Prompts designed to extract grades provided insight into the distribution of severity, with lower grades predominating for CRS and ICANS. These results are supported by the strong accuracy and IRR; however, limitations in data completeness occasionally hindered precise grade assignment. These findings underscore the potential of LLMs to reliably capture key information on CRS and ICANS following CAR-T, while also generating hypotheses for areas where heterogeneity across other AEs and incomplete documentation may affect performance, warranting cautious interpretation of the results for this task and calling for further research.

Recent work has demonstrated the effective application of LLMs to extract clinical information from unstructured data [ 31 – 35 ]. Moreover, emerging studies have shown good performance in the application of these techniques for adverse event detection. While emphasizing challenges such as domain-specific variability in model performance, large language models and LLM-based pipelines have shown strong performance for adverse drug event extraction and structured AE grades, with F1-scores between 0.62 and 0.99, outperforming traditional machine learning models and ICD-based systems [ 36 – 41 ]. Point estimates for performance derived from our study validation sample aligned with these previous findings, highlighting that a commercial LLM could demonstrate moderate to high performance metrics for extraction of CAR-T related AEs, similar to those reported in these prior studies that include oncology and also other domains [ 36 – 41 ]. These results encourage physicians and researchers to conduct further research using these tools in CAR-T monitoring, clinical trials, and real-world evidence research, while emphasizing the need for careful supervision and improving and standardizing performance.

Across pivotal CAR-T cell therapy trials and large real-world cohorts, CRS and ICANS were frequently reported which aligned with the extraction we performed on UCSF data using LLMs. Moreover, in our study, the model captured low-frequency, non-prespecified events such as rash and constipation with good accuracy and IRR, which may serve as early warning signs of broader toxicity syndromes or signal unexpected AEs. In the pivotal ZUMA-1 trial of axicabtagene ciloleucel, the most common AEs of any grade were fever/pyrexia (85% of the patients), neutropenia (84%), and anemia (66%). CRS occurred in 93% of patients, with grade ≥ 3 events in about 13%, while neurologic AEs in 64% with 28% grade ≥ 3 [ 1 ]. The JULIET trial of tisagenlecleucel reported rates of CRS (58%), anemia (48%), neutropenia (34%) and any neurologic events (12%) [ 3 ], and the TRANSCEND study of lisocabtagene maraleucel demonstrated lower frequencies of neutropenia (60%), anemia (37%), CRS (42%) and ICANS (30%) [ 42 ]. Our study included several CAR-T therapies and was conducted in a real‑world setting, which might explain differences to clinical trials of single drugs, although our results preserved relative proportions of the common AEs and fit what is clinically expected. Meta-analytic evidence also supports the prevalence of these AEs, indicating pooled CRS and ICANS incidences of 70–90% and 25–30%, respectively, with pooled high-grade CRS under 15%, under 30% for all grades ICANS about 10% for high-grade ICANS [ 8 , 43 ]. Furthermore, large multicenter registries have reported similar rates of severe CRS and ICANS, with most events occurring within the first two weeks post-infusion and resolving with tocilizumab or corticosteroids, which validate the timeline and interventions that we were able to extract [ 44 – 46 ]. This data along with our performance analysis on our validation sample shows that LLMs exhibit high point estimates for performance in identifying CRS and ICANS occurrence and their respective grades. Our study also showed the high prevalence of CRS, ICANS, neutropenia, anemia, and infections in the real-world CAR-T setting, consistent with previous clinical trials and registry-based analyses. Furthermore, we showed that these AEs typically occur early after infusion, often within the first two weeks, highlighting the critical need for close monitoring and early intervention [ 13 , 46 ]. Severity, and current and previous care are essential considerations in the management of ongoing or evolving toxicities, and can significantly influence the trajectory and recovery from CRS, ICANS, or hematologic AE [ 9 , 10 , 13 , 44 , 46 ]. Our LLM based system demonstrated high accuracy in extracting the occurrence and grades of CRS and ICANS from clinical notes, offering substantial potential to enhance real-time patient monitoring, clinical decision support, and pharmacovigilance monitoring. These findings support further supervised evaluation and validation of LLM-driven extraction tools in clinical and research settings to complement existing manual or structured data sources within oncology practice and research.

This work has several important strengths, while also having limitations that merit consideration. A key limitation is the small manual review sample relative to the number of extraction tasks assessed. The sample is not powered to provide narrow estimates of the model performances and estimates could vary with small changes in counts. In addition, multiple elements within one note were not independent and one note could include content copied forward from previous records or notes without explicit formatting, which can further complicate interpretation and manual assessment of the extracted adverse events. For these reasons, we interpreted performance estimates as preliminary and placed greater emphasis on the qualitative pattern of results (notably, higher point estimates for CRS/ICANS presence and grade). Therefore, all subgroup-level findings, particularly for prespecified and non-prespecified AEs, should be regarded as hypothesis generating. Another limitation is the variability in inter‑rater reliability (IRR), and the discrepancy patterns help localize where performance and agreement were most fragile. First, temporality was difficult to adjudicate when notes contained multiple absolute dates, relative day counts, or carry‑forward text. Evaluators assessed plain‑text exports without the usual EHR sectioning or time stamps, which further impeded determining AE occurrence relative to infusion. Second, prespecified AEs that can be inferred from both laboratory values and narrative (anemia, thrombocytopenia, neutropenia, leukopenia) could have been documented inconsistently across text and labs and were sometimes historical or resolving; clinicians and evaluators may also have applied different thresholds. In our sensitivity analysis excluding these lab‑based cytopenias, IRR improved for AE nature, supporting this interpretation, though agreement for dates and interventions remained lower. Third, AE and intervention mentions were highly heterogeneous; although causality and imputation were not requested, implicit clinical reasoning about whether an event was CAR‑T–related may have influenced extraction and evaluators’ review which also may reflect differences in the evaluators’ training backgrounds. Fourth, de‑identification placeholders (*****), abbreviations, and copy‑forward practices sometimes obscured extraction, evaluation, or whether text reflected a plan, prophylaxis, or an administered therapy, further increasing variability in IRR. Taken together, these factors suggest that weaker kappas for prespecified and non-prespecified AEs reflect documentation ambiguity and annotation-framework complexity at least as much as model limitations. Event extraction relied heavily on documentation quality, and the high proportion of “grade unknown” for CRS and ICANS highlights incomplete or inconsistent charting. This highlights variability in the way physicians document their notes and suggests that grades are not always explicitly stated but are often expected to be inferred by clinicians reading one another’s notes. This calls for further research into LLM interpretation of clinical information to determine occurrence of adverse AEs. Class imbalance (e.g., fewer positive cases for some non-prespecified events) was another limitation, reflecting the lower likelihood of occurrence for certain AEs. Limiting the analysis to progress notes optimized our use of resources and likely reduced heterogeneity, but it may have excluded clinically meaningful AE signals captured elsewhere (e.g., nursing notes for vitals and supportive care, consults for subspecialty assessments, ED notes for early symptoms, and discharge summaries for consolidated course/grade rationales). As a result, some AE dates or interventions may have been missed when they were documented exclusively in non-progress note types. An important next step would be to extend extraction to targeted additional note types. Furthermore, because only length-eligible progress notes were analyzed, the retained note set represented a selected subset of all available post-infusion documentation, which may have introduced selection bias if adverse events, interventions, or severity details were preferentially documented in excluded note types or in notes outside the token-length range. Additionally, the model’s context-window limitations required exclusion of longer notes, limiting our ability to assess performance on lengthy, longitudinal charts spanning multiple encounters and potentially obscuring cross-document dependencies. This limitation may be mitigated by newer models with substantially larger context windows than the gpt-4–0314 model used in this study, enabling more complete evaluation of longer clinical narratives and cross-note relationships. Non-prespecified AEs and clinical interventions were not embedded by any methods to reduce computational steps and minimize the time required for reviewers. Clustering these would enhance interpretation and highlight the need for further research on how to integrate this step. The study employed zero-shot prompting without additional training, context, or guidelines for the model, which may have impacted overall performance but provides a foundation for future methodological refinements, performance improvements and represents an important step in oncology toward establishing the potential of LLMs to identify and extract CAR-T-related AEs from clinical notes. Moreover, we envision evaluating this method in other healthcare systems and with additional model versions to better assess generalizability, robustness, and potential future applications.

Conclusion

We applied and evaluated an LLM to analyze clinical notes and extract AEs, including CRS and ICANS presence, grades, dates, and related interventions, from unstructured progress notes. Point estimates for performance were high for CRS and ICANS presence and grade, with strong to near-perfect inter-rater reliability, moderate-to-high point estimates for CRS and ICANS dates and interventions, and more variable performance for broader AE nature, date, and intervention categories.

These results support further prospective evaluation of supervised LLM-based extraction of CRS and ICANS grading in EHR-based research or pilot settings, accompanied by prospective monitoring and expanded validation. Broader AE extraction should remain exploratory until supported by larger, more diverse manual evaluations.

Documentation variability and class imbalance for broader AEs remain challenges, highlighting the need for further research into LLM interpretation of clinical results. Additionally, cautious use of LLMs in a zero-prompt manner for broad AE extraction is warranted due to the variable performance observed in this study. Nevertheless, this study provides a foundation for further research, including piloting the introduction of LLMs into real-world evidence studies and safety monitoring in clinical trials and oncology practice, while ensuring careful supervision and ensuring that performance metrics are carefully monitored and interpreted.

Supporting information

Annotation Guide for annotators.

(PDF)

Clinical interventions by CRS grade.

(TIFF)

Clinical interventions by ICANS grade.

(TIFF)

Prompt texts for AE (prespecified), non-prespecified AE, CRS, and ICANS extraction.

(DOCX)

Note types and patient counts (cohort overview).

(DOCX)

LLM performance by prompt and reviewer (accuracy, precision, recall, F1, kappa).

(DOCX)

Top 50 interventions for prespecified AEs.

(DOCX)

Categories and examples of non-prespecified AEs.

(DOCX)

Top 50 interventions for non-prespecified AEs.

(DOCX)

Top 30 interventions for CRS Grade 0.

(DOCX)

Top 30 interventions for CRS Grade 1.

(DOCX)

Top interventions for CRS Grades 1–2.

(DOCX)

Top 30 interventions for CRS Grade 2.

(DOCX)

All interventions for CRS Grades 2–3.

(DOCX)

All interventions for CRS Grade 3.

(DOCX)

All interventions for CRS Grade 4.

(DOCX)

Top 30 interventions for ICANS Grade 0.

(DOCX)

Top 30 interventions for ICANS Grade 1.

(DOCX)

All interventions for ICANS Grades 1–2.

(DOCX)

Top 30 interventions for ICANS Grade 2.

(DOCX)

All interventions for ICANS Grades 2–3.

(DOCX)

Top 30 interventions for ICANS Grade 3.

(DOCX)

All interventions for ICANS Grades 3–4.

(DOCX)

All interventions for ICANS Grade 4.

(DOCX)

Acknowledgments

The authors thank the Butte Lab, including Ana Maria Deluca, Boris Oskotsky, Dana Ludwig, Harry Sun, Jaysón Davidson, Michelle Wang, Noah Baker, Sanchita Bhattacharya, Sirui Ding, Travis Zack, and Reuben Sarwal. We also extend our gratitude to the Sirota Lab led by Marina Sirota, including Laura Almonte, Umair Khan, and Brenda Ametepe. Additionally, we thank the Research Data Assets & Information Commons Team, the Research Infrastructure Team, Gerald (Gerry) Fontenay, and the staff of the UCSF Bakar Computational Health Sciences Institute. We are grateful for the support provided by UCSF-Stanford CERSI, including Kathy Giacomini and Lawrence Lin, as well as Nathan Dang, Holly Ly, and Jaime Kenyon. Finally, we acknowledge Gilead Sciences, including Shahed Iqbal, as well as the EpiRWE team led by Frangiscos Sifakis, for their support and collaboration during JG’s postdoctoral fellowship. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Availability

The annotation guidelines and prompts are provided in the Supplementary Materials. Code to reproduce the analyses is available at GitHub [ https://github.com/jordan-guillot/Adverse-Event-Extraction-from-CAR-T-Clinical-Notes-Using-Large-Language-Models ] and has been reviewed to ensure it contains no sensitive information from clinical notes or patient-level data. Due to the sensitivity of clinical text data and institutional/IRB restrictions, the underlying de-identified patient-level data, including clinical notes and note-level annotations, are not publicly available and cannot be shared outside our institution, even upon request. Aggregate results and summary statistics are included in the manuscript and Supplementary Materials, and sanitized analytic code is available where feasible and consistent with institutional policies.

Funding Statement

JG was supported by a University of California, San Francisco (UCSF)-Stanford Center of Excellence in Regulatory Science and Innovation (CERSI)/Gilead Sciences Fellowship in Regulatory Science. MSu was partially supported by the National Cancer Institute of the National Institutes of Health award number P30CA082103. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1. Neelapu SS, Locke FL, Bartlett NL, Lekakis LJ, Miklos DB, Jacobson CA, et al. Axicabtagene ciloleucel CAR T-cell therapy in refractory large B-cell lymphoma. N Engl J Med. 2017;377(26):2531–44. doi: 10.1056/NEJMoa1707447

2. Maude SL, Laetsch TW, Buechner J, Rives S, Boyer M, Bittencourt H, et al. Tisagenlecleucel in Children and Young Adults with B-Cell Lymphoblastic Leukemia. N Engl J Med. 2018;378(5):439–48. doi: 10.1056/NEJMoa1709866

3. Schuster SJ, Bishop MR, Tam CS, Waller EK, Borchmann P, McGuirk JP, et al. Tisagenlecleucel in Adult Relapsed or Refractory Diffuse Large B-Cell Lymphoma. N Engl J Med. 2019;380(1):45–56. doi: 10.1056/NEJMoa1804980

4. Berdeja JG, Madduri D, Usmani SZ, Jakubowiak A, Agha M, Cohen AD, et al. Ciltacabtagene autoleucel, a B-cell maturation antigen-directed chimeric antigen receptor T-cell therapy in patients with relapsed or refractory multiple myeloma (CARTITUDE-1): a phase 1b/2 open-label study. Lancet. 2021;398(10297):314–24. doi: 10.1016/s0140-6736(21)00933-8

5. Sadelain M, Rivière I, Riddell S. Therapeutic T cell engineering. Nature. 2017;545(7655):423–31. doi: 10.1038/nature22395

6. Brudno JN, Maus MV, Hinrichs CS. CAR T Cells and T-Cell Therapies for Cancer: A Translational Science Review. JAMA. 2024;332(22):1924–35. doi: 10.1001/jama.2024.19462

7. Adkins S. CAR T-Cell Therapy: Adverse Events and Management. J Adv Pract Oncol. 2019;10(Suppl 3):21–8. doi: 10.6004/jadpro.2019.10.4.11

8. Yang C, Nguyen J, Yen Y. Complete spectrum of adverse events associated with chimeric antigen receptor (CAR)-T cell therapies. J Biomed Sci. 2023;30(1):89. doi: 10.1186/s12929-023-00982-8

9. Brudno JN, Kochenderfer JN. Toxicities of chimeric antigen receptor T cells: recognition and management. Blood. 2016;127(26):3321–30. doi: 10.1182/blood-2016-04-703751

10. Rejeski K, Jain MD, Shah NN, Perales M-A, Subklewe M. Immune effector cell-associated haematotoxicity after CAR T-cell therapy: from mechanism to management. Lancet Haematol. 2024;11(6):e459–70. doi: 10.1016/S2352-3026(24)00077-2

11. Shimabukuro-Vornhagen A, Gödel P, Subklewe M, Stemmler HJ, Schlößer HA, Schlaak M, et al. Cytokine release syndrome. J Immunother Cancer. 2018;6(1):56. doi: 10.1186/s40425-018-0343-9

12. Lee DW, Santomasso BD, Locke FL, Ghobadi A, Turtle CJ, Brudno JN, et al. ASTCT Consensus Grading for Cytokine Release Syndrome and Neurologic Toxicity Associated with Immune Effector Cells. Biol Blood Marrow Transplant. 2019;25(4):625–38. doi: 10.1016/j.bbmt.2018.12.758

13. Neelapu SS, Tummala S, Kebriaei P, Wierda W, Gutierrez C, Locke FL, et al. Chimeric antigen receptor T-cell therapy - assessment and management of toxicities. Nat Rev Clin Oncol. 2018;15(1):47–62. doi: 10.1038/nrclinonc.2017.148

14. Cohen AD, Parekh S, Santomasso BD, Gállego Pérez-Larraya J, van de Donk NWCJ, Arnulf B, et al. Incidence and management of CAR-T neurotoxicity in patients with multiple myeloma treated with ciltacabtagene autoleucel in CARTITUDE studies. Blood Cancer J. 2022;12(2):32. doi: 10.1038/s41408-022-00629-1

15. Teachey DT, Lacey SF, Shaw PA, Melenhorst JJ, Maude SL, Frey N, et al. Identification of Predictive Biomarkers for Cytokine Release Syndrome after Chimeric Antigen Receptor T-cell Therapy for Acute Lymphoblastic Leukemia. Cancer Discov. 2016;6(6):664–79. doi: 10.1158/2159-8290.CD-16-0040

16. Young IJB, Luz S, Lone N. A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis. Int J Med Inform. 2019;132:103971. doi: 10.1016/j.ijmedinf.2019.103971

17. Li F, Liu W, Yu H. Extraction of Information Related to Adverse Drug Events from Electronic Health Record Notes: Design of an End-to-End Model Based on Deep Learning. JMIR Med Inform. 2018;6(4):e12159. doi: 10.2196/12159

18. Hossain E, Rana R, Higgins N, Soar J, Barua PD, Pisani AR, et al. Natural Language Processing in Electronic Health Records in relation to healthcare decision-making: A systematic review. Comput Biol Med. 2023;155:106649. doi: 10.1016/j.compbiomed.2023.106649

19. OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al. GPT-4 Technical Report [Internet]. arXiv. 2024. doi: 10.48550/arXiv.2303.08774

20. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language Models are Few-Shot Learners. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2020. [cited 2024 Aug 5]. p. 1877–901. Available from: https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

21. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. 2019. doi: 10.48550/arXiv.1810.04805

22. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med. 2018;1:18. doi: 10.1038/s41746-018-0029-1

23. Beam AL, Kohane IS. Big Data and Machine Learning in Health Care. JAMA. 2018;319(13):1317–8. doi: 10.1001/jama.2017.18391

24. Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. arXiv. 2020. doi: 10.48550/arXiv.1904.05342

25. Modi S, Kasmiran KA, Mohd Sharef N, Sharum MY. Extracting adverse drug events from clinical Notes: A systematic review of approaches used. J Biomed Inform. 2024;151:104603. doi: 10.1016/j.jbi.2024.104603

26. Hu Y, Chen Q, Du J, Peng X, Keloth VK, Zuo X, et al. Improving large language models for clinical named entity recognition via prompt engineering. J Am Med Inform Assoc. 2024;31(9):1812–20. doi: 10.1093/jamia/ocad259

27. Guellil I, Wu J, Pradipta Gema A, Francis F, Berrachedi Y, Chenni N, et al. Natural language processing for detecting adverse drug events: A systematic review protocol. NIHR Open Res. 2024;3:67. doi: 10.3310/nihropenres.13504.3

28. Radhakrishnan L, Schenk G, Muenzen K, Oskotsky B, Ashouri Choshali H, Plunkett T, et al. A certified de-identification system for all clinical text documents for information extraction at scale. JAMIA Open. 2023;6(3):ooad045. doi: 10.1093/jamiaopen/ooad045

29. Rainio O, Teuho J, Klén R. Evaluation metrics and statistical tests for machine learning. Sci Rep. 2024;14:6086. doi: 10.1038/s41598-024-56706-x

30. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. doi: 10.11613/bm.2012.031

31. Ntinopoulos V, Rodriguez Cetina Biefer H, Tudorache I, Papadopoulos N, Odavic D, Risteski P, et al. Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation. BMJ Health Care Inform. 2025;32(1):e101139. doi: 10.1136/bmjhci-2024-101139

32. Danhauser K, Wang Y, Klein C, Tacke U, Mantoan L, Ritter LA, et al. Using large language models to extract information from pediatric clinical reports. PLOS Digit Health. 2025;4(7):e0000919. doi: 10.1371/journal.pdig.0000919

33. Cardamone NC, Olfson M, Schmutte T, Ungar L, Liu T, Cullen SW, et al. Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study. JMIR Med Inform. 2025;13:e65454. doi: 10.2196/65454

34. Far AT, Ayati A, Guillot J, Azzam S, Rudrapatna VA, Ge J. Large Language Models can Identify the Presence of MASH and Extract VCTE Measurements from Unstructured Documentation. Dig Dis Sci. 2025;71(4):1505–11. doi: 10.1007/s10620-025-09539-1

35. Silverman AL, Sushil M, Bhasuran B, Ludwig D, Buchanan J, Racz R, et al. Algorithmic identification of treatment-emergent adverse events from clinical notes using large language models: a pilot study in inflammatory bowel disease. Clin Pharmacol Ther. 2024;115(6):1391–9. doi: 10.1002/cpt.3226

36. Golder S, Xu D, O’Connor K, Wang Y, Batra M, Hernandez GG. Leveraging Natural Language Processing and Machine Learning Methods for Adverse Drug Event Detection in Electronic Health/Medical Records: A Scoping Review. Drug Saf. 2025;48(4):321–37. doi: 10.1007/s40264-024-01505-6

37. Kopacheva E, Henriksson A, Dalianis H, Hammar T, Lincke A. Identifying Adverse Drug Events in Clinical Text Using Fine-Tuned Clinical Language Models: Machine Learning Study. JMIR Form Res. 2025;9:e71949. doi: 10.2196/71949

38. Zhang K, Huang T, Malin BA, Osterman T, Long Q, Jiang X. Introducing mCODEGPT as a zero-shot information extraction from clinical free text data tool for cancer research. Commun Med (Lond). 2025;5(1):422. doi: 10.1038/s43856-025-01116-x

39. Bejan CA, Wang M, Venkateswaran S, Bergmann EA, Hiles L, Xu Y, et al. irAE-GPT: Leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets. medRxiv. 2025;2025.03.05.25323445. doi: 10.1101/2025.03.05.25323445

40. Zitu MM, Owen D, Manne A, Wei P, Li L. Large Language Models for Adverse Drug Events: A Clinical Perspective. J Clin Med. 2025;14(15):5490. doi: 10.3390/jcm14155490

41. Ludwig D, Wang M, Buchanan J, Trinh T. A Large Language Model for Extracting Post-marketing Adverse Drug Events from Clinical Notes in the Electronic Health Record. Drug Saf. 2026. doi: 10.1007/s40264-026-01682-6

42. Abramson JS, Palomba ML, Gordon LI, Lunning MA, Wang M, Arnason J, et al. Lisocabtagene maraleucel for patients with relapsed or refractory large B-cell lymphomas (TRANSCEND NHL 001): a multicentre seamless design study. Lancet. 2020;396(10254):839–52. doi: 10.1016/S0140-6736(20)31366-0

43. Han MW, Jeong SY, Suh CH, Park H, Guenette JP, Huang RY, et al. Incidence of immune effector cell-associated neurotoxicity among patients treated with CAR T-cell therapy for hematologic malignancies: systematic review and meta-analysis. Front Neurol. 2024;15:1392831. doi: 10.3389/fneur.2024.1392831

44. Shouval R, Strouse C, Kim S, Oloyede T, Ahmed S, Awan FT, et al. Cytokine Release Syndrome and Neurotoxicity Following CD19 CAR-T in B-Cell Lymphoma. Transplant Cell Ther. 2025;31(7):419–33. doi: 10.1016/j.jtct.2025.03.011

45. Jacobson CA, Locke FL, Ma L, Asubonteng J, Hu Z-H, Siddiqi T, et al. Real-World Evidence of Axicabtagene Ciloleucel for the Treatment of Large B Cell Lymphoma in the United States. Transplant Cell Ther. 2022;28(9):581.e1–e8. doi: 10.1016/j.jtct.2022.05.026

46. Ahmed N, Wesson W, Lutfi F, Porter DL, Bachanova V, Nastoupil LJ, et al. Optimizing the post-CAR T monitoring period in recipients of axicabtagene ciloleucel, tisagenlecleucel, and lisocabtagene maraleucel. Blood Adv. 2024;8(20):5346–54. doi: 10.1182/bloodadvances.2023012549