Electronic Patient Symptom Management Program to Support Patients Receiving Cancer Treatment at Home During the COVID-19 Pandemic.

Introduction

All patients with cancer will experience symptoms from either their disease or its treatment at some point in their care often with interference in activities of daily living and decreased quality of life. 1 It is well established that symptoms are under-reported and underassessed during clinical visits because of a variety of reasons such as time allotted for clinical visits, provider sensitivity toward specific treatment side effects, and the patients’ desire to minimize symptoms to receive treatment. 2 , 3 , 4 , 5 In addition, inadequate symptom control contributes to poor clinical outcomes, diminished quality of life, lost patient and family productivity, avoidable emergency department visits, hospitalizations and readmissions, poor treatment adherence, and decreased survival. 6 , 7 , 8 , 9

Multiple studies have shown proactive symptom management in response to patient reports of symptoms reduces symptom burden, decreases healthcare resource utilization, and improves survival, 5 , 10 , 11 , 12 , 13 yet widespread clinical adoption is lacking as cancer care institutions struggle to adopt patient-reported outcomes (PROs) into their workflow. Electronic PRO solutions are being developed and tested and show improved symptom experience. 12 , 14 , 15 , 16 , 17 , 18 Carevive Systems, Inc™, developed a remote symptom management solution that includes personalized, computable algorithms for tailored, evidence-based clinical symptom management with the promise of improving care by systematic delivery of symptom assessments that allow earlier identification and management of symptoms before they escalate and require unplanned clinic or emergency department visits and preventable hospitalizations. Three cancer centers were starting implementation of the Carevive electronic patient symptom management system (ePSMS) at the beginning of the COVID-19 pandemic. It became quickly apparent that patients would continue to receive cancer treatment but would need to remain away from the clinic as much as possible to follow pandemic restrictions. 19 , 20 , 21 , 22 , 23 The necessity for a system to monitor patients remotely, mitigate mild and moderate severity symptoms, and “alert” care providers of severe symptoms became paramount. This report details data on 284 patients who enrolled between September 2020 and October 2021.

Methods

Goals

The primary goal of this study was to test feasibility of implementing an ePSMS by the number of patients enrolled at each site, survey adherence, and survey completeness. Our secondary goal was to collect and describe the alerts reported to inform practice quality improvement and better understand the real-world patient experience. Given that remote symptom management was considered standard of care at the participating sites and all participants sign an end-user license agreement to allow use of their deidentified data, the institutional review board determined the study was exempt from institutional review board oversight.

Procedures

The design uses rolling enrollment methodology with patient PROs collected on a weekly basis and alerts conveyed to providers on a real-time basis. Eligible patients had one of 5 cancer diagnoses (breast, lung, or ovarian cancers, multiple myeloma, or acute myeloid leukemia), had a minimum of 12 weeks of planned chemotherapy, were 18 years or older, and had access to a smartphone, tablet, or computer. The selected cancers represent the initial diagnoses supported in the ePSMS with others in development for future expansion of the program. The 12-week minimum for weekly surveys is used by the sites as a criterion to justify the effort associated with enrollment and ongoing patient management remotely. The ePSMS was implemented at 3 US cancer centers, 2 academic centers in the Southeast, and 1 community center in the Mountain West.

Patients were enrolled in the cloud-based platform and notified via text or email that a symptom survey was available to be completed. Patients signed into a secure site to record their symptoms using a smart phone, tablet, or computer, and results were sent to the provider through the electronic medical record (EMR). Specified symptom levels such as severe or very severe or in some cases such as fever that is present, yes or no, were set to trigger a provider notification. Patients received written care guidelines for mild or moderate severity symptoms and a phone call from the care team for management of alerting symptoms. Of note, patients with acute myeloid leukemia who were hospitalized completed surveys and symptom alerts were reported to the inpatient clinical team.

Instrument

Patients complete weekly symptom reports as long as they are receiving treatment. Surveys delivered weekly include a core set of common symptoms derived from the National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE™). 24 , 25 The PRO-CTCAE library contains 124 items to measure 78 symptomatic adverse events. Reeve et al 26 (2014) identified a core set of 12 symptoms prevalent across diverse cancer populations (fatigue, insomnia, pain, appetite loss, dyspnea, cognitive problems, anxiety, nausea, depression [sadness], neuropathy, constipation, and diarrhea). Per the participating physicians, these symptoms, excluding cognitive problems, are part of the week survey along with some additional symptoms requested by providers (rash and mouth/throat sores).

Given that the PRO-CTCAE is a well-established instrument with proven reliability and validity, 27 patients did not receive specific instructions about the measures before starting the surveys. When the symptom occurs, severity is reported as mild, moderate, severe, or very severe. 28 Additional questions about frequency and interference with daily activities are included for some symptoms. A composite score of all attributes for each symptom is derived using an algorithm for mapping individual items for any given adverse event to a single (composite) numerical grade. 29

All data are collected either directly into the Carevive platform via integration with the site EMR and data entered directly by the site staff and from the patient surveys. A security and privacy provider manages all protected health information and any access to it. Carevive supports operations at all sites and has Health Insurance Portability and Accountability Act Business Associate Agreements in place with all data providers.

Data Structure

Data manipulation and analyses were performed using SAS version 9.4 30 and SPSS version 27.0. 31 The analyses focused on 3 data structures: (1) demographics with one record per patient (number of records = 282), (2) weekly data captured across all variables for each patient within a given week (number of data points = 119 088), and (3) patient-/week-level data set where data were aggregated at the weekly report level and then nested within a given patient (number of weekly nested reports = 2790). Patients provided a range of 1 to 57 weekly surveys with an average of 12.17 weeks (SD 9.54) of data; 55 of 282 patients (20%) had completed treatment and were no longer using the platform at the time of this analysis.

A critical focus of this work was to examine symptom levels immediately before an alerting episode (look back) and immediately after an alerting episode (look forward). We explored the relationship of the symptom report before the symptom alert report and the relationship of the symptom report after the alert report to explore whether an intervention was effective or not. Preliminary results showed mild symptom profiles accelerated within 1 week to trigger an alert. In the look back analyses, only “naive” alerts were studied, naive meaning that the alerting episode was either the first episode for a patient, or for patients with one or more alerting reports, an alert was considered naive if there was at least 1 nonalerting symptom report immediately before the alerting report of interest. In this look back process, the immediately preceding symptom levels were juxtaposed with the symptom alert levels to compute the alert change scores to indicate improvement, no change, or worsening of the symptom levels.

In the look forward process, the symptom report immediately after an alerting report was examined for change in symptom level after an alert. In a process like the selection of naive alerts in the look back analysis, only those records after an alert that themselves were nonalerting records were examined. In this process, the record immediately after an alert was juxtaposed with the alerting levels that generate the alert, and as above, change scores were computed to indicate improvement, no change, or worsening of the symptom levels.

In these processes (look back and look forward), the changes in symptom levels were quantified by computing change scores (Rising_delta = Composite Prealert − Composite Alert ). The same process was used to examine resolution. Change scores were computed as (Resolution_delta = Composite Alert − Composite Postalert ). These change scores could range from “worsening by 3 points” to “no change” to “improving by 3 points.” Both Rising_delta and Resolution_delta scores were examined for patterns of change overall and specifically for comparing patterns of change using divergent stacked bar charts as per recommendation by Zhou et al (2018). 32

Results

Participants had a mean age of 58.96 years (SD = 13.18), were mostly female (85.5%), were white (72.4%), and had a diagnosis of breast cancer (63.1%) (see Table 1 ). Patients completed 2860 of 3248 assigned surveys (88.0%), and 152 of 282 patients (54.0%) had symptom reports that rose to the level of an immediate notification sent to the provider. The most common symptoms reported were fatigue (84%), pain (54%), decreased appetite (52%), insomnia (51%), and nausea (50%) (see Table 2 ). The top symptoms that triggered alert reports were pain (42%), fatigue (32%), and insomnia (27%) (see Table 2 ). The top symptom alerts across the sample by cancer type, race, and sex were examined. These analyses were limited by sparse cells, and only the most robust were reported here. Cancer type was significantly associated with generation of pain alerts (χ 2 (4) = 11.38 P =.03.) The nature of this association was that patients with breast cancer were significantly more likely to trigger pain alerts. In addition, females were more likely than males to trigger pain alerts (χ 2 (2) =4.62 P =.03). Finally, we found no association between race and generation of alerts. Table 1 Sample characteristics (N = 282). Mean (SD) Age 58.96 (13.18) Number (%) Sex Male 41 (14.4) Female 241 (85.5) Race American native 5 (1.8) Asian 8 (2.9) African American 59 (21.7) White 197 (72.4) Other 3 (1.1) Missing 10 (3.5) Cancer diagnosis AML 29 (10.3) Breast 178 (63.1) Lung 19 (6.7) Myeloma 39 (13.8) Ovarian/gyn 17 (6.0) AML indicates acute myeloid leukemia; gyn, gynecological. Table 2 Incidence of symptom and alert reports. Symptom Alerted ∗ Within the 152 who reported alerts, % (95% CI) Overall † Reported binary data With N = 282, % (95% CI) Pain 42 (0.34-0.50) 47 (0.41-0.57) Constipation 22 (0.16-0.29) 48 (0.40-0.56) Insomnia 27 (0.20-0.34) 50 (0.42-0.58) Fatigue 32 (0.25-0.40) 84 (0.78-0.90) Decreased appetite 19 (0.13-0.25) 53 (0.45-0.62) Nausea 20 (0.13-0.26) 51 (0.43-0.60) Neuropathy 12 (0.07-0.17) 46 (0.38-0.55) Diarrhea 8 (0.04-0.12) 50 (0.42-0.58) Anxiety 9 (0.04-0.13) 47 (0.38-0.58) Rash 7 (0.03-0.11) 26 (0.19-0.34) Sadness/depression 5 (0.02-0.09) 28 (0.21-0.36) SOB 10 (0.05-0.15) 40 (0.32-0.48) Mouth/throat sores 5 (0.01-0.08) 34 (0.26-0.41) CI indicates, confidence interval; SOB, shortness of breath. ∗ Of those reporting yes, to having a symptom, the composite score triggered an alert (notification) to the provider. † Patients who reported yes to experiencing a symptom in the past 7 days.

With regard to the look back and look forward analyses, the change scores as discussed earlier were graphed as divergent stacked bar charts and are provided in Figure 1 A,B. In general, these figures show the improvement percentage and worsening percentage in composite scores between 1 week before an alert report (rising delta) and 1 week after an alert report (resolution delta) for 6 symptoms: pain, nausea, and insomnia in Figure 1 A and fatigue, constipation and diarrhea in Figure 1 B (graphs are separated only for ease of presentation). The graphs summarize data in a way that can be quickly compared. For instance, in Figure 1 A, the change from the prepain alert to the pain alert indicated 18.5% patients showed worsening by 3 composite score points, 44.0% showed worsening by 2 points, and 24.7% showed worsening by 1 point (the negative sign to the worsening percentages are a function of how the data are graphed.) In this figure, 12.4% showed no change from the week before the alert report to the alert report. In contrast, we see in the corresponding element immediately below, in the postpain alert, 23.2% of the patients showed improvement by 3 points, 34.8% improved by 2 points, and 16.8% improved by 1 point. Notably, 23.8% showed no change and a small percentage (< 3%) showed a worsening by 1 point. Figure 1 Divergent stacked bar charts show percent improvement and percent worsening in composite scores between one week prior to an alert and one week after an alert report. (A) Shows pain, nausea and insomnia scores. (B) Shows fatigue, constipation and diarrhea scores.

Discussion

This report describes the initial results of implementation of remote symptom management using an ePRSM system. Due to the challenges of patient care during the COVID-19 pandemic, enrollment was significantly less than expected; nevertheless, adherence and completion rates were higher than anticipated and comparable with clinical trial reports of 70% to 90% adherence. 33 , 34 , 35 High adherence is attributed to patients knowing that the reports were being monitored.

In addition, known barriers to adoption such as a lack of EMR integration and a lack of effective processes for clinical workflow integration 33 , 36 were overcome through integration of the ePSMS with the EMR and intense support for workflow design during the implementation period. Additional funds for staff support were provided and welcomed during the time of staffing shortages and reassignment. Other keys to adoption have been identified such as the immediate scoring and reporting, longitudinal display, clear workflow as to who receives and responds to alerts, and guideline-supported recommendations available. 37 , 38 , 39 The ePSMS includes each of these features.

The acceptability of digital PRO systems is growing with many clinical trials demonstrating decreased symptom severity with automated self-management and calls for poorly managed symptoms. 39 , 40 , 41 , 42 Analyzing the longitudinal symptom reports to track response to interventions provides the clinical practice feedback needed for continuous quality improvement. With more patients and further analysis of specific interventions, the potential usefulness of the data increases. In the look back for each symptom, patients who reported the symptom that did not rise to the level of an alert in the week before the alerting symptom report can be examined for effectiveness of the self-management measures and potential need for earlier calls by the care team. The look forward data can provide insight into the effectiveness (or ineffectiveness) of interventions after the alert. Using a standard approach and a validated PRO symptom instrument provides opportunities for benchmarking and sharing best practices.

Implementing the ePSMS in 3 institutions during the COVID-19 pandemic provided both opportunities and challenges. Patients with cancer are more at risk of morbidity and mortality with COVID-19 infection because of age, comorbidities, and the immunosuppression related to the tumor itself or treatment effects. 20 Millions of patients were transitioned to telemedicine visits to decrease risk for patients and staff alike, 21 and treatments were modified or delayed to reduce in person patient visits. 23 Not surprisingly, patients reported higher symptom burden and stress during the COVID-19 pandemic. 43

The ePSMS was welcomed as a tool to support patient care, yet widespread adoption was hampered by reduced personnel because of positive test results or reassigning staff to the support the increased demand for inpatient care. Staff spent enormous effort rescheduling patients for telemedicine visits, prescreening those who continued receiving treatments for COVID-19 symptoms, and adapting to evolving procedure and policy changes. Although these barriers were not addressed with ePSMS, enrollment of patients into the ePSMS supported efficient screening of patients and symptom management at home. In addition, staff could also quarantine at home and securely access the patient reports to assure continuity of care.

There are several limitations in this study. Patients were not educated on the meaning of symptom responses before taking a survey; therefore, internal consistency and reliability were not established. In addition, self-reporting of symptoms can potentially bias the responses. There is a preponderance of women in the study because of the diagnoses of breast and gynecological cancer and the sex imbalance may affect results. Eligibility required access to a smartphone, tablet, or computer. Implementation of this symptom reporting and management system may not be feasible or accessible to underserved and disadvantaged populations.

The look back and look forward analyses as portrayed in the divergent stacked bar graphs provide a useful snapshot of the rise and resolution of symptoms. Future analyses will focus on the interventions made in response to the alerting symptom reports to determine the effectiveness of the interventions in resolving the symptom. We will investigate unexpected findings such as the relatively large percentage of patients who did not improve or got worse after an alert, including the clinical and demographic factors associated with this lack of change. With increased sample size with continual enrollment, we expect to extend the prealert measurement to more than a single prealert measure so that we can better quantify the process of symptom change leading to an alert. In addition, we anticipate the use of clustering techniques to uncover latent groups of individuals based on change scores and to examine predictors of cluster membership and the impact of cluster membership on outcomes.

Conclusions

This article describes an ePSMS implemented at 3 institutions in the midst of the COVID-19 pandemic. Feasibility and acceptance were demonstrated through enrollment of 282 patients undergoing cancer treatment with a high rate of weekly survey completion. Patients reported symptoms at severe levels of 54% of the time and received self-management instructions and provider phone calls that resolved or decreased the severity of the symptom. Continued aggregation of homogenously reported symptoms provides opportunities for quality improvement, benchmarking outcomes, and sharing of best practices.

Article and Author Information

Author Contributions: Concept and design: Wujcik, W. Dudley, Brant

Acquisition of data: Wujcik, Gupta, Brandt

Analysis and interpretation of data: Wujcik, W. Dudley, M. Dudley, Gupta, Brant

Drafting of the manuscript: Wujcik, W. Dudley, M. Dudley

Critical revision of the paper for important intellectual content: Wujcik, W. Dudley, Brant

Statistical analysis: W. Dudley, M. Dudley, Gupta

Provision of study materials or patients: Wujcik

Obtaining funding: Wujcik

Administrative, technical, or logistic support: Wujcik

Supervision: Wujcik

Conflict of Interest Disclosures: Dr Wujcik and Ms Gupta are employed by Carevive Systems Inc. Dr W. Dudley and Mr M. Dudley reported providing statistical support through Piedmont Research Strategies Inc. No other disclosures were reported.

Funding/Support: The authors received no financial support for this research.

References

1 Reilly C.M. Bruner D.W. Mitchell S.A. A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment Support Care Cancer 21 6 2013 1525 1550 23314601 10.1007/s00520-012-1688-0 PMC4299699

2 Basch E. Jia X. Heller G. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes J Natl Cancer Inst 101 23 2009 1624 1632 19920223 10.1093/jnci/djp386 PMC2786917

3 Laugsand E.A. Sprangers M.A.G. Bjordal K. Skorpen F. Kaasa S. Klepstad P. Health care providers underestimate symptom intensities of cancer patients: a multicenter European study Health Qual Life Outcomes 8 2010 104 118 20858248 10.1186/1477-7525-8-104 PMC2949821

4 Atkinson T.M. Li Y. Coffey C.W. Reliability of adverse symptom event reporting by clinicians Qual Life Res 21 7 2012 1159 1164 21984468 10.1007/s11136-011-0031-4 PMC3633532

5 Basch E. Deal A.M. Kris M.G. Symptom monitoring with Patient-Reported Outcomes during routine cancer treatment: a randomized controlled trial [published correction appears in J Clin Oncol. 2016;34(18):2198] [published correction appears in J Clin Oncol. 2019;37(6):528] J Clin Oncol 34 6 2016 557 565 26644527 10.1200/JCO.2015.63.0830 PMC4872028

6 Mayer D.K. Travers D. Wyss A. Leak A. Waller A. Why do patients with cancer visit emergency departments? Results of a 2008 population study in North Carolina J Clin Oncol 29 19 2011 2683 2688 21606431 10.1200/JCO.2010.34.2816 PMC3139372

7 Kotronoulas G. Kearney N. Maguire R. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials J Clin Oncol 32 14 2014 1480 1501 24711559 10.1200/JCO.2013.53.5948

8 Howell D. Molloy S. Wilkinson Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors Ann Oncol 26 9 2015 1846 1858 25888610 10.1093/annonc/mdv181

9 Denis F. Yossi S. Septans A.-L. Improving survival in patients treated for a lung cancer using self-evaluated symptoms reported through a web application Am J Clin Oncol 40 5 2017 464 469 25811297 10.1097/COC.0000000000000189

10 Basch E. Deal A.M. Dueck A.C. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment JAMA 318 2 2017 197 198 28586821 10.1001/jama.2017.7156 PMC5817466

11 Denis F. Lethrosne C. Pourel N. Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients [published correction appears in J Natl Cancer Inst. 2018;110(4):436] J Natl Cancer Inst 109 9 2017 10.1093/jnci/djx029 28423407

12 Fjell M. Langius-Eklof A. Nilsson M. Wengström Y. Sundberg K. Reduced symptom burden with the support of an interactive app during neoadjuvant chemotherapy for breast cancer- a randomized controlled trial Breast 51 2020 85 93 32247251 10.1016/j.breast.2020.03.004 PMC7375618

13 Mierzynska J. Piccinin C. Pe M. Prognostic value of patient-reported outcomes from international randomised clinical trials on cancer: a systematic review Lancet Oncol 20 12 2019 e685 e698 31797795 10.1016/S1470-2045(19)30656-4

14 Judson T.J. Bennett A.V. Rogak L.J. Feasibility of long-term patient self-reporting of toxicities from home via the internet during routine chemotherapy J Clin Oncol 31 20 2013 2580 2586 23733753 10.1200/JCO.2012.47.6804 PMC3699724

15 Ciani O. Cucciniello M. Petracca F. Lung Cancer App (LuCAPP) study protocol: a randomised controlled trial to evaluate a mobile supportive care app for patients with metastatic lung cancer BMJ Open 9 2 2019 e025483 10.1136/bmjopen-2018-025483 PMC6398637 30772862

16 Avery K.N.L. Richards H.S. Portal A. Developing a real-time electronic symptom monitoring system for patient after discharge following cancer-related surgery BMC Cancer 10 1 2019 463 478 10.1186/s12885-019-5657-6 PMC6524308 31101017

17 Gandrup J. Ali S.M. McBeth J. van der Veer S.N. Dixon W.G. Remote symptom monitoring integrated into electronic health records: a systematic review J Am Med Inform Assoc 27 11 2020 1752 1763 32968785 10.1093/jamia/ocaa177 PMC7671621

18 Greer J.A. Jacobs J.M. Pensak N. Randomized trial of a smartphone mobile app to improve symptoms and adherence to oral therapy for cancer J Natl Compr Canc Netw 18 2 2020 133 141 32023526 10.6004/jnccn.2019.7354

19 Moazzami B. Razavi-Khorasani N. Dooghale-Modghadam A. COVID-19 and telemedicine. Immediate action required for maintaining healthcare providers well-being J Clinic Virol 126 2020 104345 10.1016/j.jcv.2020.104345 PMC7129277 32278298

20 Marandino L. Necchi A. Aglietta M. Di Maio M. COVID-19 emergency and the need to speed up the adoption of electronic patient-reported outcomes in cancer clinical practice JCO Oncol Prac 16 6 2020 295 299 10.1200/OP.20.00237 PMC7292478 32364846

21 Schrag D. Hershman D.L. Basch E. Oncology practice during the COVID-19 pandemic JAMA 323 20 2020 2005 2006 32282023 10.1001/jama.2020.6236

22 COVID-19 & telemedicine FAQs. AMA https://www.ama-assn.org/practice-management/digital/telehealth-resource-center-news

23 Waisberg F. Enrico D. Angel M. Chacón M. Cancer treatment adaptations in the COVID-19 era JCO Oncol Prac 6 6 2020 305 307 10.1200/OP.20.00218 PMC7292479 32324487

24 Hay J.L. Atkinson T.M. Reeve B.B. Cognitive interviewing of the US National Cancer Institutes’ Patient-Reported outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) Qual Life Res 23 1 2013 257 269 23868457 10.1007/s11136-013-0470-1 PMC3896507

25 Basch E, Reeve BB, Mitchell SA, et al. Development of the National Cancer Institute’s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Natl Cancer Inst. 2013;106(9):dju244. 10.1093/jnci/dju244 PMC4200059 25265940

26 Reeve B.B. Mitchell S.A. Dueck A. Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials J Natl Cancer Inst 106 7 2014 dju129 10.1093/jnci/dju129 PMC4110472 25006191

27 Dueck A.C. Mendoza T.R. Mitchell S.A. Validity and reliability of the US National Cancer Institutes’ Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) JAMA Oncol 1 8 2015 1051 1059 26270597 10.1001/jamaoncol.2015.2639 PMC4857599

28 Atkinson T.M. Hay J.L. Dueck A.C. What do “none,” “mild,” “moderate,” “severe,” and “very severe,” mean to patients with cancer? Content validity of PRO-CTCAE™ response scales J Pain Symptom Manag 55 3 2018 e3 e6 10.1016/j.jpainsymman.2017.10.024 PMC6317851 29129739

29 Basch E. Becker C. Rogak J. Composite grading algorithm for the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) Clin Trials 18 1 2021 104 114 33258687 10.1177/1740774520975120 PMC7878323

30 SAS version 9.4. SAS www.SAS.com

31 SPSS version 27.0 IBM https://www.ibm.com/analytics/spss-statistics-software

32 Zhou X. Eid D. Gnanasakthy A. Methods for reporting the patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE) data in cancer clinical trials Value Health 21 suppl 1 2018 S226

33 Basch E. Patient-reported outcomes-harnessing patients’ voices to improve clinical care N Engl J Med 376 2 2017 105 108 28076708 10.1056/NEJMp1611252

34 Benze G. Nauck F. Alt-Epping B. PROutine: a feasibility study assessing surveillance of electronic patient reported outcomes and adherence via smartphone app in advanced cancer Ann Palliat Med 8 2 2019 104 111 29156896 10.21037/apm.2017.07.05

35 Wright A.A. Raman N.N. Staples P. The HOPE pilot study: harnessing patient-reported outcome and biometric data to enhance cancer care JCO Clin Cancer Inform 2 2 2018 1 12 10.1200/CCI.17.00149 PMC6556148 30652585

36 Harle C.A. Listhaus A. Covarrubias C.M. Overcoming barriers to implementing patient-reported outcomes in an electronic health record: a case report J Am Med Inform Assoc 23 2016 74 79 26159464 10.1093/jamia/ocv085 PMC5009936

37 Duncan E.A. Murray J. The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review BMC Health Serv Res 12 1 2012 96 105 22506982 10.1186/1472-6963-12-96 PMC3358245

38 Jensen R.E. Rothrock N.E. DeWitt E.M. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care Med Care 53 2 2015 153 159 25588135 10.1097/MLR.0000000000000289 PMC4801509

39 Mooney K. Berry D. Whisenant M. Sjoberg D. Improving cancer care through the patient experience: how to use patient-reported outcomes in clinical practice Am Soc Clin Oncol Educ Book 37 2017 695 704 28561689 10.1200/EDBK_175418

40 Basch E. Stover A. Schrag D. Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: findings from the PRO-TECT trial JCO J Clin Cancer Inform 4 2020 947 957 10.1200/CCI.20.00081 PMC7768331 33112661

41 Cleeland C.S. Wang X.S. Shi Q. Automated symptom alerts reduce postoperative symptom severity after cancer surgery: a randomized controlled clinical trial J Clin Oncol 29 8 2011 994 1000 21282546 10.1200/JCO.2010.29.8315 PMC3068055

42 Chung A.E. Basch E.M. Incorporating the patient’s voice into electronic health records through patient-reported outcomes as the “review of symptoms” J Am Med Inform Assoc 22 2015 914 916 25614143 10.1093/jamia/ocu007 PMC5009898

43 Miaskowski C. Paul S.M. Snowberg K. Stress and symptom burden in oncology patients during the COVID-19 pandemic J Pain Symptom Manag 60 5 2020 e25 e33 10.1016/j.jpainsymman.2020.08.037 PMC7462969 32889039