Integration of Patient-Reported Outcome Measures in the Electronic Health Record: The Veterans Affairs Experience.

INTRODUCTION

Integration of patient-reported outcome (PRO) data in electronic health records (EHRs) is challenging, requiring consideration of systems of data acquisition, patient population, data display, provider education, and guidelines and pathways. 1 , 2 Such integration has the potential to bring distinct benefits, including alignment of symptom data with laboratory and radiologic data representing patient status, focus on patient-centered care, and opportunities for quality improvement initiatives. 1 , 3 There are limited but increasing numbers of publications that report on integrating PRO results in EHR systems, but few provide outcomes of the integration. 2 , 4 Despite the challenges of integrating PRO data in EHR systems, the federal government has strongly supported efforts to do so through the National Institutes of Health (NIH) and the Cancer Moonshot Initiative. 2

CONTEXT

Key Objective

Integrating patient-reported outcome (PRO) data in the electronic health record (EHR) system includes the patient's perspective as part of the health information reviewed and acted upon by providers. This study reports on the implementation and use of an integrated PRO measurement tool in hematology-oncology clinics in the Veterans Administration (VA) system and is, to our knowledge, the largest real-world experience in this setting to date.

Knowledge Generated

The Veteran Symptom Assessment System (VSAS) was implemented and used over a 5-year period, documenting the burden of cancer-related symptoms initially and over time and in the population as a whole and in subgroups and describing the effect of severe symptom burden on survival of Veterans with cancer.

Relevance

The integration of VSAS in the VA EHR demonstrates the feasibility of PRO integration, meets federal and professional recommendations regarding cancer care, and should be used in concert with other health data to provide cutting edge and comprehensive patient care.

Various PRO measures have been studied to systematically assess cancer symptoms. Patients with cancer commonly experience a variety of symptoms, including fatigue and pain, 5 , 6 which significantly compromise patients' quality of life and shorten their survival. 7 - 10 The NIH and ASCO recommend regular assessment of symptoms followed by continuous interventions to manage identified symptoms. 11 , 12 Within the context of research studies, implementation of PRO assessments is associated with improvements in quality of life and survival, 13 but oncology providers are often unaware of the severity and extent of their patients' symptoms. 9 Measuring patients' symptom, integrating it in the EHR system, and easily sharing it with oncology providers would address unmet needs as part of comprehensive cancer care.

The Veterans Administration (VA) is the largest integrated health care system in the United States. In 2010, there were approximately 46,170 incident invasive cancer cases reported in the VA Central Cancer Registry, 14 constituting approximately 3% of all cancers in the United States. Similar to cancers observed in males in the United States, the most common cancers diagnosed in the VA are prostate, lung/bronchus, and colon/rectum. The VA uses a shared EHR, VistA-CPRS, which presents a unique opportunity to implement an integrated PRO tool at a population level.

To meet quality recommendations for cancer symptom assessment, the VA developed a PRO called the Veterans Symptom Assessment System (VSAS), which is based on the Edmonton Symptom Assessment System (ESAS), 15 , 16 an established and validated palliative care symptom measurement tool. We describe the implementation of VSAS, integrated as part of VistA-CPRS at ambulatory hematology-oncology clinics in two VA sites over a 5-year period, assess validity and internal consistency of VSAS, and describe the cancer symptom burden and its clinical implications at the population level within the VA.

METHODS

Data Extraction and Cohort Identification

Data were extracted from multiple domains in the VA Corporate Data Warehouse, including the Veterans Affairs Cancer Registry System (VACRS) and Health Factor domains. Patients were included in this study if they received at least one VSAS assessment and were followed from that point onward, until death administrative censoring or lost to follow-up. The study was approved by the Institutional Review Boards of the investigators at the University of Utah (IRB of record for the VA Salt Lake City) and the Durham VA Health Care System. Both IRBs waived the requirement to obtain informed consent for this retrospective study.

Patient and Disease Characteristics

Patient demographic information was extracted, including date of birth, sex, race, and ethnicity. Patient residence was classified into metropolitan or rural (nonmetropolitan) using rural-urban commuting area (RUCA) codes. 17 Disease information was extracted, when available, from the VACRS, including histology, site, and year of diagnosis, or from International Classification of Diseases (ICD)-9-CM and ICD-10 cancer or psychiatric codes used within a 1-year lookback before the first VSAS administration. Quan's algorithm 18 was used to calculate the Charlson Comorbidity Index on the basis of ICD-9-CM and ICD-10 codes in inpatients and outpatient visits within a 1-year lookback before first VSAS administration. Vital status and date of death for patients who died during the study were obtained from Corporate Data Warehouse.

Statistical Analysis

VSAS adoption was calculated as the proportion of hematology/oncology in-person visits that included a VSAS assessment and vital signs out of all hematology/oncology visits, including and after the first VSAS assessment, with visits occurring on the same day combined into one.

Cronbach's alpha and Spearman's correlation coefficient were used to measure internal consistency and reliability of VSAS. 19 , 20 We calculated Cronbach's alpha between the emotional cluster and the physical cluster 21 to assess internal consistency and reliability. The chi-squared test was used to compare symptom scores in patient subgroups.

We retrieved VSAS-recorded symptoms at baseline, defined as the first hematology/oncology visit where VSAS scores were documented, in all patients in this cohort and VSAS assessments 1-6 months after the first assessment in a subset of patients. Symptom severity was defined as absent (zero), mild (1-3), moderate (4-6), or severe (7-10).

The Kaplan-Meier method was used to compare the overall survival (until date of death or end of the study on December 31, 2019) of stage IV patients with higher symptom burden versus lower symptom burden. The Kolmogorov-Smirnov test was used to evaluate the difference in distribution of cancer diagnoses used in this survival analysis. Unadjusted two-sample pairwise comparisons using proportion hypothesis tests were used to compare symptoms in subgroups.

RESULTS

Description of VSAS and Implementation

In October 2013, the original VSAS model was developed at the Durham VA Medical Center. VSAS is based on the ESAS, an established and validated symptom measurement tool in patients with advanced cancer receiving palliative care. 15 , 16 Table 1 lists the 13 VSAS components, their definitions, and ESAS correlate components.

Ten nurses were involved in usability testing in 2014, and recommendations were made to clarify definitions of symptoms, to provide an explanation of the tool within the EHR, and to align symptom names such that zero indicates the absence of the symptom and 10 indicates the worst severity of the symptom. On the basis of usability testing, modifications outlined in Table 1 were made to the template, which was incorporated into the VA EHR. VSAS was implemented in the hematology-oncology clinics at the Durham, NC, and Columbia, SC, VA Medical Centers. VSAS was administered by nurses and was documented at each clinic visit in notes in the patients' electronic chart (screenshots are included in the Data Supplement, online only).

Between January 2015 and October 2019, 9,058 patients with cancer and blood disorders at the hematology-oncology clinics in the Durham, NC (n = 5,933), and the Columbia, SC (n = 3,125), VA Medical Centers had VSAS recorded 43,883 times (33,772 VSAS in NC and 10,111 in SC). The median number of VSAS administration per patient was three (range 1-56). Of the 9,058 patients who had VSAS administered, 5,960 had a second VSAS assessment, occurring at a median of 77 days after the first VSAS measurement (range 1-1,636), with 2,862 patients having a second VSAS assessment 1-6 months after the first measure.

Patient Characteristics

Of the 9,058 Veterans who had VSAS administered, 8,106 (89%) were male and 952 (11%) were female (Table 2 ), consistent with the gender breakdown of Veterans who receive care in the VA. 14 The median age of patients at the time of their first VSAS assessments was 67 (range 21-97) years. Four thousand seven hundred twenty-nine (52%) patients identified as non-Hispanic White, 3,879 (43%) identified as African American, and 282 (3%) identified as other racial groups. The racial breakdown in this cohort reflects a higher proportion of African Americans than the proportion of African-American Veterans residing in the Carolinas (22%). 22 , 23 The cohort of patients in this study had a higher proportion of older and minority individuals compared with those described in other studies and in prospective clinical trials. 9 , 13 , 24 , 25 The proportion of patients who live in rural (nonmetropolitan) areas is similar to the population in general in North and South Carolina. 26 , 27

In 3,151 patients for whom cancer diagnosis information was available from the VACRS, the first VSAS assessment occurred at a median of 269 days from the date of diagnosis (range 0-6,609 days). Six thousand three hundred six patients had a cancer diagnosis, using VACRS data and/or ICD-9 and ICD-10 codes (Table 2 ), with the remaining patients having precancerous conditions, hematologic neoplasia not classified as cancer, and benign hematologic conditions.

VSAS Structure and Internal Consistency

We briefly evaluated VSAS validity by describing the internal structure and consistency of all items globally and separate symptom clusters: emotional (anxiety, depression, and distress), physical (anorexia, constipation, diarrhea, drowsiness, nausea, shortness of breath with exertion, shortness of breath at rest, tiredness, vomiting, and pain), and gastrointestinal (anorexia, constipation, diarrhea, nausea, and vomiting). We found high global consistency with Cronbach's alpha of .859, as well as the emotional cluster (Cronbach's alpha of .884), physical cluster (Cronbach's alpha of .803), and gastrointestinal cluster (Cronbach's alpha of .662).

There were higher correlation between symptoms expected to occur at the same time, for example, between anxiety, depression, and distress ( r = 0.63-0.67), and lower correlation between symptoms expected to not occur together, for example, constipation and diarrhea ( r = 0.16; Fig 1 ). The symptoms of shortness of breath were significantly more prevalent in patients with lung versus prostate cancer, for any degree and for severe shortness of breath ( P < .001 for all). These different analyses support our expectations regarding the performance of VSAS and suggest that VSAS is a consistent, reliable, and valid tool with which cancer-related symptoms in Veterans can be measured.

Assessment of Symptom Burden

Symptom burden reported by Veterans upon first VSAS administration was high (Table 3 and Fig 2 ). The most frequently reported symptoms and severe symptoms were tiredness (68% and 27%, respectively), shortness of breath with exertion (49% and 16%, respectively), and pain (45% and 17%, respectively). More than 20% of Veterans in this cohort reported the presence of tiredness, shortness of breath with exertion, pain, drowsiness, anxiety, depression, anorexia, distress, and constipation. The least reported symptom was vomiting (10% total and 2% severe).

We also evaluated subgroups of patients to evaluate the effect of patient- and cancer-related factors (race, psychiatric comorbidity, 28 and stage) on symptoms reported. African Americans experienced more severe levels of pain (20% v 14%) and distress (12% v 8%) than non-Hispanic White patients ( P unadj < .001). In addition, patients with a history of psychiatric disorder (n = 3,600) experienced a higher burden of any symptom and more severe symptoms than those who did not (n = 3,782, P unadj < .001), including severe distress (14% v 6%) and severe pain (21% v 13%). Finally, in patients diagnosed with cancer 0-6 months before first VSAS measurement, those with advanced-stage cancer (stage III or IV; n = 325) had significantly more severe symptoms than those with early-stage cancer (stage I or II; n = 193), including distress (10% v 4%, P unadj = .049) and pain (23% v 11%, P unadj = .015). Together, these subgroup analyses suggest that patient- and cancer-specific factors contribute to how cancer-related symptoms are experienced.

Severe Symptom Burden and Clinical Outcomes

We assessed the impact of symptom severity on overall survival (OS), focusing on patients diagnosed with stage IV cancer with a first VSAS assessment < 6 months (range 0-176 days) from diagnosis (n = 202). Similar to the entire cohort, the most frequently reported symptoms (any degree or severe) were tiredness (68% and 30%, respectively), shortness of breath with exertion (51% and 15%, respectively), and pain (52% and 26%, respectively).

In addition, a higher symptom burden, defined a priori as two or more symptoms rated as severe, was significantly associated with reduced OS (hazard ratio, 1.64; 95% CI, 1.16 to 2.30; P = .0047). As seen in Figure 3 , the median OS after initial VSAS administration of patients with lower symptom burden was 10.4 months (95% CI, 8 to 16.5), whereas the median OS of patients with higher symptom burden was 5.7 months (95% CI, 3.3 to 12.5). There was a significant association between the number of severe symptoms and inferior overall survival (hazard ratio, 1.12; 95% CI, 1.04 to 1.20; P = .001). These results are consistent with findings using other PRO measurement tools, 29 suggesting that higher symptom burden could be a marker of more aggressive cancer.

Longitudinal Changes in Symptom Burden

To evaluate changes in symptom burden, influenced by symptom management, cancer therapies, or passive changes, we compared initial and second VSAS assessments in patients where the two measurements were 1-6 months apart (n = 2,862), and change is measured as a difference of at least one point. 30 During this period, there were no required interventions on the basis of the VSAS assessment results.

As seen in Figure 4 , the majority of patients with symptoms initially present experienced improvements (median 66%, range 51%-81%). Symptoms of tiredness, pain, and shortness of breath with exertion had the lowest fraction of patients with improvement (51%, 55%, and 59%, respectively), whereas vomiting, nausea, and shortness of breath at rest had the highest fraction of patients with improvements (81%, 73%, and 72%, respectively).

We next stratified patients by their initial symptom severity. Although most patients with symptoms of zero initially remained asymptomatic, a substantial fraction developed symptoms of tiredness and pain (46% and 23%, respectively). For patients with mild, moderate, and severe tiredness initially, it improved in 42%, 45%, and 64%, but worsened in 42%, 36%, and 17%, respectively. For patients with mild, moderate, and severe pain initially, it improved in 55%, 53%, and 58%, but worsened in 30%, 25%, and 12%, respectively. Together, these data suggest that severe symptoms are more likely to be addressed and managed, whereas mild to moderate symptoms worsen in a substantial fraction of patients.

DISCUSSION

To our knowledge, this is the first implementation of an integrated PRO measure in the VA's EHR system and is the first large-scale evaluation of symptoms in Veterans with cancer and blood disorders receiving ambulatory care in the VA health system.

VSAS was developed using an established PRO measure (ESAS), modified after VA nurse usability testing, and integrated into the VA's EHR system. Although VSAS was developed to be used as a clinical tool during routine patient care, the evaluation of the aggregated data after 5 years of use is an important demonstration of the performance and utility of VSAS. The results obtained and the descriptions of data acquisition, patient population, and data display within the VA EHR are necessary steps toward fully integrated PRO data in the EHR system. 1 , 2

In addition, the cancer symptom data obtained from this implementation describe a Veteran population in need of improved cancer symptom management. We found that within two VA outpatient hematology-oncology clinics, unselected Veterans were mostly male and a high proportion were self-identified as African Americans (AAs). We suspect that this cohort has a high proportion of AA patients because (1) North Carolina and South Carolina have higher AA population than other parts of the United States, 22 , 23 (2) Veterans who receive care in the VA tend to more often be of under-represented minorities or of low income than Veterans who receive care outside of the VA, 31 , 32 and (3) certain types of cancers are represented more frequently in AA, such as advanced prostate cancer and multiple myeloma. 33 We identified a lower prevalence of prostate cancer and a higher prevalence of multiple myeloma in our cohort compared with the VA Central Cancer Registry in 2010, 14 which is likely due to exclusion of surgical or radiation oncology clinics in this study that more often treat patients with localized solid malignancies. Together, the cohort described here represents a real-world Veteran population in the Southeast United States.

The distribution of reported symptoms was similar to those described in other studies that used ESAS as a PRO measure, 24 , 25 but prevalence of severe symptoms was lower in our cohort. Differences in patient demographics (eg, sex and race) and/or clinic location (hematology-oncology v palliative care clinic) may contribute to the observed differences. In subgroup analyses, we found that patient-specific (demographics and comorbidities) and cancer-specific factors (stage) can contribute to the presence and severity of symptoms. We also found that patients with metastatic cancer reporting numerous symptoms in the severe range had shorter overall survival, consistent with other studies, 29 and that over short periods of time, significant changes in experienced symptoms occur. 30 These results add to existing data regarding symptoms experienced by Veterans and racial minorities with cancer. 34 - 38 Because of demographic factors, military exposures, and comorbidities, the Veteran population is distinct and may experience cancer differently than non-Veterans. Together, these results underscore the importance of evaluating multiple factors to identify patients at elevated risk of experiencing a high symptom burden, which can only be achieved if PRO data are integrated as part of the EHR and are easily accessible to providers.

A limitation of this study is the heterogeneity of the clinic population, which included patients with nonmalignant hematologic and premalignant conditions and those receiving survivorship care, which could have reduced the proportion of patients reporting the presence of cancer-related or treatment-related symptoms. Because patients with conditions other than cancer who receive care in hematology-oncology clinics were included in this study, the results may not be fully applicable to homogeneous groups of patients with cancer. In addition, VSAS assessments were not obtained at regularly defined intervals, and thus, patients with more frequent visits had more frequent VSAS measurements, which could affect symptom reporting. Because symptom interventions were not mandated as part of this study, improving or worsening symptoms might have reflected passive changes in symptom severity, changes in cancer status, changes in comorbidities, and/or symptom interventions. Another limitation is that cancer diagnoses in this study may be incomplete because of missing or delayed data entry in the VACRS or because of incorrect encounter coding.

As a health care system that provides care to a sizable proportion of patients with cancer in the United States, 14 it is essential that the VA incorporates cancer symptom measurement as part of standard cancer care. VSAS is an integrated tool to measure, document, and display cancer-related symptoms within the VA EHR. Opportunities exist to build upon this experience, not only with increased facility implementation of the existing VSAS template but also with future point-of-care tools that combine patient-specific data, cancer-specific data, and symptom data to identify patients in real time at high risk of experiencing severe, long-lasting, and/or numerous symptoms, and give providers opportunities to immediately intervene and improve quality of life.

ACKNOWLEDGMENT

We thank Sophia Smith, M.S.W., PhD, and James Tulsky, M.D., for reviewing this manuscript.

Daphne R. Friedman

Stock and Other Ownership Interests: Biogen (I), CVS Health Corp (CVS) (I), Johnson & Johnson (JNJ) (I), United Health Group (I), Zoetis Inc (I)

Research Funding: Loxo/Lilly, Oncopeptides, TG Therapeutics (Inst)

Zachary Burningham

Research Funding: AbbVie, Genentech/Roche, Pharmacyclics

Michael J. Kelley

Research Funding: Novartis (Inst), AstraZeneca (Inst), Bristol Myers Squibb (Inst), Regeneron (Inst), Genentech (Inst)

Other Relationship: IBM (Inst)

Open Payments Link: https://openpaymentsdata.cms.gov/physician/827136

Ahmad S. Halwani

Research Funding: Bristol Myers Squibb (Inst), Kyowa Hakko Kirin (Inst), Roche/Genentech (Inst), AbbVie/Genentech (Inst), AbbVie (Inst), Immune Design (Inst), miRagen (Inst), Amgen (Inst), Seattle Genetics (Inst), Genentech (Inst), Takeda (Inst), Pharmacyclics (Inst), Bayer (Inst)

Travel, Accommodations, Expenses: Pharmacyclics, AbbVie, Seattle Genetics, Immune Design

No other potential conflicts of interest were reported.

DISCLAIMER

This is a US Government work. There are no restrictions on its use.

PRIOR PRESENTATION

Presented in part at the 2015 AVAHO meeting, October 2-4, Washington DC and the 2018 AVAHO meeting, September 28-30, Chicago, IL.

SUPPORT

Supported by the VA National Oncology Program Office and with resources and the use of facilities at VA SLC Health Care System, Salt Lake City, UT, and the Durham VA Healthcare System, Durham, NC.

AUTHOR CONTRIBUTIONS

Conception and design: Daphne R. Friedman, Vikas Patil, Susan Hamilton-Hill, Michael J. Kelley, Ahmad S. Halwani

Provision of study materials or patients: Vikas Patil, Chunyang Li

Collection and assembly of data: Daphne R. Friedman, Vikas Patil, Kelli M. Rassmussen, Zachary Burningham, Ahmad S. Halwani

Data analysis and interpretation: Daphne R. Friedman, Vikas Patil, Chunyang Li, Kelli M. Rassmussen, Michael J. Kelley, Ahmad S. Halwani

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center .

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians ( Open Payments ).

Daphne R. Friedman

Stock and Other Ownership Interests: Biogen (I), CVS Health Corp (CVS) (I), Johnson & Johnson (JNJ) (I), United Health Group (I), Zoetis Inc (I)

Research Funding: Loxo/Lilly, Oncopeptides, TG Therapeutics (Inst)

Zachary Burningham

Research Funding: AbbVie, Genentech/Roche, Pharmacyclics

Michael J. Kelley

Research Funding: Novartis (Inst), AstraZeneca (Inst), Bristol Myers Squibb (Inst), Regeneron (Inst), Genentech (Inst)

Other Relationship: IBM (Inst)

Open Payments Link: https://openpaymentsdata.cms.gov/physician/827136

Ahmad S. Halwani

Research Funding: Bristol Myers Squibb (Inst), Kyowa Hakko Kirin (Inst), Roche/Genentech (Inst), AbbVie/Genentech (Inst), AbbVie (Inst), Immune Design (Inst), miRagen (Inst), Amgen (Inst), Seattle Genetics (Inst), Genentech (Inst), Takeda (Inst), Pharmacyclics (Inst), Bayer (Inst)

Travel, Accommodations, Expenses: Pharmacyclics, AbbVie, Seattle Genetics, Immune Design

No other potential conflicts of interest were reported.

REFERENCES

1. Gensheimer SG, Wu AW, Snyder CF, et al. Oh, the places we'll go: Patient-reported outcomes and electronic health records. Patient. 2018;11:591–598. doi: 10.1007/s40271-018-0321-9.

2. Jim HSL, Hoogland AI, Brownstein NC, et al. Innovations in research and clinical care using patient-generated health data. CA Cancer J Clin. 2020;70:182–199. doi: 10.3322/caac.21608.

3. Snyder CF, Jensen RE, Segal JB, et al. Patient-reported outcomes (PROs): Putting the patient perspective in patient-centered outcomes research. Med Care. 2013;51:S73–S79. doi: 10.1097/MLR.0b013e31829b1d84.

4. Tiase VL, Hull W, McFarland MM, et al. Patient-generated health data and electronic health record integration: A scoping review. JAMIA Open. 2020;3:619–627. doi: 10.1093/jamiaopen/ooaa052.

5. Cleeland CS, Zhao F, Chang VT, et al. The symptom burden of cancer: Evidence for a core set of cancer-related and treatment-related symptoms from the Eastern Cooperative Oncology Group Symptom Outcomes and Practice Patterns study. Cancer. 2013;119:4333–4340. doi: 10.1002/cncr.28376.

6. Reilly CM, Bruner DW, Mitchell SA, et al. A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Support Care Cancer. 2013;21:1525–1550. doi: 10.1007/s00520-012-1688-0.

7. Hamer J, McDonald R, Zhang L, et al. Quality of life (QOL) and symptom burden (SB) in patients with breast cancer. Support Care Cancer. 2017;25:409–419. doi: 10.1007/s00520-016-3417-6.

8. Matzka M, Kock-Hodi S, Jahn P, et al. Relationship among symptom clusters, quality of life, and treatment-specific optimism in patients with cancer. Support Care Cancer. 2018;26:2685–2693. doi: 10.1007/s00520-018-4102-8.

9. Quinten C, Maringwa J, Gotay CC, et al. Patient self-reports of symptoms and clinician ratings as predictors of overall cancer survival. J Natl Cancer Inst. 2011;103:1851–1858. doi: 10.1093/jnci/djr485.

10. Mierzynska J, Piccinin C, Pe M, et al. Prognostic value of patient-reported outcomes from international randomised clinical trials on cancer: A systematic review. Lancet Oncol. 2019;20:e685–e698. doi: 10.1016/S1470-2045(19)30656-4.

11. Patrick DL, Ferketich SL, Frame PS, et al. National Institutes of Health State-of-the-Science Conference Statement: Symptom management in cancer: Pain, depression, and fatigue, July 15-17, 2002. J Natl Cancer Inst. 2003;95:1110–1117. doi: 10.1093/jnci/djg014.

12. American Society of Clinical Oncology . Quality Oncology Practice Initiative. https://practice.asco.org/sites/default/files/drupalfiles/QOPI-2019-Round-1-Reporting-Tracks-Public-Posting.pdf

13. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: A randomized controlled trial. J Clin Oncol. 2016;34:557–565. doi: 10.1200/JCO.2015.63.0830.

14. Zullig LL, Sims KJ, McNeil R, et al. Cancer incidence among patients of the U.S. Veterans Affairs Health Care System: 2010 update. Mil Med. 2017;182:e1883–e1891. doi: 10.7205/MILMED-D-16-00371.

15. Hui D, Bruera E. The Edmonton Symptom Assessment System 25 years later: Past, present, and future developments. J Pain Symptom Manage. 2017;53:630–643. doi: 10.1016/j.jpainsymman.2016.10.370.

16. Nekolaichuk C, Watanabe S, Beaumont C. The Edmonton Symptom Assessment System: A 15-year retrospective review of validation studies (1991-2006) Palliat Med. 2008;22:111–122. doi: 10.1177/0269216307087659.

17. United States Department of Agriculture Economic Research Service . Rural-Urban Commuting Area Codes. https://www.ers.usda.gov/data-products/rural-urban-commuting-area-codes/

18. Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–1139. doi: 10.1097/01.mlr.0000182534.19832.83.

19. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am J Med. 2006;119:166 e7–16. doi: 10.1016/j.amjmed.2005.10.036.

20. Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ. 2011;2:53–55. doi: 10.5116/ijme.4dfb.8dfd.

21. Cheung WY, Le LW, Zimmermann C. Symptom clusters in patients with advanced cancers. Support Care Cancer. 2009;17:1223–1230. doi: 10.1007/s00520-009-0577-7.

22. Veterans Administration . VetPop2018: A Brief Description. https://www.va.gov/vetdata/docs/Demographics/New_Vetpop_Model/VP_18_A_Brief_Description.pdf

23. Veterans Administration . National Center for Veterans Analysis and Statistics. https://www.va.gov/vetdata/Veteran_Population.asp

24. Barbera L, Seow H, Howell D, et al. Symptom burden and performance status in a population-based cohort of ambulatory cancer patients. Cancer. 2010;116:5767–5776. doi: 10.1002/cncr.25681.

25. Kang JH, Kwon JH, Hui D, et al. Changes in symptom intensity among cancer patients receiving outpatient palliative care. J Pain Symptom Manage. 2013;46:652–660. doi: 10.1016/j.jpainsymman.2012.11.009.

26. United States Department of Agriculture Economic Research Service . South Carolina—Rural Definitions: State-Level Maps. https://www.ers.usda.gov/webdocs/DataFiles/53180/25595_SC.pdf?v=0

27. United States Department of Agriculture Economic Research Service . North Carolina—Rural Definitions: State-Level Maps. https://www.ers.usda.gov/webdocs/DataFiles/53180/25587_NC.pdf?v=7175

28. Trivedi RB, Post EP, Sun H, et al. Prevalence, comorbidity, and prognosis of mental health among US Veterans. Am J Public Health. 2015;105:2564–2569. doi: 10.2105/AJPH.2015.302836.

29. Trajkovic-Vidakovic M, de Graeff A, Voest EE, et al. Symptoms tell it all: A systematic review of the value of symptom assessment to predict survival in advanced cancer patients. Crit Rev Oncol Hematol. 2012;84:130–148. doi: 10.1016/j.critrevonc.2012.02.011.

30. Hui D, Shamieh O, Paiva CE, et al. Minimal clinically important differences in the Edmonton symptom assessment scale in cancer patients: A prospective, multicenter study. Cancer. 2015;121:3027–3035. doi: 10.1002/cncr.29437.

31. Agha Z, Lofgren RP, VanRuiswyk JV, et al. Are patients at Veterans Affairs medical centers sicker? A comparative analysis of health status and medical resource use. Arch Intern Med. 2000;160:3252–3257. doi: 10.1001/archinte.160.21.3252.

32. Dursa EK, Barth SK, Bossarte RM, et al. Demographic, military, and health characteristics of VA health care users and nonusers who served in or during Operation Enduring Freedom or Operation Iraqi Freedom, 2009-2011. Public Health Rep. 2016;131:839–843. doi: 10.1177/0033354916676279.

33. National Cancer Institute Surveillance Epidemiology and End Results Program . Preliminary Cancer Incidence Rates and Trends: Preliminary Estimates for 2017. https://seer.cancer.gov/statistics/preliminary-estimates/preliminary.html

34. Castel LD, Saville BR, Depuy V, et al. Racial differences in pain during 1 year among women with metastatic breast cancer: A hazards analysis of interval-censored data. Cancer. 2008;112:162–170. doi: 10.1002/cncr.23133.

35. Martinez KA, Snyder CF, Malin JL, et al. Is race/ethnicity related to the presence or severity of pain in colorectal and lung cancer? J Pain Symptom Manage. 2014;48:1050–1059. doi: 10.1016/j.jpainsymman.2014.02.005.

36. Nahin RL. Severe pain in Veterans: The effect of age and sex, and comparisons with the general population. J Pain. 2017;18:247–254. doi: 10.1016/j.jpain.2016.10.021.

37. Purnell JQ, Palesh OG, Heckler CE, et al. Racial disparities in traumatic stress in prostate cancer patients: Secondary analysis of a national URCC CCOP study of 317 men. Support Care Cancer. 2011;19:899–907. doi: 10.1007/s00520-010-0880-3.

38. Yee MK, Sereika SM, Bender CM, et al. Symptom incidence, distress, cancer-related distress, and adherence to chemotherapy among African American women with breast cancer. Cancer. 2017;123:2061–2069. doi: 10.1002/cncr.30575.