Using the Computer-based Health Evaluation System (CHES) to Support Self-management of Symptoms and Functional Health: Evaluation of Hematological Patient Use of a Web-Based Patient Portal.

Introduction

Patient-reported outcomes (PRO) are defined as all reports about the health status given directly by the patient without interpretation of the patient’s response by a clinician or anyone else [ 1 ]. While they have long been used in clinical trials, they have, in recent decades, also progressed to enriching routine clinical care [ 2 , 3 ]. Driven by technological progress and an increased availability and use of the internet in the population [ 4 ], it has become easier to incorporate the patient’s perspective into clinical care using electronic patient-reported outcome measures (ePROMs).

In oncological care, PROs can support patient-clinician communication [ 5 , 6 ] and aid early detection of symptoms [ 7 , 8 ] and have been linked to a decrease in hospitalization and emergency department visits [ 9 ]. Building on the evidence base showing the benefits of PRO use at the hospital, the use of web-based solutions to assess ePROMs outside the hospital has gained traction. In the last decade, the number of web-based patient portals that enable the completion of ePROMs has risen and recent research has demonstrated the potential benefits of patient portals in large-scale clinical trials [ 10 - 13 ]. For example, in a randomized controlled trial by Denis et al [ 10 ], web-based symptom monitoring was associated with increased survival compared to standard imaging surveillance following treatment for lung cancer. The authors argue that web-based symptom monitoring may allow for earlier symptom detection and appropriate reaction by health care professionals (HCPs). Moreover, PRO web monitoring can be highly cost-effective [ 11 ] and reduce the administrative burden of assessments inside the hospital, as patients can complete ePROMs from home. Finally, remote assessments are especially helpful in an outpatient setting, as assessments conducted on the day of chemotherapy administration at the hospital have shown to systematically underestimate patients’ symptom burden associated with treatment [ 14 ].

Despite the benefits shown in study settings, and even though detailed guidance on how to implement PROs into clinical practice exists [ 15 , 16 ], electronic PRO monitoring and especially patient portals are still only occasionally adopted in routine clinical practice. This can be attributed to the limited integration into electronic health records, a lack of financial reimbursement for ePROM assessments, and a lack of standardized assessment methods, which hinder implementation in routine care [ 2 ]. Patient portals also vary considerably regarding the focus of the implementation and their goals and use of PROs [ 17 ]. More research is needed that evaluates the usability and acceptability of different applications in routine practice to extend and strengthen the evidence base in this heterogeneous and evolving field of research.

At the Department of Internal Medicine V in Innsbruck, a patient portal for outpatients with multiple myeloma (MM) and chronic lymphocytic leukemia (CLL) was developed to conduct remote PRO assessments. The primary aims for development of the portal were to enhance patient empowerment, encourage patient engagement with PROs, and reduce the administrative burden of PRO assessments inside the hospital.

In this study, we evaluated patient use of the various components of the portal and aimed to identify patient lack of motivation for not using the patient portal and potential barriers to accessibility.

Methods

Study Design

In our observational, longitudinal study, we evaluated patient use of the portal based on two data sources: semistructured interviews conducted during 3 consecutive visits to our unit after introducing eligible patients to the patient portal and system access logs recording the duration of user sessions and how often each page of the web portal was accessed.

Description of the PRO Monitoring and the Patient Portal

In June 2016, we implemented routine ePROM assessments to supplement care of outpatients with MM at the Department of Internal Medicine V in Innsbruck. The two primary aims of the implementation of ePROM assessments were to supplement the Austrian Myeloma Registry (AMR) with PRO data and enrich clinical care with the data [ 18 ]. In July 2017, patients with CLL were added as a second patient group. The use of the Computer-based Health Evaluation System (CHES) [ 19 ] enables immediate processing and graphical representation of the results. For monitoring at the hospital, patients complete PRO measures before their medical appointment, and the results are presented to the HCP prior to the consultation. The implementation and feasibility of the assessments at the hospital and use of data from patients have been evaluated in the past for the AMR [ 18 ], and more details on HCP use of the system are presented elsewhere [ 20 ]. This study builds upon our previous implementation strategy and is focused on process evaluation, refinement, and extension (as described in phase IV of the implementation process described by Sztankay et al [ 18 ]).

There are 3 main components of CHES: (1) the HCP interface (CHES.main), which presents PRO and patient data to HCPs; (2) the survey interface (CHES.nurse), where the patients complete questionnaires at the hospital; and (3) the patient portal (CHES.portal), which allows remote questionnaire completion and access to supplemental information and self-management advice. See Figure 1 for a visualization of the patient portal and its functionality. See Multimedia Appendix 1 for screenshots of the software and portal.

In this paper, we assess patient use of the patient portal, which features the following functionalities:

Disease-specific information on CLL and MM: diagnosis, possible symptoms, possible treatments, and links to further information and self-help groups

PRO assessments with the cancer-specific European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30) in combination with the disease-specific modules EORTC QLQ-MY20 for patients with MM and EORTC QLQ-CLL17 for patients with CLL. The EORTC QLQ-C30 is the most frequently used cancer-specific questionnaire [ 22 ] and can be used to measure patient symptoms, functional health, and global quality of life. The modules QLQ-MY20 and QLQ-CLL17 supplement the QLQ-C30 and cover disease-specific issues of quality of life for MM (eg, future perspective, treatment side effects) and CLL patients (eg, symptom burden due to disease and/or treatment, worries/fears regarding health and functioning).

PRO score review by patient: results are displayed as colored bar charts (longitudinal and cross-sectional). Results that exceed the thresholds for clinical importance [ 21 ] are colored red, and results that do not exceed these threshold are colored green (see Lehmann et al [ 20 ] for more information).

Self-management tools and tailored information: based on EORTC QLQ-C30 data, patients are presented with self-management tools for the symptoms and functional health domains. If a patient reports a potentially clinically important result [ 21 ], they are directed to the self-management tools (see Lehmann et al [ 20 ] for more information). Patients are reminded that in case of severe impairments, they should contact the clinical team directly.

At the outpatient clinic, patients are introduced to the portal by a PRO facilitator (ie, person responsible for the assessments at the hospital). The role of the PRO facilitator is that of a study assistant with a background in psychology or nursing who is trained in the use of PRO data and motivates both patients and HCPs to use the PRO data. Patients are given instructions for use and their personal log-in data for accessing the portal at home by the PRO facilitator.

Typically, patients are advised to complete ePROM assessments in the week before their hospital visits. Visits are scheduled at regular intervals that range between 1 week and 12 months, depending on the disease stage and treatment plan. Patients are encouraged to complete questionnaires as often as they like, even on a daily basis. Questionnaires completed within 7 days of a hospital visit are used to inform the HCP of the patients’ health status and linked to the clinical data in the AMR. Data from assessments in between hospital visits are used for research purposes and enable more continuous tracking of patients’ health status in the AMR.

Although patients participating in the routine monitoring are encouraged to use the patient portal to report PROs, they can still complete assessments at the hospital (eg, if they forgot to use the portal before their hospital visit).

Study Sample

We defined the following inclusion criteria for participation in monitoring via the patient portal: fluency in German, age 18 years or older, diagnosis of MM or CLL, current treatment at the outpatient clinic of the Department of Internal Medicine V or the Comprehensive Cancer Center Innsbruck of the Medical University of Innsbruck, consent given to routine PRO monitoring at the hospital, and completion of at least one ePROM during a prior visit to the hospital.

Patients were deemed ineligible if they had no access to a computer or the internet or lacked sufficient knowledge to log into a website using a username and password. All patients provided informed consent to use the patient portal. Patients who declined to use the portal were asked for the reason for their refusal. The study and use of patient data are covered by the ethics approval for the AMR issued by the ethics committee of the Medical University of Innsbruck (study number AN3252 266/4.2 386/5.14).

At baseline, demographics (sex, age, marital status, education level, and employment status) and experience with the internet and computer technology (frequency, duration) were collected via a questionnaire. Diagnosis and cancer stage were obtained from the hospital’s medical records.

Selection of Outcome Measures

We used different sources to evaluate patient use and perception of the patient portal. Table 1 displays the selection of outcome measures.

Patient Interviews

Each patient participated in 3 semistructured face-to-face interviews conducted by authors LN, PB, and JL following a fixed schedule during 3 consecutive visits to the outpatient unit. See Figure 2 for a flowchart of the interview process. The topics covered in the interviews were as follows:

T1: Guided introduction to the portal and user log-in

Possible difficulties logging in

Navigating the portal

Reviewing PRO results

Other suggestions/remarks by patients

T2: Evaluation of use: acceptability and usability

Satisfaction with instructions for completion of questionnaire

Navigating the portal

Technical difficulties

Reviewing PRO results and accessing self-help tools

Relevance and usefulness of provided content

T3: Evaluation of use: acceptability and usability

Technical difficulties

Interest in content not explored during previous log-ins

Satisfaction with design

Other suggestions/remarks by patients

ePROM Completion Rate

To evaluate initial interest in the portal, we assessed the proportion of patients in routine monitoring at the hospital consenting to use the patient portal at T1. To evaluate continued use, we assessed the proportion of patients who completed ePROMs prior to hospital visits at T2 and T3; ePROMs were considered to be linked to a hospital visit if they were completed in the 7 days before the visit.

CHES Log Data

The frequency (absolute number) and duration (in minutes) of the use of the website were determined by the CHES system log. These data were gathered irrespective of the interview time points each time the patient logged into the system. Log data were collected per session but were not linked to individual patients. Therefore, patients with more sessions have a greater weight in this analysis. Log-ins that occurred on the same day were considered a single session, and durations of these sessions were summed.

Data Analysis

Sociodemographic and clinical data were analyzed at time point T1. Comparison of patients completing ePROMs via the patient portal and of those who declined to use the portal were made with t tests (for parametric variables) and chi-square tests (for nonparametric variables).

The interviewer took field notes during the interviews and for each open question. Responses from the interviews were either analyzed descriptively (for yes/no questions) or paraphrased and category-coded (for open answers) independently by two researchers (JL and PB) and harmonized by discussion in case of different coding. We translated selected quotes into English for the results section of this paper.

We also analyzed the frequency (number) of views for each portal page, excluding views on the start page, where patients are directed automatically after logging in. Further, we calculated the time patients needed to complete the questionnaires online (mean completion time per patient averaged across all patients).

Results

Participant Enrollment and Baseline Characteristics

Recruitment began in July 2017 and was open until August 2020. During the study period, we identified 142 eligible patients already participating in the electronic PRO monitoring in the hospital, of whom 85.9% (122/142) were approached for study participation and use of the patient portal. Of those, 83.6% (102/122) consented to be included in the patient portal. Of 20 patients not willing to use the patient portal, 18 patients stated a preference of questionnaire completion only at the hospital as the reason. The complete enrollment process is shown in Figure 3 .

The full sociodemographic information and clinical data are given in Table 2 . There were no statistically significant differences regarding age, sex, education, time since the initial diagnosis, and type of internet use between those who agreed and those who refused to participate (all P >.09). The age range of patients included in the portal was 39 to 83 years, and the age range of patients not included in the portal was 39 to 77 years. A statistically significant difference was found for general internet use (see Table 2 ), with participants who consented to use the portal reporting higher internet use than those who did not consent to use the portal: of patients who used the portal, 94% (94/100) reported using the internet at least multiple times per week compared to 76% (17/22) for patients who did not use the portal.

Interviews

Interviews lasted between 10 and 30 minutes. Only 5% (5/102) patients required help from the PRO facilitator because they were not able to read their username and password (small font size). After the log-in and during the first interview, no patients reported technical difficulties or had difficulties navigating the portal or the questionnaire. Table 3 shows the number of completed questionnaires before the interviews and the reasons for noncompletion. Of the patients who participated in all 3 interviews, 37% (26/71) completed the questionnaires prior to the T2 and T3 interview, while the others completed the questionnaires only at the hospital visit.

Table 4 shows the use and evaluation of the portal as reported by patients who completed questionnaires prior to the interviews. Patients who completed questionnaires prior to their follow-up appointments reported using the portal in different ways: the percentage of patients reading additional portal content (eg, disease-specific information) declined over time from 71% (27/38) at T2 to 41% (13/32) at T3. The self-management tools were used and deemed useful by 50% (17/34) of patients at the second and 42% (13/31) at the third interview.

Patient Comments

Patients were encouraged to provide additional comments following their answers to the interview questions. In the first interview, patients reported high satisfaction with the presentation of the results as bar charts.

...bar charts are a good way of presenting the results. I like that I can compare my results to those of other patients with cancer.

I like that I can see my results after having completed the questionnaire.

Patients who did not inspect their own results expressed that they were feeling fine and therefore had no interest in viewing their results.

Similarly, patients who were feeling fine or reported no impairments did not inspect the self-management tools, while others provided positive feedback on the self-management tools or the color-coding of questionnaire results.

I did not check the self-management advice because I am not feeling ill. Why would I check it?

I like the design. The arrow in the results [direct link from the results to the self-management information] was very helpful.

...liked the color coding [red/green] as it was simple and easy to understand.

Four patients expressed the wish for a reminder (email or text message) before the next appointment to complete the questionnaires online. Two patients mentioned that they would like to be able to choose their password or username themselves (which the software currently does not allow). Three patients also explicitly reported a decline in motivation toward the end of the study, caused by a lack of sufficient feedback from physicians who did not discuss their PRO results during the consultations.

CHES Log Data

Over the study period, we registered a total of 796 sessions (ie, log-ins by patients). Of the patients who logged into the patient portal, 27% (28/102) logged in once, 18% (18/102) logged in twice, 9% (9/102) logged in 3 times, 15% (16/102) logged in 4 to 6 times, 14% (14/102) logged in 7 to 10 times, and 17% (17/102) logged in more than 10 times (total range of 1-57).

The mean duration of a session was 9.4 (median 6, range 1-90) minutes. It took patients on average 2.9 minutes to complete the EORTC QLQ-C30, 1.8 minutes to complete the EORTC QLQ-MY20, and 1.5 minutes to complete the EORTC QLQ-CLL17. This adds up to an average questionnaire completion time of 4.7 minutes for patients with MM and 4.4 minutes for patients with CLL.

During an average session, patients viewed 5.3 (median 4, range 1-28) different pages. Figure 4 shows how often specific components of the patient portal were viewed. A total of 3487 views were registered. The most frequently viewed self-management pages were those providing information on dyspnea (89 views), diarrhea (80 views), cognitive functioning (64 views), and emotional functioning (49 views). The least frequently viewed self-management pages were those providing information on obstipation (11 views), role functioning (12 views), social functioning (21 views), and pain (18 views). Multimedia Appendix 2 shows the view count on all self-management pages.

Discussion

Principal Findings

In our study, we evaluated the use of the patient portal in patients with MM and CLL participating in routine electronic PRO monitoring. We found that in those patients who were eligible to use the patient portal, the majority (84%) were willing to use the portal. Only a few users reported difficulties logging in, but no users reported problems navigating the survey. However, we found that considerably fewer patients (37%) than initially included adopted a continued use of the portal across all 3 study time points. On average, patients spent 9.4 minutes in the portal per session.

Uptake of Home Monitoring

Our recruitment rate (ie, patients consenting to try the patient portal) is high compared to rates found in other studies. For example, other feasibility or usability studies report that between 21% and 64% [ 23 - 25 ] of patients were willing to use patient portals. Our high recruitment rate is most likely a consequence of the fact that patients were already participating in electronic PRO monitoring and so had already approved some form of monitoring. We would like to note the high inclusion rate for PRO monitoring at the hospital, which was, as reported previously, 94% of all MM patients treated in the department [ 18 ]. Our study emphasizes the possible synergy between assessments inside the hospital and home monitoring; patients who had already participated in electronic PRO monitoring inside the hospital were open to also using a patient portal. Moreover, the high acceptance of the patient portal might have been induced by the opportunity to get accustomed to ePROMs at the hospital with the help of a PRO facilitator before being introduced to the patient portal. We hypothesize that such a stepwise approach can reduce potential reservations about using electronic measures and contribute to user empowerment.

The adoption of home monitoring by a considerable proportion of patients also meant that those patients did not need to be assessed at the hospital. In this way, the patient portal reduced the administrative burden of assessing those patients’ PROs at the hospital and allowed us to allocate those resources elsewhere.

Long-Term Use of the Portal

Over the course of the study, slightly less than half of the participants used the portal prior to their follow-up appointments. The main reason reported by patients was that they had forgotten to report their PRO data using the patient portal. Especially for patients with an infrequent appointment schedule (intervals between hospital visits of up to 12 months), forgetting to use the portal is, in fact, not surprising. This issue might be addressed by implementing an automated reminder (email or text message) as has been done with similar home monitoring systems [ 24 , 26 ]. For example, the AmbuFlex system in Denmark has, in the past, used different forms of reminders, including letters, emails, and text messages. They ultimately implemented communication via a national secure email program, which accounts for 93.2% of communication to patients and secures high completion rates [ 27 ].

Additionally, it is important to note that due to our routine monitoring approach, ePRO data from patients who did not use the portal prior to the follow-up appointments were not lost. Instead, those patients were invited to complete the ePROMs at the hospital as was done with other patients who did not use the portal.

While only a few of our patients directly mentioned a decline in motivation to complete ePROMs due to HCPs not picking up on the results during the consultations, this is in fact a frequent problem of PRO implementations in clinical settings [ 3 , 26 , 28 , 29 ]. For patients, there is little perceived benefit of completing the questionnaires if the results are not reviewed and discussed by HCPs. Instead, sharing their health status via questionnaires might even become burdensome. Therefore, it is important to engage HCPs with the concept of PROs and train them in the use of PRO results to prevent PRO data from becoming meaningless busywork that hinders clinical practice instead of enhancing it [ 28 ]. This requires the education and training of HCPs, which can be achieved by conducting specialized training programs [ 30 ]. A buy-in strategy may be used to increase HCP engagement with PROs, for example, via the adoption of the user interface to HCP needs and preferences, regular meetings, or analyses of PRO data in the registry upon request of HCPs. Another important approach is to make PRO data comprehensible and actionable for HCPs (eg, by providing advice on how to react to results) [ 26 , 31 , 32 ]. In our system, this is done, for example, by using thresholds for clinical importance for the EORTC QLQ-C30 [ 21 ] that facilitate the interpretation of patient PRO data.

Finally, the literature shows that in order to provide a sound theoretical basis for sustainable PRO solutions in routine care settings, an implementation science approach can be followed [ 33 ]. Implementation science can help identify barriers (which are often similar across contexts) and enablers (which often depend on the given hospitals’ context) of patient-reported outcome measure (PROM) implementations.

User Patterns in the Portal

As has been found in other studies [ 13 , 17 , 25 ], the majority of our patients found the display of self-management advice linked with their results to be valuable. A recent review of electronic systems to measure PROs found that less than one-third (29%) of published electronic systems include features that provide tailored automated advice to patients, and less than half (41%) provide general educational information [ 17 ]. In a qualitative study evaluating another eHealth application, tailored feedback and advice was rated as appealing by most participants [ 34 ]. Participants valued the option of accessing information remotely between appointments and having a low threshold to receiving such information (compared to having to consult their treating physician or having to search for information) [ 34 ]. These are notions that echo the high number of views of pages with self-management advice we found in our study.

Nevertheless, despite the approval for self-management advice we found in our study, a recent clinical trial showed that, while patient portals can contribute to improved health-related quality of life [ 13 ], their measurable benefits on patient activation (knowledge, skills, and confidence for self-management) could not be shown in a diverse sample of cancer survivors [ 13 ]. However, previous evidence suggested that such effects may be more pronounced in newly diagnosed patients [ 25 , 35 ]. Another randomized controlled trial showed that weekly PRO monitoring along with the provision of tailored self-management advice, compared to usual care, significantly enhanced self-efficacy in patients with cancer [ 36 ].

Limitations

Our study was designed to be carried out during routine care with minimal interruption of the clinical workflow. Therefore, interview time points were integrated into the patients’ hospital visits. This resulted in varying intervals between interviews (between 1 week and 1 year in a few cases), which may have influenced the results. However, we point out that these are real-world visitation schedules, and any application designed for routine care should be evaluated accordingly. Similarly, we included more males than females in our study. This reflects the epidemiology of MM and CLL, and the male/female ratio was comparable to data from the AMR.

Second, the CHES log data did not allow for detailed analysis of individual user page-view patterns but only for analysis of overall page views and duration and frequency of sessions. This means that patients logging on to the portal more frequently have a greater weight in the analysis of page views. Moreover, page view numbers have to be interpreted with care as they provide no information on whether the pages were actually read.

Another limitation is that the interviews were conducted by the authors (LN, PB, and JL), who introduced patients to the portal. Therefore, some patients might have been reluctant to voice criticism in the interviews, even though we actively encouraged patients to also report negative feedback.

Finally, we consider some sample-related limitations: We included only German-speaking patients in the patient portal. While the EORTC questionnaires are available in a large number of languages (and can be completed at the hospital [ 20 ]), translating and updating the content of the portal would require considerable resources. This results in a potential bias, and our findings may not be generalizable to patients with other first languages and limited German language proficiency. We are aware that this can create an imbalance in the provision of care, as this systematically excludes certain patient groups. In fact, patients not speaking the primary language of the country might profit most from receiving disease-specific information and from being able to report symptoms online in their native language. Moreover, we observed a selection bias due to patients’ age (patients with MM were on average 11 years younger than the mean age of patients in the AMR), as older patients might be less proficient using the internet and were consequently not included in our study. While this is an important limitation to consider, its impact should decrease over time, as the population’s internet capabilities have been steadily increasing [ 37 ] and should further increase in the years to come.

Future Steps

Following the insights gained in this study, we are currently in the process of updating our monitoring procedure and software. One important step is to more closely monitor patients’ PROMs to swiftly identify deteriorations and act accordingly; we aim to have a trained onconurse monitor patient results and check on patients in case of deteriorations. If necessary, the nurse can schedule an earlier appointment or alert the treating HCP. Another planned improvement is to implement an email or text message reminder to improve PROM completion rates.

Conclusion

Our study shows that a patient portal enabling remote PRO data assessment can complement routine electronic PRO implementation at the hospital by reducing the burden of administration for the clinical team and offering an additional way for patients to engage with PROs. We found that the majority of patients were open to using the patient portal and interested in assessments from home. The low number of technical problems and absence of complaints demonstrate the general user-friendliness of our portal. While initial uptake was high, fewer patients adopted regular use of the portal prior to their appointments. To increase long-term participation rates, further motivational (eg, increasing HCP engagement with the PRO data) and technical (eg, email reminder) measures are needed.

We thank the patients who participated in this study for their valuable time and input, without which the study would not have been possible. We thank the Department of Internal Medicine V in Innsbruck and the Comprehensive Cancer Center Innsbruck for their support with the project. We thank Janny Sauter for her help in the collection of data.

Authors' Contributions: JL and PB wrote and edited the manuscript and collected and analyzed the data. BH, GR, JMG, LW, MS, RW, WWeyrer, and WWillenbacher conceptualized the study, supported the implementation of the patient portal, and edited the manuscript. LN conceptualized the study, supported the implementation of the patient portal, assisted in data collection, and edited the manuscript. All authors read and approved the final version of the manuscript.

Conflicts of Interest: BH and GR have intellectual property rights to the software tool CHES. All remaining authors declare that they have no conflict of interest.

Appendix

Examples of pages in the patient portal.

Total view count in the portal and specific view count for the self-management pages.

Abbreviations

Austrian Myeloma Registry

Computer-Based health Evaluation System

chronic lymphocytic leukemia

European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire

electronic patient-reported outcome measure

health care professional

multiple myeloma

patient-reported outcome

patient-reported outcome measure

1 Patient-reported outcome measures: use in medical product development to support labeling claims US Food and Drug Administration 2009 12 2020-08-05 https://www.fda.gov/media/77832/download

2 Basch E Patient-reported outcomes: harnessing patients' voices to improve clinical care N Engl J Med 2017 1 12 376 2 105 108 10.1056/NEJMp1611252 28076708 28076708

3 Jensen RE Snyder CF Abernethy AP Basch E Potosky AL Roberts AC Loeffler DR Reeve BB Review of electronic patient-reported outcomes systems used in cancer clinical care J Oncol Pract 2014 7 10 4 e215 e222 10.1200/jop.2013.001067 24301843 PMC4094646

4 Digital Economy and Society Index (DESI) 2020: use of internet services European Commission 2020-09-03 https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=67075

5 Detmar SB Muller MJ Schornagel JH Wever LDV Aaronson NK Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial JAMA 2002 12 18 288 23 3027 3034 10.1001/jama.288.23.3027 12479768 12479768

6 Velikova G Booth L Smith AB Brown PM Lynch P Brown JM Selby PJ Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial J Clin Oncol 2004 2 15 22 4 714 724 10.1200/jco.2004.06.078 14966096

7 Basch E Abernethy AP Supporting clinical practice decisions with real-time patient-reported outcomes J Clin Oncol 2011 3 10 29 8 954 956 10.1200/JCO.2010.33.2668 21282536 21282536

8 Takeuchi EE Keding A Awad N Hofmann U Campbell LJ Selby PJ Brown JM Velikova G Impact of patient-reported outcomes in oncology: a longitudinal analysis of patient-physician communication J Clin Oncol 2011 7 20 29 21 2910 2917 10.1200/jco.2010.32.2453 21690465

9 Barbera L Sutradhar R Seow H Earle C Howell D Mittmann N Li Q Thiruchelvam D Impact of standardized edmonton symptom assessment system use on emergency department visits and hospitalization: results of a population-based retrospective matched cohort analysis J Oncol Pract 2020 9 16 9 e958 e965 10.1200/jop.19.00660 32463762

10 Denis F Basch E Septans A Bennouna J Urban T Dueck AC Letellier C Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer JAMA 2019 1 22 321 3 306 307 10.1001/jama.2018.18085 30667494 30667494 PMC6439676

11 Lizée T Basch E Trémolières P Voog E Domont J Peyraga G Urban T Bennouna J Septans A Balavoine M Detournay B Denis F Cost-effectiveness of web-based patient-reported outcome surveillance in patients with lung cancer J Thorac Oncol 2019 6 14 6 1012 1020 10.1016/j.jtho.2019.02.005 30776447 30776447

12 Basch E Deal AM Dueck AC Scher HI Kris MG Hudis C Schrag D Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment JAMA 2017 12 11 318 2 197 198 10.1001/jama.2017.7156 28586821 28586821 PMC5817466

13 van der Hout A van Uden-Kraan CF Holtmaat K Jansen F Lissenberg-Witte BI Nieuwenhuijzen GAP Hardillo JA Baatenburg de Jong RJ Tiren-Verbeet NL Sommeijer DW de Heer K Schaar CG Sedee RE Bosscha K van den Brekel MWM Petersen JF Westerman M Honings J Takes RP Houtenbos I van den Broek WT de Bree R Jansen P Eerenstein SEJ Leemans CR Zijlstra JM Cuijpers P van de Poll-Franse LV Verdonck-de Leeuw IM Role of eHealth application Oncokompas in supporting self-management of symptoms and health-related quality of life in cancer survivors: a randomised, controlled trial Lancet Oncol 2020 1 21 1 80 94 10.1016/S1470-2045(19)30675-8 31838009 31838009

14 Giesinger JM Wintner LM Zabernigg A Gamper E Oberguggenberger AS Sztankay MJ Kemmler G Holzner B Assessing quality of life on the day of chemotherapy administration underestimates patients' true symptom burden BMC Cancer 2014 10 10 14 758 10.1186/1471-2407-14-758 25305067 25305067 PMC4198707

15 Wintner LM Sztankay M Aaronson N Bottomley A Giesinger JM Groenvold M Petersen MA van de Poll-Franse L Velikova G Verdonck-de Leeuw I Holzner B EORTC Quality of Life Group The use of EORTC measures in daily clinical practice: a synopsis of a newly developed manual Eur J Cancer 2016 11 68 73 81 10.1016/j.ejca.2016.08.024 27721057 27721057

16 LeRouge C Austin E Lee J Segal C Sangameswaran S Hartzler A Lober B Lavallee D ePROs in Clinical Care: Guidelines and Tools for Health Systems 2020 Seattle CERTAIN, University of Washington

17 Warrington L Absolom K Conner M Kellar I Clayton B Ayres M Velikova G Electronic systems for patients to report and manage side effects of cancer treatment: systematic review J Med Internet Res 2019 1 24 21 1 e10875 10.2196/10875 30679145 30679145 PMC6365878

18 Sztankay M Neppl L Wintner LM Loth FL Willenbacher W Weger R Weyrer W Steurer M Rumpold G Holzner B Complementing clinical cancer registry data with patient reported outcomes: a feasibility study on routine electronic patient-reported outcome assessment for the Austrian Myelome Registry Eur J Cancer Care (Engl) 2019 11 28 6 e13154 10.1111/ecc.13154 31465136 31465136 PMC6900154

19 Holzner B Giesinger JM Pinggera J Zugal S Schöpf F Oberguggenberger AS Gamper EM Zabernigg A Weber B Rumpold G The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring BMC Med Inform Decis Mak 2012 12 126 10.1186/1472-6947-12-126 23140270 23140270 PMC3529695

20 Lehmann J Wintner LM Sztankay M Willenbacher W Weger R Weyrer W Rumpold G Holzner B Patient-reported outcomes and psycho-oncological screening in hematology: a practical example of routine electronic monitoring Mag Eur Med Oncol 2020 7 03 13 3 285 293 10.1007/s12254-020-00628-7

21 Giesinger JM Loth FLC Aaronson NK Arraras JI Caocci G Efficace F Groenvold M van Leeuwen M Petersen MA Ramage J Tomaszewski KA Young T Holzner B EORTC Quality of Life Group Thresholds for clinical importance were established to improve interpretation of the EORTC QLQ-C30 in clinical practice and research J Clin Epidemiol 2020 2 118 1 8 10.1016/j.jclinepi.2019.10.003 31639445 31639445

22 Efficace F Collins GS Cottone F Giesinger JM Sommer K Anota A Schlussel MM Fazi P Vignetti M Patient-reported outcomes as independent prognostic factors for survival in oncology: systematic review and meta-analysis Value Health 2021 2 24 2 250 267 10.1016/j.jval.2020.10.017 33518032 33518032

23 Kuijpers W Groen WG Oldenburg HS Wouters MW Aaronson NK van Harten WH eHealth for breast cancer survivors: use, feasibility and impact of an interactive portal JMIR Cancer 2016 5 10 2 1 e3 10.2196/cancer.5456 28410178 28410178 PMC5369634

24 Girgis A Durcinoska I Levesque JV Gerges M Sandell T Arnold A Delaney GP PROMPT-Care Program Group eHealth system for collecting and utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) among cancer patients: mixed methods approach to evaluate feasibility and acceptability J Med Internet Res 2017 12 02 19 10 e330 10.2196/jmir.8360 28970188 28970188 PMC5667931

25 Duman-Lubberding S van Uden-Kraan CF Jansen F Witte BI van der Velden LA Lacko M Cuijpers P Leemans CR Verdonck-de Leeuw IM Feasibility of an eHealth application "OncoKompas" to improve personalized survivorship cancer care Support Care Cancer 2016 5 24 5 2163 2171 10.1007/s00520-015-3004-2 26563178 26563178 PMC4805720

26 Wu AW White SM Blackford AL Wolff AC Carducci MA Herman JM Snyder CF PatientViewpoint Scientific Advisory Board Improving an electronic system for measuring PROs in routine oncology practice J Cancer Surviv 2016 6 10 3 573 582 10.1007/s11764-015-0503-6 26644190 26644190 PMC4864116

27 Hjollund NHI Fifteen years' use of patient-reported outcome measures at the group and patient levels: trend analysis J Med Internet Res 2019 9 30 21 9 e15856 10.2196/15856 31573943 31573943 PMC6792031

28 Thestrup Hansen S Kjerholt M Friis Christensen S Hølge-Hazelton B Brodersen J Haematologists' experiences implementing patient reported outcome measures (PROMs) in an outpatient clinic: a qualitative study for applied practice J Patient Rep Outcomes 2019 12 28 3 1 74 10.1186/s41687-019-0166-6 31884569 31884569 PMC6935381

29 Taarnhøj GA Lindberg H Dohn LH Omland LH Hjøllund NH Johansen C Pappot H Electronic reporting of patient-reported outcomes in a fragile and comorbid population during cancer therapy: a feasibility study Health Qual Life Outcomes 2020 7 11 18 1 225 10.1186/s12955-020-01480-3 32653005 32653005 PMC7353726

30 Santana MJ Haverman L Absolom K Takeuchi E Feeny D Grootenhuis M Velikova G Training clinicians in how to use patient-reported outcome measures in routine clinical practice Qual Life Res 2015 1 15 24 7 1707 1718 10.1007/s11136-014-0903-5 25589231

31 Snyder CF Blackford AL Wolff AC Carducci MA Herman JM Wu AW PatientViewpoint Scientific Advisory Board Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice Psychooncology 2013 4 22 4 895 901 10.1002/pon.3087 22544513 22544513 PMC3415606

32 Hughes EF Wu AW Carducci MA Snyder CF What can I do? Recommendations for responding to issues identified by patient-reported outcomes assessments used in clinical practice J Support Oncol 2012 10 4 143 148 10.1016/j.suponc.2012.02.002 22609239 22609239 PMC3384764

33 Stover AM Haverman L van Oers HA Greenhalgh J Potter CM ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings Qual Life Res 2020 7 10 1 10.1007/s11136-020-02564-9 32651805 PMC8528754 32651805

34 Lubberding S van Uden-Kraan CF Te Velde EA Cuijpers P Leemans CR Verdonck-de Leeuw IM Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors J Clin Nurs 2015 5 24 9-10 1367 1379 10.1111/jocn.12753 25677218 25677218

35 Girgis A The role of self-management in cancer survivorship care Lancet Oncol 2020 1 21 1 8 9 10.1016/s1470-2045(19)30715-6 31838013

36 Absolom K Warrington L Hudson E Hewison J Morris C Holch P Carter R Gibson A Holmes M Clayton B Rogers Z McParland L Conner M Glidewell L Woroncow B Dawkins B Dickinson S Hulme C Brown J Velikova G Phase III randomized controlled trial of eRAPID: eHealth intervention during chemotherapy J Clin Oncol 2021 3 01 39 7 734 747 10.1200/JCO.20.02015 33417506 33417506

37 Austrian Internet Monitor 2021-02-08 https://de.statista.com/