Usability testing of an electronic patient-reported outcome system linked to an electronic chemotherapy prescribing and patient management system for patients with cancer.

Introduction

In 2020, there were around 20 million new cancer cases worldwide and it is estimated that this number will rise to over 27 million by 2040 [ 1 ]. Similarly, cancer incidence has increased in the United Kingdom (UK) by 19% in the last decade and will continue to do so [ 2 ]. Each year, there are around 450,000 new cancer cases (605 per 100,000) [ 3 ]. It is estimated that, in the UK, around three million people are currently living with cancer [ 4 ]. These patients may experience a wide range of disease and treatment symptoms and treatment-related toxicities that greatly affect their functioning, heath-related quality of life (HRQoL) and their impact on healthcare resource use [ 5 , 6 ]. Patients’ journeys through cancer treatment are increasingly captured with electronic systems, such as ChemoCare® which is in use in several National Health Service (NHS) trusts in the UK [ 7 ].

ChemoCare® is an electronic chemotherapy prescribing and patient management system. The electronic platform was designed by CIS Oncology ( https://www.cis-healthcare.com/ ), an ISO certified med tech company based in the UK (ISO 13485:2016, EN ISO 13485:2016, ISO 9001:2015, ISO 27001:2013). ChemoCare® is used for prescribing, preparing and administering chemotherapy, as well as scheduling treatment and sharing data. It allows instant access to data by the clinical team and patients, ensuring accurate and fully informed prescribing at all times. However, to date, the system does not include any patient reported outcomes (PROs), measures of health status coming directly from the patient without interpretation by a clinician or anyone else [ 8 ]. Clinical teams and patients are keen to see PROs included in the electronic platform to assess the impact of disease and treatment on patients’ symptoms and quality of life from a patient perspective. Electronic capture of PROs (ePROs), using online validated questionnaires, has important advantages over paper forms [ [9] , [10] , [11] ]. The collection of ePROs removes the need for manual data entry which carries the risk of human error and facilitates prompt review by the clinical team to tailor care to individual patient needs [ 10 , 11 ].

PRO assessment in an oncology setting offers a range of potential benefits for individual patient care and for clinicians, regulators, healthcare management teams, commissioners, and policy makers. Benefits include improved symptom management and symptom control and enhanced patient-clinician communication, patient satisfaction and well-being [ 9 , 12 ]. In addition, in the US the use of PROs to monitor patient symptoms in routine cancer care has been shown to lead to clinical benefits include fewer A&E visits, hospitalisations and an increase in survival [ 9 ].

ChemoPRO®

ChemoPRO® is an app linked to the ChemoCare® platform described above. It was designed by CIS Oncology, with ongoing input from clinical stakeholders. The ChemoPRO® app helps hospitals and other stakeholders maintain contact with their patients in between visits. Patients can record symptoms and wellness which can be accessed by the healthcare team. It also allows patients to access a range of information, such as appointments and medication reminders.

The ChemoPRO® app consists of several modules: ‘Appointments’, ‘How are you?‘, ‘My symptoms’, ‘Logs’, ‘Info’, and ‘Help’. The ‘How are you?’ and ‘My symptoms’ modules refer to two PROMs (EQ5D-5L and PRO-CTCAE™, as outlined in methods), as shown in Picture 1 .

CIS Oncology are GDPR and ISO 27001 compliant to ensure that all patient data is held securely in line with current best practice and legislation.

Usability testing

It is vital that the user-friendliness and usability of ePRO systems are assessed and improved prior to implementation in a clinical setting to reduce attrition rates and to ensure that the system performs efficiently and effectively [ 13 ]. Usability testing is one of the stages of system development ( Fig. 1 ).

Usability testing is a technique that involves the testing of systems, products or websites with participants drawn from the target population. It aims to assess how effective and efficient systems are, as well as users’ satisfaction with the system [ 14 ]. It can be conducted during a single session or in a more iterative manner during several sessions [ 15 ]. In our case, it allows end users to actually test ePRO systems and can assist ePRO developers in the evaluation of ePRO user interface [ 15 ].

Therefore, the aim of this study was to assess the usability of the CIS Oncology ChemoPRO® platform with individuals with lived experience of cancer.

Methods

Ethical considerations

This usability study was approved by the University of Birmingham Research Ethics committee (ERN_22-0510).

Study participants and recruitment

People with lived experience of any type of cancer were recruited by sharing the study advert on social media (Twitter) and with the National Cancer Research Institute (NCRI) Consumer Forum and a Patient and Public Involvement (PPI) panel based in Birmingham in November 2022. Further inclusion criteria can be found in Box 1.

Box 1 – list of inclusion criteria

Previous research informed our sample size [ [15] , [16] , [17] , [18] , [19] ]. It has been shown that 8-10 participants, and sometimes as low as 5 if planning to do a series of testing sessions [ 18 ], are required to detect over 80% of issues [ 17 , 19 ], as shown in Fig. 2 .

By the time we interviewed the last participant, it was apparent that we had reached saturation, as no new issue was being mentioned.

People interested in taking part in the study contacted the researcher (CM) who emailed them to confirm their eligibility and sent them a participant information sheet and a consent form to consent to participation in the study and publication of the results. A mutually suitable date and time to conduct the usability testing session was then arranged, after obtaining written informed consent. In total, nine people with cancer were recruited.

People who completed the study were given a £10 voucher in return for their participation in the study.

Clinicians were not included in the current usability study as they have been heavily involved in the planning and design of the app.

ChemoPRO® registration process

Once participants gave their informed consent, CIS Oncology emailed them a link to download and register onto the ChemoPRO® app. This invitation email included a link to GooglePlay/App Store and a numerical code to enter after downloading the app. Participants then downloaded the app and registered ( Fig. 3 ).

PROs

EQ-5d-5L

The Euroqol EQ-5D-5L [ 20 ] comprises 5 items plus 1 visual analogue scale. It has been widely validated across a range of conditions and used to assess health outcome from a wide variety of interventions on a common scale, for purposes of evaluation, allocation and monitoring [ 21 , 22 ]. It is used by the National Institute for Health and Care Excellence (NICE) in health technology assessment. EQ-5D-5L, takes only a few minutes to complete and has a same day recall period.

PRO-CTCAE™

The Common Terminology Criteria for Adverse Events (CTCAE) is routinely used in oncology as the means of categorising adverse events in trials and is a descriptive terminology that evolved alongside other toxicity grading systems to become a comprehensive, multi-modality grading system for reporting the acute and late effects of cancer treatment [ 21 , 22 ]. The PRO-CTCAE™ was developed to characterise the frequency, severity and interference of 78 symptomatic treatment toxicities that could be meaningfully reported from the patient perspective. It has been established that symptomatic adverse events are under-detected by clinicians versus patients, who are better at identifying low-grade symptomatic events [ 23 , 24 ]. As such, the PRO-CTCAE™ could generate information that complements CTCAE data and can provide a more holistic insight into patient experience on treatment, which is integral for clinical decision-making. In oncology, the PRO-CTCAE™ has demonstrated favourable validity, reliability, and responsiveness [ 25 ].

Testing sessions

The one-to-one testing sessions consisted of performing a series of tasks on the app whilst talking to the researcher about their experience (‘Think Aloud’ technique) [ 26 ]. The list of tasks performed by the participants can be found in Box 2. Although one of these tasks was to complete the two PROMs included, we did not use the responses to the questionnaires to assess the usability of the app. This will be part of the next stage of the evaluation, the feasibility study. Effectiveness was assessed by recording the type and number of errors. Critical errors were errors that prevented participants from using the app and completing the tasks unless the researcher stepped in and advised participants on how to resolve the issue. Conversely, non-critical errors were minor errors that did not prevent participants from completing the testing session. Efficiency was assessed by providing an estimation of the time it took participants to complete the EQ5D5L PROM. Participants also identified areas of improvement and made suggestions to improve the app and the users' experience. This was followed by a short qualitative interview to capture their overall experience of using the app (retrospective probing) [ 27 ]. Finally, participants were asked to complete a short demographics questionnaire and a brief satisfaction questionnaire previously used by Aiyegbusi et al. [ 16 ] The satisfaction questionnaire included four questions to rate the usability of the system and participants' satisfaction on a 5-point scale (1 = very dissatisfied, 5 = very satisfied) [ 16 ].

Box 2 – List of tasks performed by participants.

Data analysis

Participant characteristics were summarised as n (%). Comments from all the participants were recorded in a table alongside observations from the researcher, whether participants completed the tasks independently or with support, how they found the user experience, and what could be done to improve the ChemoPRO® app. Participants’ comments during the interviews were extracted as quotes and categorised under “Overall comments and impressions” ( Table 5 ). Participant ratings for the four usability questions were used to calculate a mean score per question.

Results

Participant characteristics

Ten people with lived experience of cancer took part in this usability study. This included six women and four men. The mean age of the participants was 62.3 years (range = 44–76 years). Table 1 presents the participants’ characteristics.

Assessment of effectiveness

There were three critical errors and 21 non-critical errors ( Table 2 ). The critical errors related to the email invitation to download the app which were either not received by participants (n = 2) or were illegible (n = 1). In the case of the illegible email, the researcher advised the participant to download the app directly from GooglePlay/the App store and use the login code which was in the invitation email (and easily readable). The researcher became aware that two participants who did not receive the invitation email when she checked with the participants the day before the interview. CM notified CIS Oncology staff, who sent another invitation email.

The 21 non-critical errors included missing information (under Appointments and About my Hospital), not knowing how save logs, and not knowing how to change password when invited to do so.

Assessment of efficiency

Although it was difficult to assess the time it took participants to complete each PROMs (especially PRO-CTCAE™) as they stopped to talk to the researcher during the process, it is estimated it took participants between 1 and 2 min to complete the EQ5D5L.

Suggestions for improvement

Participants identified potential areas to improve the user experience of ChemoPRO® ( Table 3 ).

These issues included (i) the app content not always fitting on a small phone screen and the participants not being able to read everything, (ii) lack of clarity, (iii) removing unnecessary items to avoid overburdening the users while at the same time giving them enough information (about their upcoming appointments, for example), (iv) extending communication with clinical team, and (v) providing additional ways to view their previous symptoms.

Assessment of satisfaction and opinion of the ChemoPRO® app

Table 4 presents participants’ rating of the usability and their satisfaction with ChemoPRO®. The mean score for individual question was high and the overall average score for usability and satisfaction was 4.5 (5-point scale).

Participants overall comments and impressions of ChemoPRO® app and researcher's observations

Overall, all participants were very positive about using the ChemoPRO® app. They found the app easy to use and navigate. They liked the colour scheme, the layout and found the buttons responsive and had no issue around online safety. Participants believed ChemoPRO® had potential for improving communication with clinicians and keeping a record for patients and clinicians of their symptoms/quality of life data over time. They were particularly impressed with the Appointments section, which they thought would help them remember when and where their appointments are as well as what their appointment was about. They liked the ability to view their previous answers to the questionnaires and being able to see if their symptoms/quality of life improved over time, or not. Participants completed and submitted the questionnaires without any issue.

Patients' overall comments and impressions of the app, as well as the researchers’ observations can be found in Table 5 .

Discussion

This article presents the results of the usability testing of the ChemoPRO® app used by people with lived experience of cancer. In addition to two PRO measures (EQ-5D-5L and PRO-CTCAE™), ChemoPRO® included several features designed to facilitate management of the users’ condition. Participants found the app easy to use and navigate and completed the questionnaires with ease, despite encountering a few issues. The errors found were mainly due to programming issues with the app, demonstrating the value of usability testing. Participants believed the app had potential to facilitate communication with their clinical team and suggested areas for further refinement.

This study adds to the existing body of research focusing on testing the usability of a new electronic applications. Usability testing is a key factor for successful implementation [ 16 ], however, this is a stage that is often overlooked [ 28 ]. All participants used their own “Bring Your Own Device” (BYOD). The app was used successfully across all age groups. Although two participants (both aged over 60 years old) said they did not feel very confident using technology, both completed the usability testing unaided without any major issues [ 16 , 29 ]. This may be due to the fact that more older people are using smartphones on a regular basis and that smartphones facilitate digital communication [ 30 ].

Although it was difficult to time how long completing the questionnaires took, as the participants stopped often to talk to the researcher, it became clear that some of the participants were starting feel ePRO fatigue (shown by lack of concentration or forgetfulness). This might cause users to disengage from using the app. Although this is an important aspect to consider, previous work has shown that cancer patients have successfully completed all items of the PRO-CTCAE™ [ 31 ].

One of the ways we recruited participants was through social media. Although recruiting participant on social media has advantages, such as being timesaving and inexpensive, it can pose some problems, such as having less control over who joins the study (in the case of passive recruitment). In our case, several people who asked to join the study (n = 6) were not included in the study as the researcher (CM) had strong doubts about their eligibility and motivation for joining the study. The email messages sent to express their interest in the study were worded exactly the same, were not signed, and contained the same grammar mistakes. The first two of these people who wanted to join the study never replied to CM's request to set a date/time for the testing session. Their motivation may have been the voucher offered to participants, which, although was not included on the study ad posted on social media, was included in the participant information sheet. As a result, a lot of time was spent to identify the potentially ‘fake’ participants, which defeats the purpose of using social media to recruit research participants.

Implications for ePROMs developers

Some participants thought that some of the features of the app would benefit from being removed in order to make the app as simple as possible and avoid overburdening users. Therefore, ePRO programmers need to ensure that its use is as simple as possible for participants.

Participants liked the fact that they were able to view their previous answers to the questionnaires. However, most of them thought that presenting previous answers in a visual way (i.e., line graph) rather than in a list form (as it currently is the case) would help them understand how their symptoms/quality of life have changed overtime. It is recommended that developers follow PROTEUS Consortium's existing guidance on presentation of PRO data [ 32 , 33 ].

Strengths and limitations

The main strength of our study is its methodology which includes a ‘think aloud’ technique which provides real-time feedback.

We are aware that our study has limitations. The sample size limited the diversity within our participant sample. Although we actively tried to recruit a diverse sample and participants in our usability study were all highly educated. Despite reaching out to a number of PPI groups and charities, including charities working specifically with ethnic minority groups, three people from ethnic minorities declined taking part in our study. One explanation for having difficulties finding participants from ethnic minorities is the lower cancer incident rate in the Asian and Black ethnic groups, and in people of mixed or multiple ethnicity, compared with the White ethnic group, in England [ 2 ]. This will be addressed in the next stage of the study which will be a feasibility study. The feasibility study will also include a larger sample and will test the electronic platform in a clinical setting on a long-term basis.

It was difficult to assess efficiency as participants stopped to talk to the researcher while completing the questionnaires, especially PRO-CTCAE™. Although it was not possible to provide an estimate of time it took participants to complete the process, prior work with PRO-CTCAE™ suggest that on average it takes users 11.1 s to answer one question (sd = 8.4) [ 34 ].

Conclusion

Our study showed people with lived experienced of cancer can easily report their symptoms/quality of life through the use of an electronic system. One of the key features of this particular ePRO system that should be developed further is the app's ability to facilitate patient-clinician communication. Future testing should include testing in a clinical setting and testing with people from ethnic minorities and people with a lower education level.

Author contribution statement

Christel McMullan: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper. Sarah E Hughes; Olalekan Lee Aiyegbusi; Melanie Calvert: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This study was funded by CIS Oncology.

Ethics statement

This usability study was approved by the University of Birmingham Research Ethics committee (ERN_22-0510).

Data availability statement

Data will be made available on request.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

1. Sung H., et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA A Cancer J. Clin. 2021;71(3):209–249. doi: 10.3322/caac.21660.

2. UK, C.R. Cancer Statistics for the UK. Available from: https://www.cancerresearchuk.org/health-professional/cancer-statistics-for-the-uk#heading-Zero .

3. UK, C.R. Cancer incidence for all cancers combined. 06/12/22]; Available from: https://www.cancerresearchuk.org/health-professional/cancer-statistics/incidence/all-cancers-combined#heading-Zero .

4. Support, M.C. Prevalence by cancer type, nation, sex and year. 2020; Available from: https://www.macmillan.org.uk/dfsmedia/1a6f23537f7f4519bb0cf14c45b2a629/5192-10061/macmillan-2020-cancer-prevalence-figures-and-methodology .

5. Basch E., et al. Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: findings from the PRO-TECT trial. JCO Clin. Cancer Inform. 2020;4:947–957. doi: 10.1200/CCI.20.00081.

6. Henry D.H., et al. Symptoms and treatment burden associated with cancer treatment: results from a cross-sectional national survey in the U.S. Support. Care Cancer. 2008;16(7):791–801. doi: 10.1007/s00520-007-0380-2.

7. McDonald L., Stein D., Carroll R., Soni M., Schultze A., Ramagopalan S., Wagstaff J. Extracting data from a chemotherapy prescription platform for real-world oncology research in the UK: a pilot study. Future Oncol. 2019;15(21):2471–2477. doi: 10.2217/fon-2019-0150.

8. FDA . 2009. Guidance for Industry - Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. https://www.fda.gov/media/77832/download Available from:

9. Basch E., et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J. Clin. Oncol. 2016;34(6):557–565. doi: 10.1200/JCO.2015.63.0830.

10. Muehlhausen W., et al. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual. Life Outcome. 2015;13:167. doi: 10.1186/s12955-015-0362-x.

11. McMullan C., et al. Care providers' and patients' attitudes toward using electronic-patient reported outcomes to support patients with traumatic brain injury: a qualitative study (PRiORiTy) Brain Inj. 2020;34(6):723–731. doi: 10.1080/02699052.2020.1740944.

12. Gilbert J.E., et al. Quality improvement in cancer symptom assessment and control: the provincial palliative care integration project (PPCIP) J. Pain Symptom Manag. 2012;43(4):663–678. doi: 10.1016/j.jpainsymman.2011.04.028.

13. Maramba I., Chatterjee A., Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int. J. Med. Inf. 2019;126:95–104. doi: 10.1016/j.ijmedinf.2019.03.018.

14. Standardization I.O.f. 2018. Ergonomics of Human–System Interaction—Part 11: Usability: Definitions and Concepts.

15. Aiyegbusi O.L. Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems. Qual. Life Res. 2020;29(2):325–333. doi: 10.1007/s11136-019-02329-z.

16. Aiyegbusi O.L., et al. Development and usability testing of an electronic patient-reported outcome measure (ePROM) system for patients with advanced chronic kidney disease. Comput. Biol. Med. 2018;101:120–127. doi: 10.1016/j.compbiomed.2018.08.012.

17. B, B. Determining the correct number of usability test participants: usability.gov. 2006 [cited April 2023; Available from: https://www.usability.gov/get-involved/blog/2006/09/correct-number-of-test-participants.html .

18. J, N. Why you only need to test with 5 users: Nielsen Norman Group. 2000 April 2023]; Available from: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ .

19. J N. Academic Press; New York: 1993. Usability Engineering.

20. EuroQol G. EuroQol--a new facility for the measurement of health-related quality of life. Health Pol. 1990;16(3):199–208. doi: 10.1016/0168-8510(90)90421-9.

21. Trotti A. The evolution and application of toxicity criteria. Semin. Radiat. Oncol. 2002;12(1 Suppl 1):1–3. doi: 10.1053/srao.2002.31353.

22. Trotti A., et al. CTCAE v3.0: development of a comprehensive grading system for the adverse effects of cancer treatment. Semin. Radiat. Oncol. 2003;13(3):176–181. doi: 10.1016/S1053-4296(03)00031-6.

23. Basch E., et al. Patient versus clinician symptom reporting using the national cancer Institute common terminology criteria for adverse events: results of a questionnaire-based study. Lancet Oncol. 2006;7(11):903–909. doi: 10.1016/S1470-2045(06)70910-X.

24. Basch E., et al. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J. Natl. Cancer Inst. 2009;101(23):1624–1632. doi: 10.1093/jnci/djp386.

25. Feldman-Stewart B., Brundage M.D., Tong C. Information that affects patients' treatment choices for early stage prostate cancer: a review. Can. J. Urol. 2011;18(6):5998–6006.

26. Eccles D.W., Arsal G. The think aloud method: what is it and how do I use it? Qualitative Research in Sport, Exercise and Health. 2017;9(4):514–531.

27. van den Haak M., De Jong M., Jan Schellens P. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 2003;22(5):339–351.

28. Rezaee R., et al. Development, usability and quality evaluation of the resilient mobile application for women with breast cancer. Health Sci. Rep. 2022;5(4):e708. doi: 10.1002/hsr2.708.

29. Gatto S.L., Tak S. Computer, internet, and email use among older adults: benefits and barriers. Educ. Gerontol. 2008;34(9):800–811.

30. Tsetsi E., Rains S.A. Smartphone Internet access and use: extending the digital divide and usage gap. Mob. Media Commun. 2017;5(3):239–255.

31. Aiyegbusi O.L., et al. Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection. Nat. Commun. 2022;13(1):6026. doi: 10.1038/s41467-022-33826-4.

32. Snyder C., et al. Making a picture worth a thousand numbers: recommendations for graphically displaying patient-reported outcomes data. Qual. Life Res. 2019;28(2):345–356. doi: 10.1007/s11136-018-2020-3.

33. PROTEUS Consortium. 14/12/2022]; Available from: https://theproteusconsortium.org/proteus-trials/study-reporting/displaying-results/ .

34. Bennett A.V., et al. Mode equivalence and acceptability of tablet computer-, interactive voice response system-, and paper-based administration of the U.S. National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) Health Qual. Life Outcome. 2016;14:24. doi: 10.1186/s12955-016-0426-6.

Associated Data

Data Availability Statement

Data will be made available on request.