Development of Web-Based Education Modules to Improve Carer Engagement in Cancer Care: Design and User Experience Evaluation of the e-Triadic Oncology (eTRIO) Modules for Clinicians, Patients, and Carers.

Introduction

Background

Carers (including but not limited to spouses, partners, adult children, siblings, parents, or friends [ 1 ]) of adults with cancer assume many responsibilities in supporting and caring for their loved one [ 2 ]. Carers can experience many challenges in this demanding role and often report high distress [ 3 , 4 ], poor physical health, low quality of life, and unmet needs [ 5 , 6 ]. As carer burden increases, carers may neglect their own needs, which can also impact their ability to support and care for their loved one [ 7 , 8 ].

While issues faced by carers are well recognized by health care professionals [ 9 ], many clinicians report that they do not know how to appropriately engage with carers or address their unique challenges [ 9 , 10 ]. Oncologists have reported a lack of education about communicating with carers [ 10 ], and suboptimal carer-clinician communication is common [ 11 ]. Some carers report being overlooked in medical consultations and feeling disempowered and unprepared in their caregiving role [ 12 ]. Clinician inclusion and support of carers have been reported as highly valued by both carers and patients [ 12 ].

Improving carer engagement and support needs to be addressed from multiple perspectives. Not only are clinicians uncertain about how to include carers in consultations [ 9 ] but also many carers often lack confidence and skills in caregiving [ 12 , 13 ], and some patients are unsure about what role their carer should assume in medical consultations and decision-making [ 14 ]. Therefore, interventions targeting all members of the clinician-patient-carer trio are needed.

Web-based delivery of education offers efficacy, efficiency, ability to undertake training in discrete periods, lower cost, flexibility, and greater reach than traditional face-to-face formats [ 15 ]. A systematic review of web-based health education by George et al [ 16 ] found web-based training for health professionals to be as effective as or better than face-to-face formats on outcomes such as knowledge, skills, and attitudes. Web-based communication skills interventions have been found to be effective in improving self-rated clinician confidence, communication skills, and knowledge among cancer clinicians [ 15 ]. For example, a web-based module developed by our group to educate nurses about managing conflict involving patients and carers (the Triadic Oncology [TRIO]–Conflict module) was found to improve cancer nurses’ attitudes and confidence in interacting with carers [ 17 ].

Patients and carers can also benefit from web-based resources and educational tools [ 18 ]. A systematic review of digital psychosocial interventions for patients with cancer and carers found web-based interventions to be both feasible and acceptable [ 19 ]. Digital interventions for carers have been shown to improve carer outcomes, knowledge, and skills, with the additional benefit of being accessible from home, thus minimizing the demands on carers’ time [ 20 ]. For example, a web-based psychosocial intervention for patients with cancer, Stress-Aktiv-Mindern (STREAM), has demonstrated beneficial patient outcomes including reduced stress and improved quality of life [ 21 ]. Similarly, the psychoeducational platform, Comprehensive Health Enhancement Support System (CHESS), has demonstrated favorable outcomes among carers such as significant reduction in negative mood and carer burden [ 22 ]. These beneficial effects were comparable to those of traditional psychoeducation interventions [ 23 , 24 ]. While STREAM and CHESS demonstrate the efficacy of web-based patient and carer support, their focus is on psychosocial support. To date, there have been no web-based education modules dedicated to empowering and upskilling patients and carers in carer-relevant communication and engagement with cancer clinicians and in carer participation in cancer treatment decision-making. Therefore, we aimed to develop and evaluate a web-based learning tool to address these needs.

Interventions to support cancer carers are often difficult to implement in clinical practice and face barriers to implementation including problems with design, feasibility, acceptability, and cost [ 25 ]. One way to improve the acceptability and sustainability of an intervention is to use a co-design approach with the target population as stakeholders, to ensure that the program targets user needs and preferences. The person-based approach [ 26 ] ensures that intervention development is grounded in the perspectives and psychosocial context of end users via iterative, qualitative research with relevant stakeholders. This approach has been effectively used in the development of web-based health care interventions [ 27 , 28 ].

Objectives

This paper describes the development, iterative refinement, and user testing of evidence-based and interactive web-based interventions designed to improve engagement and communication with carers in cancer care. We have published the study protocol for a randomized controlled trial to test the efficacy of the e-Triadic Oncology (eTRIO) modules elsewhere [ 29 ]. However, necessary amendments to the planned randomized controlled trial due to the COVID-19 pandemic were made after publication of the protocol. The evaluation approach was revised to hybrid effectiveness and implementation studies using a pre-post, single-arm intervention design.

In this paper, we have reported about the development of web-based education modules for all 3 relevant stakeholder groups, including oncology health professionals and patients with cancer and carers (eTRIO for patients and carers [eTRIO-pc]).

Methods

Overview

The person-based co-design approach by Yardley et al [ 26 ] underpinned the module design. Development and user experience testing of the clinician (eTRIO) and patient-carer (eTRIO-pc) modules was undertaken in multiple cyclical phases of data collection, analysis, and integration, in a process of iterative refinement [ 30 ]. Consistent with the approach by Yardley et al [ 26 ], this involved (1) planning : development of module content based on evidence, qualitative interviews with stakeholders, and input from our expert advisory group; (2) design : iterative review and refinement based on advisory group feedback; and (3) development and evaluation of acceptability and feasibility : formal heuristic evaluation, System Usability Scale (SUS) questionnaire, and think-aloud review of the eTRIO modules by stakeholders ( Figures 1 and 2 ). The final phase of implementation and trialing is currently being conducted in a separate pre-post evaluation study, which will be reported elsewhere.

Phase 1: Development of eTRIO and eTRIO-pc Module Content

Development of the eTRIO Clinician Module

The content of the modules was informed by our extensive Triadic Oncology (TRIO) research program, which includes a systematic review of carer involvement in consultations [ 31 ]; qualitative interviews with oncology clinicians, patients, and carers [ 9 , 14 , 32 ]; quantitative and qualitative analyses of audiotaped oncology consultations [ 11 ]; a conceptual framework of carer involvement in medical decisions [ 33 ]; and carer communication guidelines for clinicians (TRIO Guidelines) developed via a Delphi consensus process [ 34 , 35 ]. Key clinician training needs, strategies, and behaviors relevant to the module were ascertained through this extensive research program.

On the basis of this prior research, we developed an initial draft of the eTRIO content. The draft module comprised 14 study sections (1 introductory section and 13 strategy areas covered in the TRIO Guidelines [ 34 , 35 ]). A clinician expert advisory group was formed to provide feedback about the module content, comprising medical oncologists (3/13, 23%), oncology nurses (2/13, 15%), psychologists (2/13, 15%), a radiation oncologist (1/13, 8%), an oncology surgeon (1/13, 8%), and the research team comprising psycho-oncologists (4/13, 31%). Each member of the clinician expert advisory group reviewed a text-based draft of the module content and provided written feedback about each module section, including interactive activities, reflective exercises, and wording of strategies. Multiple teleconferences were conducted, where group members provided feedback about the content and structure of each section. Major changes were discussed with the group until consensus was reached. Feedback from the advisory group was collated, and the module content was iteratively refined.

Development of the eTRIO-pc Patient-Carer Module

The eTRIO-pc module content was drafted based on a review of current web-based guidance for carers about involvement in medical consultations [ 18 ], qualitative studies of patients and carers [ 9 , 14 , 32 ], and analyses of audiotaped consultations [ 11 ]. A meeting with the staff at a leading nongovernment cancer support and advocacy organization (n=5) was also conducted to inform the content of the eTRIO-pc initial draft. The staff members were asked to describe the key content that should be included in the eTRIO-pc module, based on their experience in supporting patients and carers via a telephone information and support service.

Consumer advisers (3/5, 60% cancer carers and 2/5, 40% patients with cancer) also provided iterative feedback about the module content during a half-day workshop and via email. Consumer advisers were asked to comment about whether the module content was understandable, the relevance of the module content and feasibility of the suggested strategies, the language, and tone of the module. All feedback from the Cancer Council New South Wales support staff and consumer advisers was collated and discussed with the project team until consensus was reached through revisions.

After the development and iterative revision of the module content was complete, video vignettes modeling key carer communication skills were developed to supplement the written content. Video vignettes have been demonstrated as an effective educational tool for patients and carers and can improve accessibility for those with low literacy [ 36 , 37 ]. We engaged a professional medical education and communication production company to develop a script covering key learning areas for carers, as determined by the consumer advisory groups. The script was iteratively reviewed by the research team, consumer advisers, and a physician to ensure that the videos aligned with the TRIO communication guidelines [ 34 , 35 ] and were clinically relevant and feasible.

Phase 2: Iterative Design, Review, and Refinement of eTRIO and eTRIO-pc Web-Based Modules

Design and Refinement of eTRIO Clinician Module

As shown in Figure 1 , phase 2 involved consumer input and refinement of the modules. To translate the text-based content into an interactive web-based educational module, we studied the best practice principles for the delivery of e-learning to health professionals [ 16 , 38 - 40 ]. This included a review by de Leeuw et al [ 38 ] about e-learning features targeted at postgraduate medical students and health professionals completing ongoing professional development, which identified 6 domains of important elements for e-learning quality (preparation, design, communication, content, assessment, and maintenance). Informed by a previous review [ 38 ], we developed a base design and catalog of potential design features.

A prototype web platform was developed by a professional web development company. In 2 sessions conducted via Zoom (Zoom Video Communications), the clinician advisory group completed a walk-through of the module and provided comprehensive feedback. Their verbal and written feedback was collated and integrated into a revised web-based module.

Design and Refinement of the eTRIO-pc Patient-Carer Module

Similarly, as displayed in Figure 2 , phase 2 involved the conversion of the text-based module content for patients and carers into an interactive web-based platform. We conducted a review of the content and design features of other available evidence-based web-based platforms for carers [ 18 ], drew on the evidence base surrounding education for carers [ 41 - 43 ], and received input from the consumer advisory group. To inform the website design, we reviewed the publicly available web-based resources for carers.

The final design features of eTRIO and eTRIO-pc were implemented by a professional web development company and included interactive activities, video vignettes, and text-based content. The clinician and consumer advisory groups were given access to the draft module, and its content and format were revised based on their extensive feedback. An expert in human-centered IT design was involved in all stages of development of the clinician and patient-carer modules.

Phase 3: Heuristic Evaluation and “Think Aloud” User Experience Evaluation of eTRIO and eTRIO-pc Web-Based Modules

As shown in Figures 1 and 2 , phase 3 involved usability evaluations of the developed web-based module. We conducted a heuristic evaluation to discover technical and usability issues [ 44 ]. The modules were examined by the researchers to identify problems that did not comply with the usability principles recognized by Nielsen [ 45 ], which include consistency and standards, error prevention, and aesthetic and minimalist design. The severity and prevalence of the issues were ranked from 1 to 5, with a high rank indicating that the problem was a priority to fix, and the web platform was updated accordingly.

Usability and user experience testing for the penultimate versions of eTRIO and eTRIO-pc were conducted using think-aloud methodology with 11 participants, including clinicians, patients, and carers, all of whom were naïve to the TRIO Guidelines and modules. Think aloud is an effective evaluation method in which participants are provided with an interface and asked to verbalize their thoughts as they work through it [ 46 , 47 ]. Potential participants were identified through the research team’s professional networks and via social media advertisements.

The consenting participants completed a demographic questionnaire and a 4-item self-report measure of health literacy [ 48 ]. Participants were provided access to the relevant eTRIO module and asked to speak aloud their thoughts and impressions as they were completing the module (think-aloud). These sessions were conducted face to face or via videoconferencing. After working through the module, participants completed the SUS [ 49 ]. Think-aloud evaluations were audio recorded and transcribed verbatim. Transcripts were qualitatively analyzed using thematic analysis [ 50 ], which involved familiarization with the transcripts, coding of salient initial ideas as codes, identification of patterns in the codes to generate themes and subthemes, and iterative review of the themes and subthemes to ensure a coherent and comprehensive thematic structure. This process was conducted collaboratively and through iterative discussion by RLP, PB, ZB, MB, and IJ. Themes were related to the following: usability and technical issues, positive aspects of design and function, attitudes toward the content of the program, and perspectives about the impact or implementation of the program. All transcripts were analyzed based on the established thematic framework and were grounded in illustrative quotations. Subsequently, the modules were iteratively refined based on this feedback.

Ethical Considerations

Ethics approval was obtained from the University of Sydney Human Research Ethics Committee (protocol 2015/468). Participants provided informed consent and were given the opportunity to opt out at any point in time. Participant data were deidentified. Participants were provided a gift card worth Aus $20 (US $13.22) as compensation for their time.

Results

This section describes the clinician, patient, and carer feedback; iterative revisions made; and lessons learned in the design and development of the eTRIO and eTRIO-pc modules.

Phase 1: Development of eTRIO and eTRIO-pc Module Content

eTRIO Clinician Module

The clinician advisory group members (n=13) emphasized the importance of the module being concise. They suggested more content for the introductory section such as including a broad and inclusive definition for “carers,” content about culturally diverse carers, and more information about the legal and ethical aspects of involving carers. Clinicians also suggested the inclusion of self-reflections about one’s own attitudes and potential biases toward carers. Additional suggestions included addressing the diversity of settings in which family or carer interactions can occur (eg, outside traditional outpatient consultations such as at the patient’s bedside or via the telephone). Several clinicians stressed the importance of including clear learning outcomes and summaries for each of the 14 sections.

eTRIO-pc Patient-Carer Module

Cancer support staff (n=5) suggested a clear definition of the role of carers, tailoring based on the cultural backgrounds of patients and carers, and consideration of power imbalances that may exist in patient-carer relationships. They emphasized checking in on patient and carer emotions such as grief and distress, suggested that modules could include opportunities for self-reflection, and highlighted the need to include information about available support for carers.

The overall impression of the consumer advisory group (n=5) was that the language and tone of the draft module was very formal and academic; they wanted the tone to be more “personal,” “empathetic,” and “softer” and the language to be less prescriptive. They suggested additional strategies for patients with newly diagnosed cancer and carers, such as making notes during medical consultations, and suggested including quotes and stories from actual carers to illustrate examples.

Phase 2: Iterative Design, Review, and Refinement of eTRIO and eTRIO-pc Web-Based Modules

Overview

Table 1 describes the results from phase 2 using the e-learning design features by de Leeuw et al [ 38 ] applied to the eTRIO and eTRIO-pc modules.

eTRIO Clinician Module

During the transformation of content to a web-based module, features of e-learning [ 38 ] were applied as described in Table 1 . The design features of other web-based clinician training modules were examined, revealing display, navigation, and interactive activity styles (eg, minimal use of text, prominent navigation buttons, and clickable and expandable content). Our team worked closely with graphic and web designers to develop a consistent color scheme and intuitive navigation system and aimed to minimize visual noise on each page. The refined content and design features were transformed into a web-based web platform.

All members of the clinician advisory group (13/13, 100%) commented that there was excessive content and that there would not be clinician appetite for web-based training that extended beyond 2 hours in total. The content was subsequently condensed, with the core content displayed with the option of more extensive content, which could be expanded for clinicians interested in deeper learning regarding an issue.

The final eTRIO clinician module comprises 14 sections (submodules), of which clinicians must complete a minimum of 8. The sections range between 3 and 15 minutes in duration. The following 4 sections were deemed to be mandatory by the clinician advisory group, based on their critical relevance to all clinicians: section 1—introduction, section 4—building rapport with carers, section 7—supporting carers’ emotional and informational needs, and section 10—managing conflicting patient-carer treatment preferences. Clinicians could select additional 4 sections based on their interest and preference. The eTRIO module requires approximately 1.5 to 2 hours to complete, as determined by multiple stakeholders working through the content and documenting the amount of time each section required to complete.

eTRIO-pc Patient-Carer Module

Consistent with the principles of computer-based teaching for adult learners by Lau [ 51 ], the web-based eTRIO-pc module was created by transforming the written content into interactive, engaging learning activities. Our review of carer resources demonstrated several useful stylistic, formatting, and usability features, for example, the use of bullet points to convey written information, 1-page displays eliminating the need to scroll, and use of simple navigation buttons. These features and principles of web-based education were collated and discussed with the team’s academic IT expert and web developers to select and finalize the most appropriate features to be included. The resultant module prototype included video vignettes that could easily be played and paused, interactive activities such as “drag-and-drop” and “click to reveal” exercises, and type-your-response activities ( Multimedia Appendix 1 ). We maintained consistency in design and formatting across the clinician, patient, and carer modules.

We sent the prototype to the members of the consumer advisory group (n=5), and they provided written feedback via email and offered additional personal quotes that could be included in the module to personalize the content. They re-emphasized the need for content that was empathetic and offered practical advice. The final eTRIO-pc modules contain 7 sections for patients and 11 sections for carers and requires approximately 1 to 1.5 hours to complete.

Phase 3: “Think Aloud” Usability Evaluation of eTRIO and eTRIO-pc Web-Based Modules

Heuristic Evaluation

Using the heuristic evaluation method [ 44 ], we identified 37 usability issues across the draft eTRIO and eTRIO-pc modules, and each was rated for severity. The main areas of the identified problems were as follows: (1) inconsistency of icons and redundancy in buttons (5/37, 14% of the issues; eg, inconsistent use of star and book icons to indicate the bookmark function), (2) buttons and interactions were not working (16/37, 43% of the issues; eg, nothing happens when the print button is clicked), (3) layout problems (6/37, 16% of the issues; eg, text is not aligned with the textbox), and (4) presentation of content (10/37, 27% of the issues; eg, color selection in the bar-slider activity may be confusing) [ 52 ]. Following this evaluation, problems with high severity and prevalence were prioritized, and all issues that could be corrected were fixed before conducting the think-aloud user evaluations.

Think-Aloud User Experience Evaluations

Overall, 11 individuals (n=5, 45% health professionals; n=3, 27% patients; and n=3, 27% carers) participated in the think-aloud evaluations in individual sessions lasting between 40 and 60 minutes. Participant characteristics are displayed in Table 2 .

eTRIO Clinician Module

The usability of the module was rated by 5 clinicians, with a mean SUS score of 75 (range 68-80), which is interpreted as good [ 49 ]. All clinicians gave high ratings to their ability to use the module independently without technical assistance. Clinicians identified technical and navigation issues, which were subsequently rectified (such as the side scroll bar not appearing, text appearing outside the text bubble, and a sliding bar not working responsively). For some, the use of specific web browsers corrected these issues. Clinicians described the overall navigation through the module as “straightforward.” Formatting issues with font size and background color were highlighted. Clinicians commented that the ability to easily navigate back to certain sections to “refer back to later” was valued.

Content analysis of think-aloud evaluations revealed 7 categories related to clinicians’ attitudes toward the design and formatting of eTRIO. Clinicians appreciated that the modules could be completed in small “snackable” periods in any order, that they could keep track of what sections were completed ( trackable ), and that they were able to refer back to any module at any time . Clinicians enjoyed the “clickable” activities where they interacted with the content. Despite attempts to make the sections as short as possible (average 5-10 min/section), a few clinicians still perceived them as “too long,” with some stating that the videos were “slow” at times. They highlighted a preference for material that is brief, uses simple language, is easy to digest, and “skimmable.” A few clinicians reported “glossing over” or “tuning out” when sections were perceived as very long. They suggested simplifying the language and formatting the text to highlight important information (eg, use of bullet points and bold and italic style). Revisions were made to the text to further improve conciseness, including rephrasing the core content, moving some content to the expandable ‘additional information’ section, and greater use of bullet points and bold text. Where possible, videos were edited to remove nonessential scenes. Most participants appreciated that the content and activities were relevant and “relatable” to them as clinicians, that claims were “supported” by evidence, and that the activities and media were “diverse” and varied to facilitate engagement and interest. Illustrative quotes are provided in Table 3 .

eTRIO-pc Patient and Carer Module

The average SUS score as rated by 6 patients and carers was 78 (SD 16.2; range 55-97.5), which is interpreted as good [ 49 ] . Patients and carers were generally happy with the content and usability of the eTRIO-pc module. They commented that the content was relatable and were pleased by the emphasis placed on carers. Overall, they found the web platform easy to navigate and enjoyed the interactive activities; however, 1 (17%) of the 6 patients found the interface to be “overwhelming.” A major critique of the formatting and layout was that the pages were “too busy” and contained excessive information. Illustrative quotes are provided in Table 4 .

The final eTRIO and eTRIO-pc modules were updated based on this feedback. All technical and navigation issues were addressed by the web developers.

For both modules, the text was condensed and reformatted with the use of bold and italic style to highlight the important points and allow for easier reading and a more streamlined user interface.

Final Web Platform Design and Content Summary

The eTRIO modules reflect the reported informational needs of health professionals, patients with cancer, and carers. A full description of the module content has been published elsewhere [ 29 ]. The eTRIO modules have been rigorously designed to be easy to use, require minimal time commitment, and be flexible in terms of when and how the platform can be used. The modules are optimized for use on a computer but can also be used on a smartphone or tablet. Some notable features include the following: navigation buttons and a progress bar along the bottom of the page, expandable content for those who want deeper information about a specific topic, and downloadable summaries and lists. Notable interactive activities include the following: testing of knowledge through true-or-false exercises, identifying specific behaviors in a short video vignette, and building a question prompt list. Refer to Table 5 for descriptions and images of key features; full explanations of the interactive activities are provided in Multimedia Appendices 1 and 2 .

Discussion

Principal Findings

The web-based modules described in this paper represent a crucial step in the development and design of education for clinicians, patients, and carers that is evidence based, practical, and interactive and can be easily disseminated. Drawing on the evidence for best practice web-based learning design [ 38 , 51 ], we sought input from a variety of stakeholders to develop a unique learning experience strongly informed by the needs of the target populations. Rigor was ensured via 3 stages of development in which module content and design were continually revised and refined. Overall, participants were positive about the content and interface. The final prototype was appraised as highly acceptable, relevant, and feasible among the small sample of users; however, more studies are needed to confirm this and to ascertain the effectiveness of the intervention. We are currently conducting a pre-post evaluation of these modules to explore their potential effectiveness in improving communication within the patient-carer-clinician trio.

Lessons Learned

Throughout the development and design of these modules, we observed the specific needs and preferences of end users. The person-based approach to developing eTRIO and eTRIO-pc was highly dynamic, and the modules underwent numerous iterations throughout all phases of the design process, which included the involvement of consumers and user-driven evaluations. While there are multiple approaches to developing health interventions, the benefits of the person-based approach include grounding the design in user contexts and lived experiences, integrating feedback based on the actual use of an intervention, and investigating user needs and perspectives beyond just the usability of the intervention [ 26 ]. The utility of the person-based approach has been extolled in recent studies [ 53 - 55 ] and is supported by the findings of this study. The eTRIO development process ( Figures 1 and 2 ) provided the necessary building blocks to revise and refine the module for effective use in the real world. Consistent with other studies [ 56 , 57 ], we found that the collaborative co-design process led to positive evaluations of acceptability and usability and high levels of end-user satisfaction.

As highlighted in the person-based approach, the 3 user groups (clinicians, patient, and carers) demonstrated diverse learning preferences and needs. This was accommodated via tailoring the formatting or content to the strengths and contextual demands of different user groups and differentiating the content based on user needs. We found that clinicians had a strong desire for content that was written in simple, concise, and “sharp” language; could be “skim read”; and could be completed in brief, “snack-sized” sections. For example, clinicians in our advisory group often stressed that they lacked time and that training needed to be short, precise, and able to be stopped and restarted due to interruptions. On the other hand, the structure and time demands of training appeared to be less important to patients and carers. Instead, these groups emphasized the need for the module to be easy to use and navigate and for the content to be more conversational, empathetic, and in plain language (in contrast to the preferences of clinicians). Clinicians in our study valued the integration of academic literature and referencing, whereas some carers advocated for greater inclusion of carer experiences and quotes. The preferences of carers in our study are consistent with previous studies, which have similarly found that carers often prefer web-based education to have an empathetic and supportive tone, the web program to be easy to navigate, and the integration of other carers’ experiences into the content [ 58 - 60 ]. While several differences were identified between the clinician and carer user groups, there were also several similarities across all user groups in how the web-based modules should be structured and delivered. This is reflected in the evidence base, where health professionals, patients, and carers alike report that they prefer flexible, self-paced delivery of web-based programs that are interactive and include a variety of activities across media (visual, written, and auditory) [ 19 , 38 ]. These detailed insights are valuable in designing future training modules to facilitate their acceptability among users in each specific group.

The final interface used design principles to ensure engaging and interactive content. There is robust empirical evidence suggesting that interactivity in e-learning improves quality, efficacy, and learning outcomes [ 38 , 61 ]. For example, users of a web-based public health program had better learning outcomes when they used a gamified, interactive version featuring responsive design, learning challenges, visible progress, and rapid feedback compared to those using a minimally interactive, survey-based program [ 62 ]. Such interactivity was also demonstrated as important for users of the eTRIO modules. For example, in the initial design phases, when content was largely text based, the advisory committee members noted how dense the information appeared. While this was never intended to be the final format of the educational intervention, comments obtained from users in phase 1 highlighted the limitations of passive, didactic, text-heavy information. There is evidence suggesting that people do not learn effectively when information is given without any opportunity to reflect on, test, or demonstrate their knowledge and views [ 63 ]. Interactive activities, including assessments of learning and personal reflection activities, offer users the opportunity to reflect and reinforce their learning and become active participants in their education rather than passive consumers of information. Multimedia Appendices 1 and 2 display the engaging interactive activities that were acceptable to eTRIO users, which may be used in other web-based learning interventions and resources.

For both the clinician and patient-carer modules, we also incorporated a variety of media (text, audio, video, graphics, and images to cater to different learning styles and preferences. There is evidence suggesting that the use of multimedia may increase user satisfaction, acceptability, and engagement [ 64 , 65 ] and thus may improve adherence and broad implementation. The modules were designed such that users could navigate through them at their own pace and read, view, and explore the sections in a self-directed manner based on how they like to engage with and process content. For example, we found that users had mixed responses to the videos embedded in the training module. Some users commented that the videos were very long and that they would mentally switch off or skip them. Others claimed to be “visual learners” and thoroughly enjoyed the opportunity to observe scenarios in this format, especially because the videos included interactive “trigger” questions such as “What would you do next? ” where they were required to apply some of their learning to a scenario . This approach has been used in other web-based health interventions [ 66 , 67 ], which include complementary text, images, videos, audios, and interactive content to convey the educational content and cater to these diverse user preferences.

Strengths and Limitations

A thoughtful process of iterative design was conducted over a 2-year period, ultimately producing a suite of web-based interventions intended to improve communication between cancer clinicians, patients, and carers. However, important limitations should be noted. While extensive end-user feedback was collected through iterative feedback from clinician and consumer advisory groups, the sample size of participants (patients with cancer and carers) naive to the modules in phase 3 was small, and there was limited diversity among consumer advisers and participants. In addition, we did not measure the computer literacy of the participants, which may have impacted their views about the program’s usability. Thus, the attitudes and preferences of participants may not be reflective of the wider population. For example, we were unable to recruit a carer with low health literacy, and there was an overrepresentation of women.

Further usability and acceptability testing is currently underway in a larger study with a more diverse sample of patients and carers. Recruitment of participants in phase 3 was conducted through professional networks and social media, and therefore, the participants may have had a strong interest web-based learning or carer communication, which could have biased their views. This study focused only on development and user testing, and therefore, no assessment of the effectiveness or uptake of the modules has been conducted. Larger evaluation studies of the modules are currently being conducted, which will provide insight into the utility of the eTRIO modules in improving carer-related communication and inclusion.

Finally, while most patients with cancer have a carer or support person, some patients do not. Further studies are required to better understand the needs of people without a carer, which is beyond the scope of this study.

Future Directions

The eTRIO and eTRIO-pc modules are now undergoing pre-post evaluation with additional qualitative learner feedback to inform the broad implementation and uptake of these educational resources.

Conclusions

By including and being receptive to the needs of our user groups throughout the design process, we were able to create interventions that end users are likely to be more engaged and satisfied with.

This study was supported by Cancer Australia and Cancer Council New South Wales Grant, through the Priority-Driven Collaborative Cancer Research Scheme (project 1146383).

Conflicts of Interest: None declared.

e-Triadic Oncology patient and carer module features.

e-Triadic Oncology clinician module features.

Abbreviations

Comprehensive Health Enhancement Support System

e-Triadic Oncology

e-Triadic Oncology for patients and carers

Stress-Aktiv-Mindern

System Usability Scale

Triadic Oncology

1 Castro A Arnaert A Moffatt K Kildea J Bitzas V Tsimicalis A "Informal caregiver" in nursing: an evolutionary concept analysis ANS Adv Nurs Sci 2023 46 1 E29 E42 10.1097/ANS.0000000000000439 36006014 00012272-202301000-00012 36006014

2 Molassiotis A Wang M Understanding and supporting informal cancer caregivers Curr Treat Options Oncol 2022 04 23 4 494 513 10.1007/s11864-022-00955-3 35286571 10.1007/s11864-022-00955-3 35286571 PMC8918600

3 Geng HM Chuang DM Yang F Yang Y Liu WM Liu LH Tian H Prevalence and determinants of depression in caregivers of cancer patients: a systematic review and meta-analysis Medicine (Baltimore) 2018 09 97 39 e11863 10.1097/MD.0000000000011863 30278483 00005792-201809280-00005 30278483 PMC6181540

4 Bedaso A Dejenu G Duko B Depression among caregivers of cancer patients: updated systematic review and meta-analysis Psychooncology 2022 11 31 11 1809 20 10.1002/pon.6045 36209385 36209385 PMC9828427

5 Lambert S Girgis A Descallar J Levesque JV Jones B Trajectories of mental and physical functioning among spouse caregivers of cancer survivors over the first five years following the diagnosis Patient Educ Couns 2017 06 100 6 1213 21 10.1016/j.pec.2016.12.031 28089132 S0738-3991(16)30586-9 28089132

6 Lambert SD Girgis A Unmet supportive care needs among informal caregivers of patients with cancer: opportunities and challenges in informing the development of interventions Asia Pac J Oncol Nurs 2017 04 4 2 136 9 10.4103/2347-5625.204485 28503646 S2347-5625(21)00428-5 28503646 PMC5412151

7 Høeg BL Frederiksen MH Andersen EA Saltbæk L Friberg AS Karlsen RV Johansen C Dalton SO Horsbøl TA Bidstrup PE Is the health literacy of informal caregivers associated with the psychological outcomes of breast cancer survivors? J Cancer Surviv 2021 10 15 5 729 37 10.1007/s11764-020-00964-x 33169190 10.1007/s11764-020-00964-x 33169190

8 Kershaw T Ellis KR Yoon H Schafenacker A Katapodi M Northouse L The interdependence of advanced cancer patients' and their family caregivers' mental health, physical health, and self-efficacy over time Ann Behav Med 2015 12 49 6 901 11 10.1007/s12160-015-9743-y 26489843 10.1007/s12160-015-9743-y 26489843 PMC4825326

9 Laidsaar-Powell R Butow P Bu S Fisher A Juraskova I Oncologists' and oncology nurses' attitudes and practices towards family involvement in cancer consultations Eur J Cancer Care (Engl) 2017 01 01 26 1 e12470 10.1111/ecc.12470 26931469 26931469

10 Røen I Stifoss-Hanssen H Grande G Kaasa S Sand K Knudsen AK Supporting carers: health care professionals in need of system improvements and education - a qualitative study BMC Palliat Care 2019 07 16 18 1 58 10.1186/s12904-019-0444-3 31311536 10.1186/s12904-019-0444-3 31311536 PMC6636145

11 Laidsaar-Powell R Butow P Bu S Dear R Fisher A Coll J Juraskova I Exploring the communication of oncologists, patients and family members in cancer consultations: development and application of a coding system capturing family-relevant behaviours (KINcode) Psychooncology 2016 07 30 25 7 787 94 10.1002/pon.4003 26514374 26514374

12 McCarthy B Family members of patients with cancer: what they know, how they know and what they want to know Eur J Oncol Nurs 2011 12 15 5 428 41 10.1016/j.ejon.2010.10.009 21094087 S1462-3889(10)00153-5 21094087

13 Morris S Thomas C The carer's place in the cancer situation: where does the carer stand in the medical setting? Eur J Cancer Care (Engl) 2001 06 10 2 87 95 10.1046/j.1365-2354.2001.00249.x 11829054 11829054

14 Laidsaar-Powell R Butow P Bu S Charles C Gafni A Fisher A Juraskova I Family involvement in cancer treatment decision-making: a qualitative study of patient, family, and clinician attitudes and experiences Patient Educ Couns 2016 07 99 7 1146 55 10.1016/j.pec.2016.01.014 26873544 S0738-3991(16)30035-0 26873544

15 Berg M Ngune I Schofield P Grech L Juraskova I Strasser M Butt Z Halkett GK Effectiveness of online communication skills training for cancer and palliative care health professionals: a systematic review Psychooncology 2021 09 30 9 1405 19 10.1002/pon.5702 10.1002/pon.5702 33909328 33909328

16 George PP Zhabenko O Kyaw BM Antoniou P Posadzki P Saxena N Semwal M Tudor Car L Zary N Lockwood C Car J Online digital education for postregistration training of medical doctors: systematic review by the digital health education collaboration J Med Internet Res 2019 02 25 21 2 e13269 10.2196/13269 30801252 v21i2e13269 30801252 PMC6410118

17 Laidsaar-Powell R Keast R Butow P Mahony J Hagerty F Townsend J Young J Butt Z Juraskova I Improving breast cancer nurses' management of challenging situations involving family carers: pilot evaluation of a brief targeted online education module (TRIO-Conflict) Patient Educ Couns 2021 12 104 12 3023 31 10.1016/j.pec.2021.04.003 33941422 S0738-3991(21)00240-8 33941422

18 Keast R Butow PN Juraskova I Laidsaar-Powell R Online resources for family caregivers of cognitively competent patients: a review of user-driven reputable health website content on caregiver communication with health professionals Patient Educ Couns 2020 12 103 12 2408 19 10.1016/j.pec.2020.04.026 32739067

19 Heynsbergh N Heckel L Botti M Livingston PM Feasibility, useability and acceptability of technology-based interventions for informal cancer carers: a systematic review BMC Cancer 2018 03 02 18 1 244 10.1186/s12885-018-4160-9 29499663 10.1186/s12885-018-4160-9 29499663 PMC5834845

20 Zhai S Chu F Tan M Chi NC Ward T Yuwen W Digital health interventions to support family caregivers: an updated systematic review Digit Health 2023 05 19 9 20552076231171967 10.1177/20552076231171967 37223775 10.1177_20552076231171967 37223775 PMC10201006

21 Grossert A Urech C Alder J Gaab J Berger T Hess V Web-based stress management for newly diagnosed cancer patients (STREAM-1): a randomized, wait-list controlled intervention study BMC Cancer 2016 11 03 16 1 838 10.1186/s12885-016-2866-0 27809796 10.1186/s12885-016-2866-0 27809796 PMC5093991

22 DuBenske LL Gustafson DH Namkoong K Hawkins RP Atwood AK Brown RL Chih M McTavish F Carmack CL Buss MK Govindan R Cleary JF CHESS improves cancer caregivers' burden and mood: results of an eHealth RCT Health Psychol 2014 10 33 10 1261 72 10.1037/a0034216 24245838 2013-39991-001 24245838 PMC4121384

23 Kaltenbaugh D Klem M Hu L Turi E Haines AJ Hagerty Lingler J Using Web-based interventions to support caregivers of patients with cancer: a systematic review Oncol Nurs Forum 2015 03 42 2 156 64 10.1188/15.ONF.156-164 25806882 25806882

24 Northouse LL Katapodi MC Song L Zhang L Mood DW Interventions with family caregivers of cancer patients: meta-analysis of randomized trials CA Cancer J Clin 2010 08 16 60 5 317 39 10.3322/caac.20081 20709946 caac.20081 20709946 PMC2946584

25 Ugalde A Gaskin CJ Rankin NM Schofield P Boltong A Aranda S Chambers S Krishnasamy M Livingston PM A systematic review of cancer caregiver interventions: appraising the potential for implementation of evidence into practice Psychooncology 2019 04 07 28 4 687 701 10.1002/pon.5018 30716183 30716183 PMC6594143

26 Yardley L Morrison L Bradbury K Muller I The person-based approach to intervention development: application to digital health-related behavior change interventions J Med Internet Res 2015 01 30 17 1 e30 10.2196/jmir.4055 25639757 v17i1e30 25639757 PMC4327440

27 Beatty L Koczwara B Butow P Turner J Girgis A Schofield P Hulbert-Williams NJ Kaambwa B Kemp E Development and usability testing of a web-based psychosocial intervention for women living with metastatic breast cancer: finding my way-advanced J Cancer Surviv 2021 06 15 15 3 403 9 10.1007/s11764-021-01019-5 33723741 10.1007/s11764-021-01019-5 33723741

28 Wagner LI Duffecy J Begale M Victorson D Golden SL Smith ML Penedo FJ Mohr DC Cella D Development and refinement of FoRtitude: an eHealth intervention to reduce fear of recurrence among breast cancer survivors Psychooncology 2020 01 08 29 1 227 31 10.1002/pon.5297 31760667 31760667 PMC7376499

29 Juraskova I Laidsaar-Powell R Keast R Schofield P Costa DS Kay J Turner S Koczwara B Saunders C Jefford M Yates P Boyle F White K Miller A Morton RL Butt Z Butow P eTRIO trial: study protocol of a randomised controlled trial of online education modules to facilitate effective family caregiver involvement in oncology BMJ Open 2021 05 28 11 5 e043224 10.1136/bmjopen-2020-043224 34049902 bmjopen-2020-043224 PMC8166623 34049902

30 Gould JD Lewis C Designing for usability: key principles and what designers think Commun ACM 1985 28 3 300 11 10.1145/3166.3170

31 Laidsaar-Powell R Butow P Bu S Charles C Gafni A Lam W Jansen J McCaffery K Shepherd H Tattersall M Juraskova I Physician-patient-companion communication and decision-making: a systematic review of triadic medical consultations Patient Educ Couns 2013 04 91 1 3 13 10.1016/j.pec.2012.11.007 23332193 S0738-3991(12)00455-7 23332193

32 Laidsaar-Powell R Butow P Bu S Fisher A Juraskova I Attitudes and experiences of family involvement in cancer consultations: a qualitative exploration of patient and family member perspectives Support Care Cancer 2016 10 30 24 10 4131 40 10.1007/s00520-016-3237-8 27137213 10.1007/s00520-016-3237-8 27137213

33 Laidsaar-Powell R Butow P Charles C Gafni A Entwistle V Epstein R Juraskova I The TRIO framework: conceptual insights into family caregiver involvement and influence throughout cancer treatment decision-making Patient Educ Couns 2017 11 100 11 2035 46 10.1016/j.pec.2017.05.014 28552193 S0738-3991(17)30286-0 28552193

34 Laidsaar-Powell R Butow P Boyle F Juraskova I Facilitating collaborative and effective family involvement in the cancer setting: guidelines for clinicians (TRIO Guidelines-1) Patient Educ Couns 2018 06 101 6 970 82 10.1016/j.pec.2018.01.019 29526389 S0738-3991(18)30041-7 29526389

35 Laidsaar-Powell R Butow P Boyle F Juraskova I Managing challenging interactions with family caregivers in the cancer setting: guidelines for clinicians (TRIO Guidelines-2) Patient Educ Couns 2018 06 101 6 983 94 10.1016/j.pec.2018.01.020 29526388 S0738-3991(18)30042-9 29526388

36 Cruz-Oliver DM Pacheco Rueda A Viera-Ortiz L Washington KT Oliver DP The evidence supporting educational videos for patients and caregivers receiving hospice and palliative care: a systematic review Patient Educ Couns 2020 09 103 9 1677 91 10.1016/j.pec.2020.03.014 32241583 S0738-3991(20)30148-8 32241583 PMC7495206

37 Gysels M Higginson IJ Interactive technologies and videotapes for patient education in cancer care: systematic review and meta-analysis of randomised trials Support Care Cancer 2007 01 23 15 1 7 20 10.1007/s00520-006-0112-z 17024500 17024500

38 de Leeuw RA Westerman M Nelson E Ket JC Scheele F Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model BMC Med Educ 2016 07 08 16 1 168 10.1186/s12909-016-0700-7 27390843 10.1186/s12909-016-0700-7 27390843 PMC4939034

39 Hillen MA van Vliet LM de Haes HC Smets EM Developing and administering scripted video vignettes for experimental research of patient-provider communication Patient Educ Couns 2013 06 91 3 295 309 10.1016/j.pec.2013.01.020 23433778 S0738-3991(13)00048-7 23433778

40 Scott KM Baur L Barrett J Evidence-based principles for using technology-enhanced learning in the continuing professional development of health professionals J Contin Educ Health Prof 2017 37 1 61 66 10.1097/CEH.0000000000000146 28252469 00005141-201703710-00010 28252469

41 Berry LL Dalwadi SM Jacobson JO Supporting the supporters: what family caregivers need to care for a loved one with cancer J Oncol Pract 2017 01 13 1 35 41 10.1200/JOP.2016.017913 27997304 10.1200/JOP.2016.017913 27997304

42 Sherwood PR Given BA Given CW Bellizzi KM Gosney M Caregiver Knowledge and Skills Cancer and Aging Handbook: Research and Practice 2012 Hoboken, NJ Wiley-Blackwell 445 58

43 Connor KI Siebens HC Chodosh J Gaugler JE Kane RL Person-centered approaches to caregiving Family Caregiving in the New Normal 2015 San Diego, CA Academic Press 251 68

44 Alonso-Ríos D Mosqueira-Rey E Moret-Bonillo V A systematic and generalizable approach to the heuristic evaluation of user interfaces Int J Hum Comput Interact 2018 01 24 34 12 1169 82 10.1080/10447318.2018.1424101

45 Nielsen J Enhancing the explanatory power of usability heuristics Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1994 CHI '94 April 24-28, 1994 Boston, MA 152 8 10.1145/191666.191729

46 Duncker K Lees LS On problem-solving Psychol Monogr 1945 58 5 1 113 10.1037/h0093599

47 Nielsen J Usability Engineering 1994 Cambridge, MA Morgan Kaufmann

48 Chew LD Bradley KA Boyko EJ Brief questions to identify patients with inadequate health literacy Fam Med 2004 09 36 8 588 94 15343421 15343421

49 Brooke J SUS: A 'Quick and Dirty' Usability Scale 1995 Boca Raton, FL CRC Press

50 Braun V Clarke V Reflecting on reflexive thematic analysis Qual Res Sport Exerc Health 2019 06 13 11 4 589 97 10.1080/2159676x.2019.1628806

51 Lau KH Computer-based teaching module design: principles derived from learning theories Med Educ 2014 03 48 3 247 54 10.1111/medu.12357 24528459 24528459

52 Bonnaudet M A usability evaluation of TRIO's e-learning modules enhancing the communication between cancer patients, clinicians and carers KTH Royal Institute of Technology 2020 2024-02-29 https://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A1466296&dswid=-4538

53 Murfield J Moyle W O'Donovan A Planning and designing a self-compassion intervention for family carers of people living with dementia: a person-based and co-design approach BMC Geriatr 2022 01 14 22 1 53 10.1186/s12877-022-02754-9 35031015 10.1186/s12877-022-02754-9 35031015 PMC8759225

54 O'Cathain A Croot L Sworn K Duncan E Rousseau N Turner K Yardley L Hoddinott P Taxonomy of approaches to developing interventions to improve health: a systematic methods overview Pilot Feasibility Stud 2019 3 12 5 1 41 10.1186/s40814-019-0425-6 30923626 425 30923626 PMC6419435

55 Band R Bradbury K Morton K May C Michie S Mair FS Murray E McManus RJ Little P Yardley L Intervention planning for a digital intervention for self-management of hypertension: a theory-, evidence- and person-based approach Implement Sci 2017 02 23 12 1 25 10.1186/s13012-017-0553-4 28231840 10.1186/s13012-017-0553-4 28231840 PMC5324312

56 Yardley L Ainsworth B Arden-Close E Muller I The person-based approach to enhancing the acceptability and feasibility of interventions Pilot Feasibility Stud 2015 10 26 1 1 37 10.1186/s40814-015-0033-z 27965815 33 27965815 PMC5153673

57 Bradbury K Morton K Band R van Woezik A Grist R McManus RJ Little P Yardley L Using the person-based approach to optimise a digital intervention for the management of hypertension PLoS One 2018 13 5 e0196868 10.1371/journal.pone.0196868 29723262 PONE-D-17-30284 29723262 PMC5933761

58 Köhle N Drossaert CH Oosterik S Schreurs KM Hagedoorn M van Uden-Kraan CF Verdonck-de Leeuw IM Bohlmeijer ET Needs and preferences of partners of cancer patients regarding a web-based psychological intervention: a qualitative study JMIR Cancer 2015 12 29 1 2 e13 10.2196/cancer.4631 28410157 v1i2e13 28410157 PMC5367664

59 Vaughan C Trail TE Mahmud A Dellva S Tanielian T Friedman E Informal caregivers' experiences and perceptions of a web-based peer support network: mixed-methods study J Med Internet Res 2018 08 28 20 8 e257 10.2196/jmir.9895 30154074 v20i8e257 30154074 PMC6134228

60 Teles S Ferreira A Paúl C Attitudes and preferences of digitally skilled dementia caregivers towards online psychoeducation: a cross-sectional study Behav Inf Technol 2022 01 09 42 4 345 59 10.1080/0144929x.2021.2021285

61 Chang V Review and discussion: E-learning for academia and industry Int J Inf Manag 2016 06 36 3 476 85 10.1016/j.ijinfomgt.2015.12.007

62 Trevors G Ladhani F It’s contagious! examining gamified refutation texts, emotions, and knowledge retention in a real-world public health education campaign Discourse Process 2022 06 27 59 5-6 401 16 10.1080/0163853x.2022.2085477

63 Karpicke JD Roediger HL Repeated retrieval during learning is the key to long-term retention J Mem Lang 2007 08 57 2 151 62 10.1016/j.jml.2006.09.004

64 Zhang D Interactive multimedia-based e-learning: a study of effectiveness Am J Distance Educ 2005 09 19 3 149 62 10.1207/s15389286ajde1903_3

65 Violante MG Vezzetti E Virtual interactive e-learning application: an evaluation of the student satisfaction Comput Appl Eng Educ 2013 08 13 23 1 72 91 10.1002/cae.21580

66 Scott AF Ayers S Pluye P Grad R Sztramko R Marr S Papaioannou A Clark S Gerantonis P Levinson AJ Impact and perceived value of iGeriCare e-learning among dementia care partners and others: pilot evaluation using the IAM4all questionnaire JMIR Aging 2022 12 22 5 4 e40357 10.2196/40357 36150051 v5i4e40357 36150051 PMC9816950

67 Heynsbergh N Heckel L Botti M Livingston PM A smartphone app to support carers of people living with cancer: a feasibility and usability study JMIR Cancer 2019 01 31 5 1 e11779 10.2196/11779 30702432 v5i1e11779 30702432 PMC6374733