The use of virtual nominal groups in healthcare research: An extended scoping review.

Introduction

Consensus group methods are often used to synthesize expert opinions when evidence is lacking or contradictory. They have been increasingly utilized in healthcare, business, engineering, and education [ 1 – 4 ].

These methods have been used to inform and evaluate a variety of healthcare-related activities (e.g. defining diagnostic criteria, informing management guidelines [ 2 , 5 , 6 ]; course evaluation [ 7 ]; amongst other uses [ 8 – 12 ]–Foundational principles of consensus methods include anonymity, iteration, controlled feedback, statistical group response, and structured interaction [ 1 ].

The NGT was developed for idea generation and group decision-making [ 3 ]. The structured format allows for effective generation and prioritization of ideas. The NGT has several key steps: presentation of the nominal question, silent generation of ideas in writing, round-robin sharing of ideas from participants, group discussion and clarification of each idea, followed by anonymous individual voting to rank priority areas. Feedback of results followed by more discussion and re-rating may occur [ 13 , 14 ].

The NGT differs from other consensus techniques by employing an in-person meeting among 5–12 participants [ 15 ]. This is touted by many to be a strength, as it not only allows discordant ideas on topics of mutual interest to be freely expressed and synthesized, but it also affords the opportunity to explore any differences in opinions. Further, the collaborative feature of the NGT may increase ownership of research among stakeholders and enhance the potential of informing policy or practice [ 3 ]. However, the small number of participants is also regarded as a limitation, and the potential for dominant members to unduly influence group decision-making cannot be dismissed, even if NGT aims to counteract this effect in its structured procedure.

The coronavirus disease of 2019 (COVID-19) pandemic has fundamentally changed the way we work, learn, and conduct research [ 16 ] and NGT has not been exempt from the shift; many researchers transitioned the in-person NGT meeting to a virtual environment [ 17 – 20 ]. For instance, Nelson et al. (2022) employed 3 nominal groups to identify burnout strategies in resident physicians, one being the traditional in-person meeting followed by two of which were held virtually via Zoom [ 17 ]. Timmermans et al. (2022) conducted three virtual NGTs (vNGT) and provided recommendations on the transition to the synchronous, online environment, but their claims were drawn solely from the authors’ subjective experiences [ 20 ].

There remains ambiguity regarding the extent to which other researchers have shifted the NGT to a virtual format, the types of platforms used, and modifications made to the technique. In addition to this exploratory inquisition, it is also important to consider whether researchers voiced any challenges, or perceived advantages or disadvantages. Taken collectively, these lessons learned would provide guidance for future users of vNGT.

The COVID-19 pandemic has supported an upsurge in information and communication technologies, which fundamentally changed how individuals interact. The Media Synchronicity Theory (MST) is a conceptual framework that considers the effectiveness of information and communication technologies in facilitating group work [ 21 ]. Through the lens of MST, all communication activities are grouped into two simple processes: conveyance (transmission of new information to the receiver that enables the creation and revision of individual understanding of a problem) and convergence (mutual process that governs how individuals understand and negotiate a common ground for a problem), the latter of which may involve more authentic, rapid back and forth transmission of information. For the NGT, the idea generation phase would presumably require more conveyance, while the discussion and clarification phase emphasize more convergence. Both processes are necessary, but the proportion depends on the complexity and inherent characteristics of the research.

In summary, the NGT has been used in a variety of research settings, to inform important decisions. Although interaction between panel members is fundamental to the process, many in-person meetings have moved online due to the pandemic. At present, it is unclear to what extent the NGT has been undertaken virtually, and what implications this may have for the decision process.

Objective

The overarching purpose of this study is to explore the use of the virtual Nominal Group Technique (vNGT) in healthcare research. Specific objectives are to answer the following questions: 1) To what extent has the NGT been used virtually? 2) What virtual communication platforms are used? 3) What modifications to the technique were made to accommodate this online format? And 4) What advantages and disadvantages were noted by authors?

Methods

Following a study protocol that has been published [ 22 ], a scoping review was conducted. A scoping review was considered appropriate since this topic is poorly defined, and the purpose was to map the literature, find key concepts, types and sources of evidence and identify gaps in the literature. The study followed the Arksey and O’Malley framework [ 23 ] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Review [ 24 ]. Written ethics approval was granted from the University of Ottawa Research Ethics Board on September 26, 2022. All participants answering the survey provided written informed consent.

Step 1: Identifying the research question

Our study began with a broad objective: to explore the use of the vNGT in healthcare research. This study included all English-language published research in healthcare and healthcare education that used the NGT in a virtual format. This included using any non-in-person format, such as videoconferencing. The outcomes of interest included the author’s description of how the vNGT was used, and the perceived success of the process, benefits, risks, and challenges. Informed by a preliminary search of the literature, we decided upon the following questions:

To what extent has the NGT been used virtually?

What virtual communication platforms were used?

What modifications to the technique were made to accommodate this online format?

What advantages and disadvantages were noted by authors?

Step 2 & 3: Identifying relevant articles and article selection

We started with the following framework: Population, Concept, Context. The population included any published research studies using the nominal group technique, the concept entailed the use of virtual modalities to execute the nominal group technique, and context involved any study topic.

Several pilot searches were undertaken to define the search strategy. Details can be found in the published protocol [ 22 ]. Final inclusion and exclusion criteria are noted in Table 1 .

The final search strategies were executed by an information specialist (KF) and peer reviewed using the PRESS guideline [ 25 ]. The searches were conducted July 15th, 2022, in: MEDLINE(R) ALL (OvidSP), Embase (OvidSP), CINAHL (EBSCOHost), ERIC (OvidSP), Education Source (EBSCOHost), APA PsycInfo (OvidSP), Web of Science, and Scopus to retrieve references published January 2020 to July 2022. This time period was selected to detect studies conducted during the COVID-19 pandemic to reflect the more rapid shift to virtual formats. No search filters, or language limits were used, but conference abstracts were removed since only full papers were of interest. The search strategies are included in S1 File .

Using the Covidence software [ 26 ], all titles and abstracts were reviewed in duplicate with varying pair combinations by three co- authors (SHL, SHM, MG and a research assistant). Those meeting inclusion criteria, or if not clearly conducted in person were pulled for full text review. Duplicate review of full text articles against inclusion/exclusion criteria was completed. All conflicts were resolved via consensus discussion with a third member of the team.

Step 4: Charting the data

Through an iterative process the data extraction form was developed. In addition to publication-level information and demographics, concepts related to the MST were explored. Details can be found in the published protocol [ 22 ].

Following best practices from the manual for evidence synthesis [ 27 ] at least two members (SL, SHM) independently reviewed 20% of articles applying the final data extraction form. Thereafter, one member of the research team carried out data extraction with verification from a second member. Any ambiguous items that arose were resolved through discussion with the senior author (SHM).

Step 5: Collating, summarizing, and reporting the results

Both quantitative and thematic analyses were used to synthesize study results. Quantitative analysis focused on the nature (e.g., education, clinical research, guideline development) and distribution of relevant articles. Two members of the research team (SL, SHM) independently reviewed the data to identify preliminary themes as informed by the MST. Several group meetings with all team members were held to review the data and to agree on a final summary of findings.

Step 6: Survey of the authors

Upon review of several studies, it became apparent that many of the articles did not comment on the “virtual” aspect of the NGT. As a result, we disseminated an online survey to the corresponding authors for articles included in the study. The survey sought to confirm which virtual platform was used and for which steps of the NGT, if additional functions were used (e.g., chats features) or modifications made to the NGT to accommodate the virtual platform, why the virtual platform was used, their general impressions and perceived benefits and challenges, comparing their experience with in-person NGTs, and any lessons learned. The survey is available in S2 File . The Survey. The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist can be found in S3 File .

Results

A total of 2,589 records were identified through database searching, of which 1,754 were removed as duplicates ( Fig 1 ). Excluding irrelevant titles and abstracts, 598 full-text articles were assessed for eligibility based on our exclusion criteria. Resulting in 32 full-text articles meeting inclusion criteria.

Study demographics

Demographic information can be found in Table 2 . The largest number of studies were published in 2022 (50%; 16/32) [ 6 , 28 – 42 ], followed by 2021 [ 43 – 54 ] and 2020 (13%; 4/32) [ 19 , 55 – 57 ]. As seen in Table 2 geographic distribution, virtual studies were predominantly carried out at the national level (53%; 17/32), followed by intercontinental (19%; 6/32), international (16%; 5/32), and local settings (6%; 2/32).

Research questions were more related to healthcare (87.5%; 28/32) than to healthcare education (12.5%; 4/32). Within healthcare, topics were varied and included items such as improving care for inmate dementia [ 30 ], home rehabilitation for stroke survivors [ 49 ], COVID-19 vaccine rollout in pharmacies [ 51 ] barriers to testing lipids and achieving disease control in rheumatoid arthritis [ 56 , 57 ]. Patients were often central to the vNGT with topics including patient views on the treatment of osteoporosis [ 28 ], multiple myeloma [ 50 ] or osteoarthritis [ 43 ] and patient reported outcomes in heart failure [ 33 ].

vNGT participants

Five studies did not report the number of participants (15.6%) [ 6 , 38 , 41 , 46 , 47 ]. For those that did, the total number of participants per NGT group ranged from 2–20, however it was not always clear how many participants were in each NGT group as some studies ran several NGT groups in parallel ( Table 2 ).

Virtual modality

Table 3 provides details of the technique itself. All studies used an online conferencing modality, but many did not report which platform was used (43.7%; 14/32). Of those that did (n = 18), the most common software noted was Zoom (66.6%; 12/18), followed by Microsoft Teams (11.1%; 2/18), GoTo (11.1%; 2/18), Crisco (5.6%; 1/18) and Skype (5.6%%; 1/18). Half of the articles also added an email or e-survey modality to the study design (50%; 16/32) either before, during, or after participants convened online. The rest used telephone communication in addition to online conferencing (9%; 3/32). Some studies held virtual sessions in parallel with in-person offerings to conduct the NGT simultaneously among various groups (16%; 5/32).

vNGT modifications for virtual

Modifications to accommodate the technique virtually were often implied to have taken place, but rarely commented upon (25%; 8/32). Notably, the online break-out room functionality replaced physical meeting rooms to conduct multiple nominal group sessions [ 51 ]. Investigators also implemented online sheets [ 51 ], interactive whiteboards, and slideshows [ 39 ] to complement the process.

Asynchronous communication modalities, defined as non-interactive, non-simultaneous platforms such as email threads and e-survey, were noted in a quarter of the articles (25%; 8/32). The idea generation step was conducted asynchronously in 8 of 32 articles (25%). At the discussion and clarification stage, Bavelaar et al., 2022 [ 29 ] and Hoops et al., 2022 [ 31 ] offered asynchronous participation in conjunction with the interactive discussion in the form of email threads.

Five studies held in-person and virtual sessions in parallel, two of which allowed for both in-person and virtual within the same meeting based on context-dependent needs (e.g., geographic difference, member preference, technical challenges) [ 29 , 31 ].

vNGT method

A minority of studies defined consensus a priori (16%; 5/32). Four of the articles used a percentage score ranging from 50% to 75% agreement as the cut-off value [ 31 , 48 , 49 , 54 ].

Documentation

In many instances, the discussion phase of the NGT was formally documented either via audio or video recording (41%; 13/32) or transcription in the form of written notes during the meeting (69%; 22/32). Some opted to using both recording and live transcription (34%; 11/32).

Other findings

Our inclusion criteria required that all stages of the vNGT be virtual (i.e., no in-person components). However, during the screening we did note that 52 articles incorporated at least one virtual component.

Descriptive characteristics

Benefits and challenges of vNGT

Only eight articles (25%;8/32) provided comments on the benefits or challenges (see Table 4 ). Online participation afforded geographical and scheduling flexibility, allowing a more representative sample of participants to convene in nominal group sessions [ 29 ]. One study suggested that the online nature of discussions may also have promoted the sharing of ideas among participants [ 51 ]. Janssens et al., (2021) suggested for older participants that the telephone was preferable [ 50 ]. The addition of asynchronous platforms was advantageous for one study as it provided more time for researchers to procure stakeholder perspectives, participants to understand study material through repeated review, and to efficiently prioritize and rank ideas.

In contrast, some comments directly addressed the challenges of adapting onsite activities to videoconferencing, email threads, and phone calls, which could have affected participant engagement. Two articles suggested that virtual modalities could have acted as a barrier to participation to some individuals, such as older participants [ 50 , 57 ]. Prospective participants may have declined invitations to the study due to the extra effort required to partake in online meetings and asynchronous activities [ 44 ]. Furthermore, differences in geographical time zones posed time constraints for participants, which could have influenced group discussions, and therefore the results of the study [ 51 ]. One study mentioned that results could have been further compromised via asynchronous modalities as participants would be directed to a priori information online [ 44 ].

Participant feedback

One article openly gathered feedback from participants regarding the virtual nature of the NGT [ 44 ]. Participants expressed that the asynchronous format provided ample time to prepare ideas before of convening online.

Corresponding author survey results

The survey was completed by half of corresponding authors (50%; 16/32). Authors reported having been involved in in-person sessions 0 to 4 times in the past, and virtual sessions from 1 to 10 times. Participants were also asked their general impression of how the virtual worked compared to in-person NGT. Overall, 44% (7/16) felt that the vNGT was better, 36% (6/16) felt that both media performed similarly, and 19% (3/16) felt that vNGT was inferior to in-person NGT. Thematic analysis across the questions fell under benefits and concerns of using the vNGT as seen in Table 5 .

Discussion

This study set out to explore the use of the vNGT. Our findings demonstrate researchers use virtual platforms considering restrictions for travel and the requirements for social distancing. Although 8 databases were included in our study, the vast majority of articles reporting use of a virtual platform were from medical journals and related to healthcare.

The pandemic has imposed unprecedented sanctions in the way research is conducted, necessitating the transition from physical to virtual interactive modalities. Recent guidelines have suggested that researchers should report any modifications made as a result of extenuating circumstances (e.g., COVID-19), report important modifications to the methods, including mitigating strategies and impact on the results [ 58 ]. Our review demonstrated that most published research did not adhere to these guidelines; approximately a quarter of studies reported modifications to the method and just under a half which virtual platform was used. None considered the impact on outcomes. Overall, this is consistent with poor reporting seen in consensus research [ 4 , 59 , 60 ].

A few articles discussed the benefits and challenges of conducting the NGT virtually. Commonly cited benefits were increased accessibility for participants by overcoming barriers to participation, such as psychological discomfort in group settings [ 19 ], transportation issues [ 29 , 49 ], or time constraints [ 32 ]. This advantage may have been amplified for healthcare professionals seeking to actively participate in research in the midst of battling the COVID-19 pandemic and has been previously reported [ 58 ]. However, lack of access or ability to use digital platforms may have reduced accessibility for some participants, due to socioeconomic disadvantage [ 28 ] or advanced age [ 50 , 57 ]. Previous literature has certainly described age, race and literacy-related disparities in the use of technology with online populations being younger and more affluent [ 61 , 62 ]. However, some report that online focus group participants were more likely to be non-white, less educated, and less healthy than the in-person sample [ 63 ]. Thus, online methods may lead to less representative participant demographics.

Several other limitations were noted such as timing of sessions if participants are from different time zones [ 51 ], and technical difficulties [ 28 ]. Testing the technology prior to use and having designated staff to address technical issues has been recommended [ 20 ], but is not always sufficient [ 28 ].

From the perspective of MST, the virtual platforms offered a variety of options to enhance communication. The capacity of communication modalities to support synchronicity is further informed by qualities intrinsic to the media, including, but not limited to transmission velocity, which refers to the speed at which a message reaches the recipient; parallelism, the number of simultaneous transmissions at any given time, and symbol sets, meaning different ways in which a message can be encoded. In general, symbol sets and transmission velocity enhance synchronicity, whereas parallelism decreases it [ 21 , 64 , 65 ].

The MST helped to inform our understanding of vNGT in pragmatic terms. Studies employed a diverse array of modalities depending on the NGT stage. In idea generation, which predominantly comprises one-way input, several studies [ 29 , 31 , 35 , 36 , 44 , 48 , 53 , 54 ] adopted asynchronous modalities such as e-surveys or emails. On the other hand, discussion and clarification phases were conducted synchronously as demonstrated in all studies in our review with two also providing asynchronous additional options [ 29 , 31 ]. This is in keeping with the idea that convergent communication benefits from higher transmission velocity. See Fig 2 for considerations at each stage.

Concerns about engagement and interactivity were raised by several corresponding authors in our study, but none explored this in detail. Media richness theory suggests that computer-mediated group communication has lower social presence and less task focus [ 66 ] but this theory predates the current advances in platforms, which have improved our ability to see and interact with colleagues. One study that directly compared the web-based NGT to a traditional in-person format was using asynchronous technology with no visual capabilities [ 66 ]. Not surprisingly, the online participants were significantly less satisfied with the decision process compared to the traditional session, but only marginal differences were found in the outcomes [ 66 ]. Another study compared video to in person interviews. Interestingly Skype interviews yielded some of the most unguarded responses and richest data [ 67 ].

For this review, the vNGT was defined as having all stages in a virtual format. Several authors used a combination of virtual and in-person, but these were not included and are worthy of further exploration.

Conclusion

The vNGT offers several potential benefits; inclusion of geographically dispersed participants, reduced time and expense for travel/meeting accommodations, and flexible scheduling. Reduced engagement and reduced participation for technology-challenge individuals are considerations. Since research on the vNGT itself is lacking, authors should clearly report modifications made and risks/benefits as well as potential impact on the consensus decision. Any minor adjustments in research protocols can introduce unanticipated vulnerabilities, thus compromising results and improved reporting is essential. In addition, more research is required to directly compare in-person to vNGTs including potential implications for cognitive biases [ 68 ].

Supporting information

Search strategies.

(DOCX)

The survey.

(DOCX)

PRISMA SCR checklist.

(DOCX)

Dr. Humphrey-Murto would like to acknowledge the work of Amanda Pace and Kate Scowcroft from the Department of Innovation in Medical Education for assisting with the study.

References

1 Jones J , Hunter D . Consensus methods for medical and health services research . BMJ : British Medical Journal 1995 ; 311 : 376 . doi: 10.1136/bmj.311.7001.376 7640549 PMC2550437

2 Murphy M , Black N , Lamping D , et al . Consensus development methods, and their use in clinical guideline development . Health Technology Assessment (Winchester , England) 1998 ; 2 : i – 88 . 9561895

3 Harvey N , Holmes CA . Nominal group technique: an effective method for obtaining group consensus . International journal of nursing practice 2012 ; 18 : 188 – 194 . doi: 10.1111/j.1440-172X.2012.02017.x 22435983

4 Humphrey-Murto S , Varpio L , Wood TJ , et al . The use of the Delphi and other consensus group methods in medical education research: a review . Academic Medicine 2017 ; 92 : 1491 – 1498 . doi: 10.1097/ACM.0000000000001812 28678098

5 Salvi A , Nitti C , Fabbri A , et al . Diagnosis and Treatment of Deep Vein Thrombosis in the Emergency Department: Results of an Italian Nominal Group Technique Study . Clinical and Applied Thrombosis/Hemostasis 2020 ; 26 : 1076029620959720 . doi: 10.1177/1076029620959720 33112649 PMC7791439

6 Ridgway A , Cotterill N , Dawson S , et al . Nocturia and chronic kidney disease: systematic review and nominal group technique consensus on primary care assessment and treatment . European Urology Focus . doi: 10.1016/j.euf.2021.12.010 35031353

7 Coker J , Castiglioni A , Massie FS , et al . Evaluation of an advanced physical diagnosis course using consumer preferences methods: The nominal group technique . The American Journal of the Medical Sciences 2014 ; 347 : 199 – 205 . doi: 10.1097/MAJ.0b013e3182831798 23552288

8 Myong Y , Shin H-I , Lee J-E , et al . Development and Validation of a New Scale to Assess Attitudes and Perspectives Toward Persons With Disabilities . Annals of Rehabilitation Medicine 2021 ; 45 : 331 – 340 . doi: 10.5535/arm.21046 34496476 PMC8435470

9 Tran C , Archibald D , Humphrey-Murto S , et al . What makes a high-quality electronic consultation (eConsult)? A nominal group study . Journal of Telemedicine and Telecare 2020 ; 26 : 239 – 247 . doi: 10.1177/1357633X18822885 30717627

10 Halman S , Dudek N , Wood T , et al . Direct observation of clinical skills feedback scale: development and validity evidence . Teaching and Learning in Medicine 2016 ; 28 : 385 – 394 . doi: 10.1080/10401334.2016.1186552 27285377

11 Kottmann A , Krüger AJ , Sunde GA , et al . Establishing quality indicators for pre-hospital advanced airway management: a modified nominal group technique consensus process . British journal of anaesthesia 2022 ; 128 : e143 – e150 . doi: 10.1016/j.bja.2021.08.031 34674835 PMC8792832

12 Virumbrales M , Elorduy M , Graell M , et al . COVID-19: Making the Best out of a Forced Transition to Online Medical Teaching—a Mixed Methods Study . Medical science educator 2022 ; 32 : 337 – 347 . doi: 10.1007/s40670-022-01518-9 35223143 PMC8864206

13 Van De A , Delbecq AL . Nominal versus interacting group processes for committee decision-making effectiveness . Academy of management Journal 1971 ; 14 : 203 – 212 .

14 Campbell S , Cantrill J . Consensus methods in prescribing research . Journal of clinical pharmacy and therapeutics 2001 ; 26 : 5 – 14 . doi: 10.1046/j.1365-2710.2001.00331.x 11286603

15 Humphrey-Murto S , Varpio L , Gonsalves C , et al . Using consensus group methods such as Delphi and Nominal Group in medical education research . Medical teacher 2017 ; 39 : 14 – 19 . doi: 10.1080/0142159X.2017.1245856 27841062

16 Shamsuddin A , Sheikh A , Keers RN . Conducting research using online workshops during COVID-19: Lessons for and beyond the pandemic . International Journal of Qualitative Methods 2021 ; 20 : 16094069211043744 .

17 Nelson V , Dubov A , Morton K , et al . Using nominal group technique among resident physicians to identify key attributes of a burnout prevention program . Plos one 2022 ; 17 : e0264921 . doi: 10.1371/journal.pone.0264921 35303009 PMC8932600

18 Alkwai HM , Mirza A , Abdwani R , et al . Consensus clinical approach for a newly diagnosed systemic juvenile idiopathic arthritis among members of the pediatric rheumatology Arab group . International Journal of Pediatrics and Adolescent Medicine 2021 ; 8 : 129 – 133 . doi: 10.1016/j.ijpam.2021.05.003 34350323 PMC8319680

19 Johnson BJ , Zarnowiecki D , Hutchinson CL , et al . Stakeholder generated ideas for alternative school food provision models in Australia using the nominal group technique . International journal of environmental research and public health 2020 ; 17 : 7935 . doi: 10.3390/ijerph17217935 33138030 PMC7663195

20 Timmermans L , Huybrechts I , Decat P , et al . Recommendations for Researchers on Synchronous, Online, Nominal Group Sessions in Times of COVID-19: Fishbone Analysis . JMIR Formative Research 2022 ; 6 : e34539 . doi: 10.2196/34539 35225814 PMC8963262

21 Dennis AR , Fuller RM , Valacich JS . Media, tasks, and communication processes: A theory of media synchronicity . MIS quarterly 2008 ; 575 – 600 .

22 Humphrey-Murto S , Ho Lee S , Gottlieb M , et al . Protocol for an extended scoping review on the use of virtual nominal group technique in research . PloS one 2023 ; 18 : e0280764 . doi: 10.1371/journal.pone.0280764 36662907 PMC9858029

23 Arksey H , O’Malley L . Scoping studies: towards a methodological framework . International journal of social research methodology 2005 ; 8 : 19 – 32 .

24 Tricco AC , Lillie E , Zarin W , et al . PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation . Annals of internal medicine 2018 ; 169 : 467 – 473 . doi: 10.7326/M18-0850 30178033

25 McGowan J , Sampson M , Salzwedel DM , et al . PRESS peer review of electronic search strategies: 2015 guideline statement . Journal of clinical epidemiology 2016 ; 75 : 40 – 46 . 27005575 10.1016/j.jclinepi.2016.01.021

26 Covidence systematic review software, Veritas Health Innovation.

27 Peters MD , Godfrey C , McInerney P , et al . Scoping reviews . Joanna Briggs Institute reviewer’s manual 2017 ; 2015 : 1 – 24 .

28 Beaudart C , Silverman S , Gold DT , et al . A Qualitative Study to Assess US Patient Preferences between new Transdermal System and Injectable Anabolic Therapies for Osteoporosis Treatment . Archives of Osteoporosis 2022 ; 17 : 57 . doi: 10.1007/s11657-022-01075-z 35378644 PMC8979159

29 Bavelaar L , Nicula M , Morris S , et al . Developing country-specific questions about end-of-life care for nursing home residents with advanced dementia using the nominal group technique with family caregivers . Patient Education and Counseling 2022 ; 105 : 965 – 973 . doi: 10.1016/j.pec.2021.07.031 34376304

30 du Toit S , Ng S . Improving care for older prisoners living with dementia in Australian prisons: perspectives of external organizations . The Gerontologist 2022 ; 62 : 543 – 555 . doi: 10.1093/geront/gnab077 34570214

31 Hoops K , Fahimi J , Khoeur L , et al . Consensus-driven priorities for firearm injury education among medical professionals . Academic medicine 2022 ; 97 : 93 – 104 . doi: 10.1097/ACM.0000000000004226 34232149

32 Klaic M , Kapp S , Hudson P , et al . Implementability of healthcare interventions: an overview of reviews and development of a conceptual framework . Implementation Science 2022 ; 17 : 10 . doi: 10.1186/s13012-021-01171-7 35086538 PMC8793098

33 Lawson CA , Lam C , Jaarsma T , et al . Developing a core outcome set for patient-reported symptom monitoring to reduce hospital admissions for patients with heart failure . European Journal of Cardiovascular Nursing 2022 ; 21 : 830 – 839 . doi: 10.1093/eurjcn/zvac019 35404418

34 Li S , Honey A , Coniglio F , et al . Mental Health Peer Worker Perspectives on Resources Developed from Lived Experience Research Findings: A Delphi Study . International Journal of Environmental Research and Public Health 2022 ; 19 : 3881 . doi: 10.3390/ijerph19073881 35409560 PMC8998053

35 Liem SI , Vliet Vlieland TP , van de Ende CH , et al . Consensus-based recommendations on communication and education regarding primary care physical therapy for patients with systemic sclerosis . Musculoskeletal Care 2023 ; 21 : 45 – 55 . doi: 10.1002/msc.1664 35689435

36 Love A , Cornwell P , Hewetson R , et al . Test item priorities for a screening tool to identify cognitive-communication disorder after right hemisphere stroke . Aphasiology 2022 ; 36 : 669 – 686 .

37 Mcclunie-Trust P , Jones V , Winnington R , et al . Doing case study research collaboratively: the benefits for researchers . International Journal of Qualitative Methods 2022 ; 21 : 16094069221096296 .

38 Ryan N , Vieira D , Gyamfi J , et al . Development of the ASSESS tool: a comprehenSive tool to Support rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes . Implementation Science Communications 2022 ; 3 : 1 – 13 . 35346390 10.1186/s43058-021-00236-4 PMC8959802

39 Singh JA . Disease modification in gout: a qualitative study of gout expert rheumatologists . Rheumatology Advances in Practice 2022 ; 6 : rkab107 . 35028499 10.1093/rap/rkab107 PMC8751810

40 Smith D , Cartwright M , Dyson J , et al . Selecting intervention content to target barriers and enablers of recognition and response to deteriorating patients: an online nominal group study . BMC Health Services Research 2022 ; 22 : 766 . doi: 10.1186/s12913-022-08128-6 35689227 PMC9186287

41 van Merode NA , Dawson S , Coulthard E , et al . Assessment and treatment of nocturia in neurological disease in a primary care setting: systematic review and nominal group technique consensus . European Urology Focus . 10.1016/j.euf.2021.12.012 35031351

42 Volkmer A , Cartwright J , Ruggero L , et al . Principles and philosophies for speech and language therapists working with people with primary progressive aphasia: An international expert consensus . Disability and Rehabilitation 2023 ; 45 : 1063 – 1078 . doi: 10.1080/09638288.2022.2051080 35352609

43 Singh JA . “I wish it had a place to go”: a nominal group study of barriers to the effectiveness of non-surgical treatments for knee osteoarthritis inclusive of minority populations . Arthritis Research & Therapy 2021 ; 23 : 1 – 13 . 34852836 10.1186/s13075-021-02676-8 PMC8633910

44 Al-Yateem N , Ahmad AM , McCreaddie M , et al . Synthesizing core nursing skills to support behavioural-based interviews for nurses in the UAE: a nominal group study . Journal of Nursing Management 2021 ; 29 : 953 – 961 . doi: 10.1111/jonm.13232 33301631

45 Aronson KI , O’Beirne R , Martinez FJ , et al . Barriers to antigen detection and avoidance in chronic hypersensitivity pneumonitis in the United States . Respiratory Research 2021 ; 22 : 225 . doi: 10.1186/s12931-021-01817-6 34376172 PMC8353836

46 Choquette D , Chan J , Bardi M , et al . Monitoring the transition of patients on biologics in rheumatoid arthritis: Consensus guidance for pharmacists . Pharmacy Practice (Granada) ; 19 . doi: 10.18549/PharmPract.2021.3.2377 34621449 PMC8455126

47 Dawson S , Duncan L , Ahmed A , et al . Assessment and treatment of nocturia in endocrine disease in a primary care setting: systematic review and nominal group technique consensus . European Urology Focus . 10.1016/j.euf.2021.12.008 34996740

48 Evangelista CB , Larsen KL , Cervero RM , et al . Corneal laser refractive surgery curriculum development in the military: using the nominal group technique . Military Medicine 2023 ; 188 : e145 – e151 . doi: 10.1093/milmed/usab262 34191032

49 Fisher RJ , Riley-Bennett F , Russell L , et al . Nominal group technique to establish the core components of home-based rehabilitation for survivors of stroke with severe disability . BMJ open 2021 ; 11 : e052593 . doi: 10.1136/bmjopen-2021-052593 34857570 PMC8640659

50 Janssens R , Lang T , Vallejo A , et al . Patient preferences for multiple myeloma treatments: a multinational qualitative study . Frontiers in medicine 2021 ; 8 : 686165 . doi: 10.3389/fmed.2021.686165 34295912 PMC8289885

51 Michel DE , Iqbal A , Faehrmann L , et al . Using an online nominal group technique to determine key implementation factors for COVID-19 vaccination programmes in community pharmacies . International journal of clinical pharmacy 2021 ; 43 : 1705 – 1717 . doi: 10.1007/s11096-021-01335-x 34633625 PMC8504097

52 Occomore-Kent LC , Hatch E , Cruice M . Scoping opinion: Speech and language therapists’ views on extending their role to the urgent ear, nose and throat pathway . International Journal of Language & Communication Disorders 2021 ; 56 : 975 – 988 . 34309979 10.1111/1460-6984.12650

53 Østbø N , Jimenez EY , Harb S , et al . Nutrition Information Resources Used by People With Systemic Sclerosis and Perceived Advantages and Disadvantages: A Nominal Group Technique Study . ACR Open Rheumatology 2021 ; 3 : 540 – 549 . doi: 10.1002/acr2.11293 34196508 PMC8363851

54 Wood L , Bishop A , Lewis M , et al . Treatment targets of exercise for persistent non-specific low back pain: a consensus study . Physiotherapy 2021 ; 112 : 78 – 86 . doi: 10.1016/j.physio.2021.03.005 34029781

55 Lee RE , O’Neal A , Cameron C , et al . Developing content for the Food Environment Assessment Survey Tool (FEAST): A systematic mixed methods study with people with disabilities . International journal of environmental research and public health 2020 ; 17 : 7781 . doi: 10.3390/ijerph17217781 33114296 PMC7660641

56 Navarro-Millán I , Cornelius-Schecter A , O’Beirne RJ , et al . Views of primary care physicians and rheumatologists regarding screening and treatment of hyperlipidemia among patients with rheumatoid arthritis . BMC rheumatology 2020 ; 4 : 1 – 7 . 32159074 10.1186/s41927-020-0112-5 PMC7057468

57 Owensby JK , Chen L , O’Beirne R , et al . Patient and rheumatologist perspectives regarding challenges to achieving optimal disease control in rheumatoid arthritis . Arthritis care & research 2020 ; 72 : 933 – 941 . doi: 10.1002/acr.23907 31008566 PMC6810709

58 O’Brien EC , Sugarman J , Weinfurt KP , et al . The impact of COVID-19 on pragmatic clinical trials: lessons learned from the NIH Health Care Systems Research Collaboratory . Trials 2022 ; 23 : 1 – 9 . 35597988 10.1186/s13063-022-06385-8 PMC9123729

59 Sinha IP , Smyth RL , Williamson PR . Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies . PLoS medicine 2011 ; 8 : e1000393 . doi: 10.1371/journal.pmed.1000393 21283604 PMC3026691

60 Boulkedid R , Abdoul H , Loustau M , et al . Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review . PloS one 2011 ; 6 : e20476 . doi: 10.1371/journal.pone.0020476 21694759 PMC3111406

61 Bailey SC , O’conor R , Bojarski EA , et al . Literacy disparities in patient access and health-related use of I nternet and mobile technologies . Health Expectations 2015 ; 18 : 3079 – 3087 . doi: 10.1111/hex.12294 25363660 PMC4417455

62 Davies L , LeClair KL , Bagley P , et al . Face-to-face compared with online collected accounts of health and illness experiences: a scoping review . Qualitative Health Research 2020 ; 30 : 2092 – 2102 . doi: 10.1177/1049732320935835 32667257

63 Rupert DJ , Poehlman JA , Hayes JJ , et al . Virtual versus in-person focus groups: Comparison of costs, recruitment, and participant logistics . Journal of Medical Internet Research 2017 ; 19 : e80 . doi: 10.2196/jmir.6980 28330832 PMC5382259

64 Daft RL , Lengel RH . Organizational information requirements, media richness and structural design . Management science 1986 ; 32 : 554 – 571 .

65 Dennis AR, Valacich JS. Rethinking media richness: Towards a theory of media synchronicity. In: Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences . 1999. HICSS-32 . Abstracts and CD-ROM of Full Papers . IEEE, 1999, pp. 10-pp.

66 Lago PP , Beruvides MG , Jian J-Y , et al . Structuring group decision making in a web-based environment by using the nominal group technique . Computers & Industrial Engineering 2007 ; 52 : 277 – 295 .

67 Jenner BM , Myers KC . Intimacy, rapport, and exceptional disclosure: A comparison of in-person and mediated interview contexts . International Journal of Social Research Methodology 2019 ; 22 : 165 – 177 .

68 Bhandari S , Hallowell MR . Identifying and controlling biases in expert-opinion research: Guidelines for variations of Delphi, nominal group technique, and focus groups . Journal of Management in Engineering 2021 ; 37 : 04021015 .