Pulling back the curtain on the consensus process: A case study of a modified-hybrid Delphi process

Abstract

Pulling Back the Curtain on the Consensus Process: A case study of a modified-hybrid Delphi process. Gaining consensus among experts is vital for developing guidelines, standards, and curriculum. The Delphi process and Nominal Group Technique (NGT) are commonly employed, with hybrid models emerging. This paper explores group dynamics during a modified-hybrid Delphi process. This study explores the presence, interplay and impact of group dynamics of a modified-hybrid Delphi consensus building process by examining facilitators’ perceptions and observations during the in-person (NGT) phase with expert panelists. A qualitative single-case study design was employed, utilizing observation notes, audio recordings and post-event facilitator reflections. By applying both inductive and deductive coding, a deeper understanding of the rarely examined aspects of group dynamics in the consensus process was achieved. Analysis revealed two themes: “Gaining Momentum” and “Sharing Perspectives.” Under “Gaining Momentum,” three observations included the development of shared understanding, perspective-taking, and a decreasing time to consensus. “Sharing Perspectives” highlighted concerns about hierarchy, the absence of dominant individuals, and the accommodation of diverse perspectives without succumbing to Groupthink. Key insights included thorough preparation to manage potential group dynamics during the NGT including facilitator training and a participant package to foster shared understanding for process and content. Despite minor issues considering hierarchies, the in-person (NGT) phase supported consensus building while effectively mitigating negative group dynamic issues. This study expands our understanding of group dynamics in consensus building during an NGT phase of a modified-hybrid Delphi process, offering valuable insights and practical recommendations preparing for and implementing successful in-person sessions.

Keywords

delphi technique < RESEARCH DESIGNS & METHODS implementing the study and quality control < MANAGING & SOURCING DATA qualitative research < RESEARCH DESIGNS & METHODS single-case research methods for the behavioural and health sciences (S) < RESEARCH DESIGNS & METHODS case study methods (S) < RESEARCH DESIGNS & METHODS

Gaining agreement on a specific topic among a group of experts in a field is a common approach to developing guidelines, standards, definitions, or curriculum. The Delphi process is one method used extensively across a broad array of domains to garner consensus on a topic.^1,2 This consensus process is a controlled and structured interaction between experts to encourage independent thought, aid in the gradual formation of opinion, and avoid direct confrontation among panelists to provide direction and recommendations on a topic of interest.^3,4 Most simply, the Delphi process involves three-steps: (1) collecting expert opinions or ideas through a questionnaire and summarizing the results, (2) returning the results as a second set of questions for feedback and evaluation, and (3) re-evaluating the results and producing a final product.^5,6 A Delphi process can be run in many ways, with several possible modifications, making it a flexible method,⁷ that is not overly prescriptive,⁸ more art than science with no true method.⁹ Another consensus building process is the Nominal Group Technique (NGT). The NGT provides a method for structured in-person interaction to obtain expert opinions that generates results similar to the Delphi process and is conceptually the same as the Delphi process while producing higher participant satisfaction than the Delphi process.¹⁰ Though less common, hybrids of the NGT and Delphi processes have been used and shown to be valid for gaining consensus across key stakeholders.^11–13

An essential aspect and advantage of the Delphi process is the structure and anonymity of the approach avoids many of the pitfalls of group dynamics that can compromise the credibility of group processes.⁶ Group dynamics are “the contexts within which individuals interact in groups… [including] within-group processes, group outcomes, and the perceptions or experiences among group processes” with a group defined as “three or more people who have come together for common reasons… and engage in some form of ongoing interpersonal interaction”.¹⁴ Much of the literature indicates one of the main purposes of the Delphi process, and the NGT, is to avoid the negative effects of group interactions. The NGT is identified as being less protective against the negative aspects of group interaction though even in the Delphi process there can be influences of conformity.¹⁵

In reality, following strict Delphi or NGT guidelines may not be ideal. A modified-hybrid Delphi process with an in-person session may be desirable for a group discussion to garner consensus and foster buy-in to the outcome of the process. In these situations, the specific strategies to develop a clear process for conducting a Delphi or NGT are lacking, specifically related to managing the in-person group dynamics. Though there is a wide body of literature on the Delphi process and its variations¹⁶ in general, the literature on the Delphi process and the NGT, as well as hybrid methods, do not discuss or examine the non-productive aspects of group dynamics in detail. Additionally, there is little to no literature or research on group dynamics, social influence and the psychology of participants during either of these consensus building processes.¹⁵ This is not surprising as the purpose of the Delphi and NGT methods is to obviate the influence of group dynamics on decision-making.¹⁷ Yet, despite the design of the Delphi process and NGT it is still possible group dynamics will have a negative effect.

This project was part of a broader project to develop expert consensus around guidelines for interprofessional competencies during health professional students’ practicum placements.¹⁸ For details about the overall method and context of the project refer to King et al.¹⁸ The purpose of this paper is to evaluate the group dynamics of a modified-hybrid Delphi and NGT process. As a less common modified-hybrid method was being used^16,19 the researchers were interested in unveiling process issues and exploring if any potential negative group dynamic influences were unconstrained during the process, providing an opportunity to pull the curtain back on rarely explored processes and interactions. To better understand the group dynamics of the modified-hybrid Delphi process the present paper has two purposes:

1. Explore facilitators’ perceptions and observations of the in-person phase of the Delphi process.

2. Compare the reviewed group dynamics influences with the facilitators’ perceptions to determine if the in-person session was biased by any group dynamic influences.

Group dynamic influences

The main unproductive group dynamic aspects identified in the literature include conformity to a certain viewpoint through groupthink, the bandwagon effect, the halo effect, and hierarchies and dominant individuals.^{1,2,9,10,17,20–22} During in-person interactions such as the NGT, these influences may arise. Table 1 provides an overview of these select group dynamics.

Table 1.

Relevant Negative Effects of Group Dynamics for the modified-Delphi and NGT.

Group dynamic	Prevalent characteristics during a hybrid or NGT technique	References
Groupthink	• Pressure placed on individuals that do not share the groups’ view or question the validity of group decisions through subtle social pressure such as being required to keep criticism within a range of acceptable dissent.	Janis, 1971²³; Kramer and Dougherty, 2013²⁴
	• Self-censorship through unwillingness to be seen as a dissenter and concern for the social repercussions of dissent.
	• Unanimity exists through an illusion that everyone shares the same opinion, enhanced by the idea that silence represents accord. When a unanimous view is arrived at, consensual and consensual validation occurs rather than critical thinking.
Bandwagon effect	• People conform to the behaviour or thinking of a group because it is pursued by the majority.	Bolger and Wright, 2011;¹⁵
	• Participants initially holding a minority position can be influenced by social comparison, social desirability, and a desire to minimize conflict to move towards the majority to reach consensus, e.g. getting on the wagon.	Winkler and Moser, 2016¹⁷
Halo effect or halo error	• Positive regard for other raters or experts may cause a rater to overvalue the opinions of the other raters and place too great a value on the other raters’ evaluations leading to inflated rater intercorrelations.	Feeley, 2002;²⁵
	• The researcher may overvalue the evaluations made by the panelists and will be uncritical of the results.	Goodman, 1987²⁶; Jacob et al., 2018²⁷
	• It has been suggested the selection of knowledgeable raters or panelists as opposed to experts can avoid this problem.
Hierarchies and dominant individuals	• Status differentials present in human interactions.	Islam and Zyphur, 2005²⁸
	• May be real or perceived, both are influential.	Calhoun et la., 2014²⁹
	• Hierarchies may support dominant individuals and those who look to maintain social dominance in a group.	Green et al., 2017³⁰
	• A dominant individual may impose their influence regardless of the presence of hierarchies.	Weiss et al., 2014³¹

Groupthink was originally developed by Janis³² as an explanation for political blunders by expert groups, such as the Bay of Pigs Invasion. Janis identified common threads of behaviour in these groups very close to findings from social-psychological experiments on group pressures such as conformity and maintenance of group cohesion. Though a highly detailed theory, most simply groupthink is concurrence-seeking amongst cohesive individuals that becomes so dominant that appraisal of alternative courses of action are overridden and people become disengaged from critical thinking. As a result of group pressure there is a deterioration in “mental efficiency” and “reality testing” producing conformity to group values and ethics.³³ The bandwagon effect is related to groupthink where people follow a majority opinion¹⁷ rather than supporting their own position.³⁴ The halo effect, or halo error, is a complex cognitive process of global attitudes and individual evaluations of a rater. Characteristics of the rating situation, the ratee, and the rater are all influential.^35,36 The effect typically occurs for individuals’ general impressions of people, such as the impression that a person with a slovenly appearance will also be slovenly in their work. Halo effects can also occur for abstract attributes, such as might be evaluated during a Delphi process.²⁵ Hierarchies are a fundamental aspect of social organization and status differentials exist in almost every human interaction.²⁸ Hierarchies can be real or perceived and, in either case, can be influential.^29–31 Hierarchies can support dominant individuals where those holding a position of power and/or having a Social Dominance Orientation to maintain group-based hierarchies³⁷ tend to impose their opinions on others, although hierarchies are not necessary for dominant individuals to be influential.³⁸

The extent to which people engage in discussion or interaction in a group context is contingent on group characteristics, as well as individual psychological characteristics. Removing barriers to expression does not immediately engender the ability to speak up, rather the right social and attitudinal frameworks are necessary.²⁸ While the Delphi process is well designed to avoid the influence of hierarchies and dominant individuals, it is possible during in-person interactions such as the NGT, these influences will arise. Awareness of group dynamics and their potential influence on the hybrid Delphi and NGT approaches allows for strategies to mitigate negative effects.

Methods

We used a qualitative single-case study design to gather multiple data sources including observation notes, audio recordings, individual panelist reflections and post-event facilitator reflections, to build a deeper understanding of the group process during the in-person phase of a modified-hybrid Delphi process.³⁹ The design supports the collection of both narrative and observational data for the in-person phase, shining a light on psychological phenomena and revealing rarely exposed group dynamics within the consensus process. This study received ethics approval from the Health Research Ethics Board at the University of Alberta (Study approval #: Pro00091041). Participants provided written consent.

Case description

The purpose of the modified-hybrid Delphi study (the case) was to classify interprofessional competencies that novice, intermediate and advanced health professional students need to know and/or demonstrate in the placement setting.¹⁸ This modified-hybrid Delphi process^12,13 differed from the typical Delphi process in three primary ways. First, we used an existing reference, the Canadian Interprofessional Health Collaborative (CIHC) Competency Framework,⁴⁰ for the initial survey versus idea generation. The CIHC Competency Framework⁴⁰ offers a shared foundation for educators, researchers, administrators, and care partners guiding interprofessional education and collaborative practice (IPECP) for diverse teams and practice contexts. Second, we included an in-person phase (NGT). Finally, we included senior-level students during the in-person phase. Table 2 outlines the modified-hybrid Delphi phases highlighting modification details for each phase. If a 70% rate of consensus¹⁶ was obtained for the competency domain and descriptor in phase two it was not included in the in-person phase (phase three).¹⁸

Table 2.

Modified-hybrid Delphi.

Phase	Description
1	Online • Participants emailed survey link • Asked to rank level of learner applicable to each Canadian Interprofessional Health Collaborative (CIHC)⁴⁰ competency domain (*CIHC six domains include: Role clarification; patient-, client-, family-, community-centred care; team functioning; collaborative leadership; interprofessional communication; interprofessional conflict resolution) and related descriptors
1	Modified-hybrid modifications • The initial survey was not developed using idea generation • Instead the existing set of interprofessional competency domains and descriptors from the CIHC competency framework⁴⁰ were used • As outlined by Hsu and Sandford¹ when information exists for the relevant construct it is appropriate to use a more structured starting point for the process
2	Online • Participants emailed survey link • Asked for agreement for ranking level of learner and at what level it was important to know and demonstrate the competency • Approximately 50% of the competency domains and descriptors achieved consensus (more than 70% agreement) by the end of this phase (see King et al., 2024⁴⁰ for more details about competency consensus results)
2	Modified-hybrid modifications • None
3	Nominal group technique (NGT), in-person • Facilitators guided the process at each table • Discussed each competency item that did not achieve consensus during previous phases • Voted at the table on which level of learner the competency descriptors aligned by each panelist holding up a labeled voting card for either novice, intermediate and advanced for each competency item (see voting card image in left column) • If reached consensus (more than 70% agreement), moved on to the next competency descriptor • If consensus is not reached after the first vote, an intentional step where participants discussed and explained their responses to attempt to come to consensus occurred • After discussion, a second vote taken • If no consensus after the second vote, participants could write comments/rationale on a provided reflection document
	Modified-hybrid modifications• As the first two phases progressed it was also apparent the in- person phase would be needed, as consensus was only reached for approximately 50% of the competency domains and descriptors (see King et al., 2024¹⁸ for more details)• Inclusion of senior level students at the in-person sessions as panelists and as table-facilitators was essential as they are one of the intended user groups of the guidelines• Inclusion of a second facilitator was essential to support the process including recording voting results and timekeeping (for this project the researcher-facilitators doubled as table co- facilitators)

^aThe CIHC 2010 framework was used in this project and those competency domains are used when referring to contextual aspects of the study. The process of the NGT is the focus here rather than CIHC competencies. An updated CIHC 2024 framework⁴⁰ is provided in the reference list.

Phase three in-person phase (nominal group technique)

An in-person round was planned as part of the modified-hybrid Delphi process to gain better buy-in from the key stakeholders (panelists) for the final guidelines document than could be achieved through online surveys alone. The invited panelists were the intended end users of the guidelines, and the in-person round would provide a sense of ownership for the users due to their role in the creation of the guidelines.¹⁰ The collaborative practice community is familiar with collaborative approaches, so an in-person round was beneficial. Any differences in professional perceptions about the competencies leading to lack of consensus (less than 70% agreement) could be better addressed through an in-person discussion than through further surveys.

The in-person phase included three groups with five panelists in two groups and six in the third group. Two facilitators were assigned to each group. One facilitator’s primary role (table-facilitator) was leading the table through the process while the other facilitator (researcher-facilitator) kept track of time, recorded votes, observed and took notes. The three table-facilitators were nursing after-degree students each assigned to different tables. Two of the researcher-facilitators were doctoral students (one with a health professional background, the other a psychology background) and one was an interprofessional educator from the university.

There were four rounds of discussion for six competency domains, descriptors and related Knowledge, Skills, and Attitudes (KSAs) in each round. The NGT started with a timed discussion of a specific competency domain. At the end of this time, the first round of voting occurred to classify the target KSAs for each competency descriptor as being novice, intermediate, or advanced level of learner. If consensus was achieved, the facilitator started discussion of the next competency descriptor. If consensus was not reached with the voting, planned discussion occurred, and a second round of voting was conducted. After two rounds if consensus was not achieved participants with a dissenting vote expressed their position in writing and the round was concluded with no consensus. Consensus was desirable, however divergent perspectives and robust discussions were considered equally desirable to consensus or unanimity.^41,42 A draft Interprofessional Competency Guide for Practicums was created by the end of the NGT.¹⁸

Panelists

Through purposive sampling, we obtained panelists representing a diverse range of health professions from post-secondary institutions and practice settings based on their ability to inform the study. The panelists represented over 10 different health professions and included educators (n = 2), clinical coordinators (n = 7), preceptors (n = 4), and students (n = 4). Panelists were intentionally grouped to represent diversity of health backgrounds and roles.

During the in-person phase, panelists shared their perspectives providing an opportunity for researcher observations during the group process. The table-facilitators also participated in the voting. The in-person phase lasted 3 hours.

The inclusion of senior-level students at the in-person sessions as panelists and as table-facilitators was essential as they are one of the intended user groups of the guidelines.

Gathering their perspectives would help inform the expectations and language of the final competency guidelines. One student participated as only a panelist. The three after-degree nursing students participating as table-facilitators were completing a leadership placement with the unit conducting the process during this period. These three senior students were part of the planning for the in-person session, and the process required table-facilitators who were well-informed and trained in the process.

Preparation for the in-person phase

The in-person interactions and lack of anonymity among participants (many have an existing working relationship) allowed for the potential interference of social influence and negative group dynamic effects the Delphi process is meant to avoid. Facilitators were well prepared for the NGT process. Before implementing the in-person phase dry runs and practice sessions were held. The facilitators went through the stages of the process and practiced handling possible scenarios that could be detrimental to the process. These scenarios included dominant individuals, non-productive conflict, conversational divergence, and non-participatory individuals. This preparation ensured the facilitators were confident to run the session smoothly and consistently across the three groups. See Table 3 for an overview of the preparation and process for the in-person phase.

Table 3.

Overview of the preparation and process for the in-person (NGT) phase.

Phase 3
	Preparation• A set of guidelines was created following an NGT process including small group facilitator training• Each participant was provided a package of information about the interprofessional competencies, definitions, overview of the complete study, CIHC competency guide, consensus results from the online survey phases, glossary of terms for level of learner (novice, intermediate, advanced) and knowledge, skills, attitudes, an individual reflection worksheet, three voting cards for each level of learner (that would be held up by panelists during voting) and sheets for each discussion round• Several practice sessions and a dry run were conducted with the facilitators to familiarize them with guidelines and discuss possible issues that may arise• The following anticipated behaviors and associated facilitator strategies were integrated into facilitator practice session:∘ dominant individuals - The facilitator redirects the discussion to other individuals and politely limits the time the dominant individual speaks as well as asking the dominant individual to record their thoughts in more detail through writing∘ Non-productive conflict -The facilitator takes control of the conversation to bring it back to the focal topic, reminding the group what the purpose of the process is in general and of the time limit∘ Conversation diverged from the focal topic - The facilitator makes explicit the divergence and asks the group to stay on task∘ Non-participatory individuals/individuals appearing unwilling to share their viewpoints -The facilitator would directly ask these participants for their opinions∘ Inadequate amount of discussion - The facilitator prompts individuals for additional discussion• Small interprofessional table groups were created including panelists with diverse roles such as educators and clinicians (2 tables x 5 panelists, 1 table x 6 panelists).• Two facilitators were assigned per group including a table- facilitator and a researcher-facilitator with each groupProcess∘ Timed group discussion of the interprofessional domains, descriptors and KSAs that had not reached consensus in phase two• A first round of voting where panelists hold up their voting card to identify the level of learner for each outstanding KSA• if consensus was not reached, planned discussion was facilitated• A second round of voting occurred where panelists would again hold up their voting card for the level of learner• After two rounds of discussion and voting, if consensus was not achieved participants with a dissenting vote were given the opportunity to express their position in writing and the round was concluded with no consensus

Phase 3

Preparation• A set of guidelines was created following an NGT process including small group facilitator training• Each participant was provided a package of information about the interprofessional competencies, definitions, overview of the complete study, CIHC competency guide, consensus results from the online survey phases, glossary of terms for level of learner (novice, intermediate, advanced) and knowledge, skills, attitudes, an individual reflection worksheet, three voting cards for each level of learner (that would be held up by panelists during voting) and sheets for each discussion round• Several practice sessions and a dry run were conducted with the facilitators to familiarize them with guidelines and discuss possible issues that may arise• The following anticipated behaviors and associated facilitator strategies were integrated into facilitator practice session:∘ dominant individuals - The facilitator redirects the discussion to other individuals and politely limits the time the dominant individual speaks as well as asking the dominant individual to record their thoughts in more detail through writing∘ Non-productive conflict -The facilitator takes control of the conversation to bring it back to the focal topic, reminding the group what the purpose of the process is in general and of the time limit∘ Conversation diverged from the focal topic - The facilitator makes explicit the divergence and asks the group to stay on task∘ Non-participatory individuals/individuals appearing unwilling to share their viewpoints -The facilitator would directly ask these participants for their opinions∘ Inadequate amount of discussion - The facilitator prompts individuals for additional discussion• Small interprofessional table groups were created including panelists with diverse roles such as educators and clinicians (2 tables x 5 panelists, 1 table x 6 panelists).• Two facilitators were assigned per group including a table- facilitator and a researcher-facilitator with each groupProcess∘ Timed group discussion of the interprofessional domains, descriptors and KSAs that had not reached consensus in phase two• A first round of voting where panelists hold up their voting card to identify the level of learner for each outstanding KSA• if consensus was not reached, planned discussion was facilitated• A second round of voting occurred where panelists would again hold up their voting card for the level of learner• After two rounds of discussion and voting, if consensus was not achieved participants with a dissenting vote were given the opportunity to express their position in writing and the round was concluded with no consensus

Data collection

Data was collected during the in-person session in four ways. First, observations of the group process were recorded as field notes by researcher-facilitators embedded within each small group with the goal of observing social phenomena.^43,44 The researcher-facilitators were non-voting members of the group tasked with time keeping, recording votes and were also attentive to the group interaction including verbal and non-verbal (nodding, facial expressions, body posture) indicators. The facilitator role was more prominent than the researcher role. An embedded approach allows for the researcher to have an emic or insider view of the process.⁴³ Second, the small group discussions were audio-recorded for later review. Next, panelists were invited to record additional thoughts on an individual reflection sheet which was collected and reviewed after the NGT. Finally, one day after the in-person phase, both table-facilitators and researcher-facilitators independently recorded their observations and thoughts about the in-person session. They were then asked to review their notes and the audio recording for their group and summarize their observations and thoughts as text in a free narrative format.

Facilitators could add to and edit their thoughts and observations about the process for up to a month after the in-person session.

Data analysis

Thematic analysis supported identification, organization, and insights as to patterns or themes across the data set.⁴⁵ A combination of inductive and deductive analysis was used to explore the data. The inductive approach revealed emerging themes about the group process while the deductive coding was attentive to the presence and absence of group dynamics and social influence issues. One researcher (EV) who was also a researcher-facilitator for one of the small groups analyzed the data. The reviewer brings a psychological lens to the analysis with an understanding of group influences. The reviewer used word documents in conjunction with handwritten notes to document the analysis. Initial themes were shared with the study team and facilitators for member checking to ensure accuracy⁴⁶ and allow for deeper exploration from diverse perspectives. The researcher (EV) summarized the discussion and refined the themes before completing another round of member checking to ensure that the facilitators’ perspectives were accurately represented.

Methodological rigor

The researchers took a critical realist approach to examine the three data sources related to the in-person phase. The critical realist approach was used to differentiate between the observable (epistemological) and unobservable (ontological), including social psychological aspects of the observations.⁴⁷ Three study team members participated as researcher-facilitators (MG, EV, CM) with one also completing the data analysis (EV) providing an emic or insider perspective. One study team member (SK) did not actively participate in the table discussions thus providing an etic or outsider perspective during analysis. The study adhered to qualitative measures of quality⁴⁸ including credibility, transferability, dependability and confirmability. Credibility ensures the accuracy and truthful representation of the data.⁴⁹ Triangulation of data methods, observers and the study team supported a robust understanding of the data. Member checking with facilitators further supported accuracy of the interpretations and conclusions from the analysis. Dependability addresses quality aspects of transparency and logical research processes which is addressed with a rich description of the in-person phase context, process, and preparation. Further, a thorough description of the methods coheres with the findings.⁵⁰ Although qualitative findings do not expect generalizability, the in-depth description of both context¹⁸ and process along with purposive sampling supports transferability to other contexts. Confirmability addresses issues of researcher bias or motivation. Reflexivity of the reviewer and multiple rounds of member checking addressed potential researcher bias in the analysis. An audit trail was maintained including raw data such as the observation notes, reflections, and audio recordings and documentation of the analysis process.

Results

Case study results included unique and overlapping information from the facilitators’ observations and select panelist comments. Examining the data across facilitators two themes emerged: Gaining Momentum and Sharing Perspectives including six main observations. Three observations contribute to the Gaining Momentum theme including (1) development of shared understanding of terminology and concepts, (2) a decreasing time to consensus, and (3) perspective taking. The data also revealed how individuals were Sharing Perspectives despite the potential negative influence of group dynamics including (4) a concern for hierarchy, (5) an absence of dominant individuals, and (6) accommodation of others’ perspectives without groupthink. Table 4 summarizes the themes and main observations providing supporting quotes from facilitators and panelists.

Table 4.

Themes, main observations, and supporting quotes.

Theme	Main observations	Supporting quotes
Gaining momentum	1. Development of shared understanding of terminology and concepts	“There was a period of adjusting to both the process of the voting, as well as the language used, including multiple referencing to the glossary terms.” Facilitator for Table 2“Our process was fairly consistent throughout each round, but we relied more on the glossary to guide our decision-making during rounds two and three.” Facilitator for Table 3
	2. A decreasing time to consensus	“Earlier rounds progressed at a slower pace compared to later rounds, but this can be attributed to factors such as the length of round one and the uncertainty around the process of each round.” Facilitator for Table 1“The time spent in each round became lessened, which could be attributed to less clarification about process and terminology.” Facilitator for Table 2
	3. Perspective taking	“In the context of med lab sciences an earlier learner would not be at that level.” Panelist Notes“The group was not afraid to engage in respectful disagreement and appeared to be genuinely interested in everyone’s perspectives.” Facilitator for Table 3
Sharing perspectives	4. A concern for hierarchy	“Having the three perspectives at the table (preceptor, educator/clinical coordinator, student) was critical to the process” Paraphrased from Facilitators for Tables 1, 2, and 3“Also, there were no overly dominant personalities at the table. I found this supported the consensus building process as no individual(s) dominated the group discussions or put unnecessary pressure on others to change their votes.” Facilitator for Table 3
	5. An absence of dominant individuals	“Also, there were no overly dominant personalities at the table. I found this supported the consensus building process as no individual(s) dominated the group discussions or put unnecessary pressure on others to change their votes.” Facilitator for Table 3“Occasionally, group members initially believed that a KSA belonged to a particular level but changed their answer after reviewing the glossary.” Facilitator for Table 3
	6. Accommodation of others’ perspectives without groupthink	“I do not think that groupthink was a factor during the meeting because although consensus was reached more readily, participants would voice their reason as to why they chose to change their vote or not.” Facilitator for Table 1

Gaining Momentum

As the groups worked together their shared understanding of topic, process and perspectives allowed for more efficient moving forward as the NGT progressed. All facilitators noted the panelists had, and took, the opportunity to voice differing opinions throughout the NGT process. This ensured that diverse perspectives were included across students, preceptors, clinical coordinators and educators despite time limits. Based on panelist discussions, facilitators recognized that panelists had differing understandings of terminology and concepts, despite a plenary address that outlined the NGT terminology and concepts. Given these differences, small groups relied on the glossary of terms to support their discussions and decision-making.

At the initiation of the session panelists took a longer time to discuss the competency domains, descriptors, and KSAs than at the end of the session. The more rapid consensus was noted for all three groups. Facilitators felt this decreasing time to consensus resulted from increased common understanding of the topic and the NGT process. As the format of discussion and voting repeated, facilitators believed that panelists’ familiarity with the process improved efficiency.

Finally, discussions in the initial voting rounds revealed rationale and frames of reference for panelist decisions. As each round progressed, the facilitators noticed that panelists would transfer learning from previous discussions to subsequent voting rounds to inform their vote. Panelists would comment, “Based on what was discussed in the previous round, I would vote in this way…”. This momentum of understanding from previous rounds did not appear to facilitators to be a result of group pressure to conform, yet an acknowledgment and at times acceptance of agreement with others’ rationale and perspectives for decision making. Despite these shared frames of reference, consensus was not always attained. Debate occurred for almost all the domains, descriptors, and KSAs discussed and participants frequently sought clarification of other’s opinions and expressed disagreement.

Sharing Perspectives

By including multiple levels of professionals in the NGT process it was not possible to avoid hierarchical influences. The anonymity in the Delphi process intentionally avoids the influence of hierarchies, yet it was necessary to have an in-person aspect to the study. Preceptors, clinical coordinators, educators, and students were members of all three tables. The facilitators noted “Having the three perspectives at the table (preceptor, educator/clinical coordinator, student) was critical to the process”, however there was potential detrimental influence of real or perceived hierarchy during the process. One facilitator reflected post-event:

During the debrief, the student-facilitators expressed that perceived hierarchy impacted their decision making. For example, when faculty from their programs had a different perspective of when a student should know or be able to do a competency, the students felt discomfort and concern that their answers may impact their professional image. Being aware of the real and perceived power differential between stakeholders, particularly students and faculty/instructors would be an important consideration for future focus groups.

Facilitator for Table 3

As the purpose of the session was to differentiate what KSAs were appropriate for different levels of learners, the student-facilitators were concerned about whether their opinions and the way they voted represented an adequate level of knowledge and ability regarding the KSAs they were evaluating. Hierarchies, and the authority of others, can be a very powerful influence on people’s behaviours and cognitions and the influence of hierarchies functioned almost exactly as predicted in the literature.^51,52 Interestingly, student-facilitators expressed a sense of discomfort and second guessing of their opinions but not of altering their voting to comply with those perceived as being in a higher position. The differences between students and other experts at the table influenced students’ cognitions and emotions but not their behaviour or voting as shared in the post-event facilitator reflections. Despite these statements, student perspectives may still have been swayed to some extent by the influence of the hierarchies created by the different groups at the table, whether the influence was real or perceived.

Facilitators noted that it did not appear that any member of the group emerged as a dominant voice or as a leader. As noted earlier, reference to the glossary could alter participants’ decisions. Some participants frequently directed the group back to the glossary of terms provided or were aware of the time limits set out for each discussion, however, these participants functioned more as an ‘assistant facilitator’ than trying to guide or dominate the group’s decisions.

Due to the in-person nature of the NGT, facilitators were sensitive to the potential for negative group dynamics occurring and noted that, outside of the influence of hierarchy, it did not appear that any observable negative group dynamics occurred during the NGT phase. An absence of groupthink and dominant personalities allowed for consideration of others’ perspectives while minimizing social influence. There was a time limit set for discussions, however, the discussions were frequently concluded before the time limit. As the NGT progressed, consensus was more frequently reached in the first round of voting. Despite the shorter discussions and more rapid consensus the facilitators noted that it did not appear the group was reaching premature closure or that participants felt uncomfortable voicing differing opinions. Facilitators and some panelists were aware of the potential influence of groupthink. In one group a panelist joked about groupthink occurring. There was potential that groupthink could occur as participants moved to consensus in less time, however achieving consensus may have occurred more quickly through perspective taking as discussed earlier. Other aspects of groupthink such as self-censorship, unanimity, and conformity to maintain group cohesion²³ were distinctly absent as panelists engaged in open discussion and debate and consensus was not always achieved.

Panelists did occasionally change their vote to achieve consensus but explicitly stated so. The reasons stated for changing votes do not indicate groupthink or a bandwagon effect. Rather than being influenced by social pressures panelists would change their vote because they felt their opinion fell between levels of learners, e.g. Novice-Intermediate, and they did not have a strong opinion on the domain, descriptor, or KSA. Panelists would also indicate that it was not particularly important for their profession to differentiate the precise level of the domain, descriptor, or KSA, or that they felt the KSA and level of learner was fluid enough that it could belong to either group. Additionally, in these cases panelists only changed their vote to an adjacent response category, e.g. between Novice and Intermediate or between Intermediate and Advanced. Panelists never changed between Novice and Advanced.

Beyond the absence of groupthink and dominant personalities, there was also an absence of non-productive conflict amongst the group. Debate occurred and panelists challenged each other’s ideas and perspectives, however, respect and collegiality were maintained throughout the process. There was no apparent judgment of conflict over final decisions regarding the domains, descriptors, and KSAs. Table 5 provides a summary of how group dynamics and facilitator strategies influenced each step of decision-making during the in-person NGT phase, highlighting both positive and negative observed and potential factors along with their outcomes.

Table 5.

Decision influence matrix for in-person (NGT) phase.

Phase Three Steps	Decision- making Step	Positive influences	Negative influences	Facilitator Strategies	Outcome
1	Initial discussion of competency descriptors and KSAs	Diverse perspectives	Confusion on terminologyPotential for halo effect	Redirect to glossary for shared definitions	Clarified terms
2	First non-anonymous vote	Perspective- taking	Emerging or perceived hierarchyPotential for bandwagon effect	Invite student voice and overall individual sharing	Consensus reached or not
3	Second discussion (if no consensus)	Clarification and shared understanding	Time pressuresPotential for hierarchy	Clarified concepts and managed timeInvite quieter voices	Improved understanding
4	Second non-anonymous vote	Respectful debate	Potential hierarchy and groupthink	Time monitoringDissent as acceptable	Consensus reached or not
5	Post-vote reflection	Opportunity for individual reflection	Potential self-censoring	Encourages written reflection providing privacy	Dissenting views documented

Discussion

A case study was conducted to determine the influence of group dynamics on the final phase of a modified-hybrid Delphi process held in-person utilizing NGT. The most notable aspect of the group dynamics of the NGT phase was that despite the lack of anonymity, open voting, and direct interpersonal interactions, outside of the influence of hierarchies, we did not observe the expected negative aspects of group dynamics. Although, without interviewing the panelists we cannot be certain about their cognitions and emotions experienced during the NGT. The use of the modified-hybrid Delphi process was able to meet the study goals without succumbing to potential biasing influences. The present study indicates it is possible to meet stakeholder and study needs and successfully achieve the study goals through using an in-person stage in consensus building. The researchers revealed key learning about the group dynamics and identified contributing factors that likely reduced the influence of negative group dynamics including thorough preparation and attention to processes for the NGT phase.

Hierarchy

The only apparent negative influence of group dynamics during the NGT phase was a hierarchical difference between the student participants and faculty panelists. Senior student-facilitators who also participated in the voting round, debriefed with the researchers after the NGT. The student-facilitators shared that although they felt the influence of the hierarchical difference, it did not seem as though the faculty were attempting to exert any influence on them or were acting judgmentally. Hierarchical behavior such as unequal speaking time or self-censorship was not noted by the second facilitator-researchers at the tables. Student-facilitators did not think their voting was influenced by the senior faculty members at their group. These students (3 of 4 students) were involved in planning the event and trained to facilitate the small groups including mitigating group dynamics which may have impacted their ability to avoid being influenced to change their vote.

In an in-person session with participants at various levels of a professional experience, it is likely impossible to avoid any real or perceived hierarchical influence. Anonymous voting and avoiding feeding back information about individual choices, judgments, and reasoning can be used as well and is generally recommended for the NGT process.^13,15 In certain contexts, such as in the present study, anonymous voting and feedback may not be desirable as it would inhibit cross-disciplinary discussion and understanding when these interactions are seen to be desirable. In future studies, the existence of these influences could be addressed prior to the NGT or during the initial introduction where it is made explicit that differences in professional levels should not influence voting. It should be noted though that the influence of hierarchies can be very powerful, which may not be sufficient for avoiding their influence.

The expert panelists were individuals who were advocates for Interprofessional Education and Collaborative Practice (IPECP) presumably experienced in group interaction and communication including how to express and accept divergent options and the necessity for contributions from different team members. The commitment by the participants to IPECP may have contributed to the collegiality during the NGT, but this cannot be assumed. Setting the scene and acknowledging the power of hierarchy and other group influences is advisable with any NGT process. Relatedly, the nature of collaboration of diverse professions meant that the domains, descriptors, and KSAs were highly contextual, and participants discussed this and felt comfortable disagreeing based on how the domain, descriptors or KSA would apply to their profession and context. This diversity of professions and context likely helped prevent groupthink as it was important and necessary to express how the domains, descriptors, and KSAs functioned for diverse learners.

Preparation for the NGT

Anticipation of the potential influence of negative group dynamics and the inclusion of three senior student-facilitators prompted the study team to be thorough and methodical in preparation for the NGT phase. This attention to detail likely resulted in less influence from group dynamics. Preparing facilitators for the NGT emerged as a key factor in meticulously outlining processes and mitigating potentially negative group behaviors and influences.

Originally the study team looked for experts in the IPECP field to support small group processes. Failing to find specified facilitators, senior nursing students completing their leadership practicum with the interprofessional unit and already supporting the NGT phase were recruited. To support the senior students, facilitator training included numerous practice sessions in which group behaviors were simulated and the voting process was refined. The additional attention paid to facilitator training and anticipating group dynamics may not have occurred if experts in the field were recruited. This critical step helped the study team think through the small group process more intentionally and emerged as a pivotal success factor for the NGT. One factor we failed to fully attend to yet warranted further attention involved the potential for real or perceived hierarchy between the student-facilitators and faculty members from their profession.

Bridging preparation and process for the NGT, resources were created and provided as a participant package to each panelist. This package connected the previous phases with the NGT phase including an update from phases one and two (agreed upon domains and descriptors document) and clarification of the overview purpose (study overview visual). Additional resources supported shared understanding of concepts and terminology (Quick Guide and Glossary for Participants) and process (round sheets, voting cards with Level of Learner).

Panelists also received an individual reflection sheet to capture additional thoughts for later review by the study team, including when their vote did not align with the majority which may have fostered a sense of being heard. The participant package was instrumental in supporting the group process during the NGT.

Process during the NGT

The NGT included a plenary address and introduction to set the collaborative tone, focus on shared expectations and clearly established the roles of panelists and facilitators. Survey study results from the modified Delphi phases were shared with participants. The purpose of the study was recapitulated and questions addressed. It was also made clear that though the overarching purpose was to develop consensus (more than 70% agreement) dissenting opinions were not only welcomed and encouraged but it was acceptable not to reach consensus. Participants were also informed that if they disagreed with the consensus vote they were encouraged to explain their position in writing on their individual reflection sheet to be incorporated in the ultimate analysis. Finally, a practice round for the NGT was conducted as a large group so that all participants knew how the session would be conducted.

The process was run as a half-day session. The time limit for the entire session and time limits set for each discussion of the domains, descriptors and KSAs helped to keep the discussions on track and focused. Facilitators ensured the panelists were aware of the time limit, likely limiting the opportunity for unproductive conversation or for a single individual to dominate the conversation. The time limit was also useful in maintaining the structure of the NGT. The time limit naturally limited the extent of the discussion that could occur and though this may have reduced the ideas being shared the limits placed on potential negative group dynamics likely outweighed the cost.

The modified-hybrid Delphi process for developing Interprofessional Practicum Competencies was successful in fulfilling the intended purpose. The process avoided potential negative group dynamics and social influence through preparation for potential negative interactions, tone setting, time constraints, and including panelists with a vested interest in the outcome. Additionally, several recommendations laid out by Bolger and Wright¹⁵ were met it was not assumed the process could eliminate all social influence, rich qualitative feedback in the form of reasons and justifications were elicited between panelists, the panelists were highly motivated and invested stakeholders in the process and outcome, and finally the panelists selected were real experts, rather than “social experts”, at each level of health professional education.

Lessons learned and consideration of future implementation

Implementing a modified-hybrid Delphi process with an in-person NGT phase required significant preparation and resources. Key lessons learned include:

• Preparation is time-intensive: Allocate enough time to prepare. Approximately 2 weeks were needed to develop materials, train facilitators, and finalize processes.

• Facilitator availability and training: Ensure trained facilitators are available to support consistent and effective execution of the NGT. Provide facilitator training for all facilitators, including practice sessions and dry runs, to anticipate and mitigate potential group dynamic challenges.

• Operational Support Needs: Plan for operational requirements to support an in-person NGT, including organizing spaces, technology, materials, communication and logistics, before and after the session including any follow reports.

• Content complexity matters: Recognize that highly detailed or contested topics increase discussion time and require clear time management and strategies to address group dynamics.

• Time management is critical: Set clear time limits for the overall NGT and individual round to help maintain structure, focus discussions, and prevent dominance by a single participant. Time constraints should balance rich discussion while minimizing negative group dynamics.

• Participant Mix: Anticipate that diversity of roles adds value but also complexity requiring intentional structures and facilitation to balance perspectives.

More practical guidance is provided in Table 6 to support specific preparation and process tips.

Table 6.

Preparation and process tips for an in-person (NGT) phase.

Preparation tips

• Plan a highly structured method following a NGT controlled by two facilitators with each small group.• An awareness of the potential negative group dynamic influences on the process is also important. How these negative influences function and manifest can allow for planning to mitigate them before they occur and develop methods to deal with negative group dynamics if they develop.• Having smaller groups with a high panelist to facilitator ratio can help to maintain the process and distribute the workload of the facilitators to ensure the process is carried out smoothly.• Holding several practice sessions beforehand with all members of the participating research team was highly beneficial. The practice sessions identified issues with the planned process, allowed for practicing difficult situations and planning contingencies, ensured familiarity with the process among all team members, and created confidence amongst the team.• Careful preparation and practice were possibly the most important components for success.

Process tips

• Include an introductory address to outline the purpose of the NGT and the process the panelists will participate in.• During the plenary address, thank participants thus far and invite open discussion about the study to support buy-in from panelists.• Indicate to panelists that dissension and differing perspectives are welcome and desired. Consensus should be identified as being desirable but should not be held to be mandatory. Making consensus mandatory could function as a form of social dominance from the research team and create groupthink through premature closure and a need to behave according to the social norm of consensus established by the research team.• Provide a participant package to support shared understanding and reference points as well as resources to support a clear voting process. For example, providing participants with a glossary is useful for ensuring a common understanding of fundamental terms and ideas. Having an established reference can help to avoid unproductive discussion and conflict or allow any individual’s understanding of terms and ideas to become dominant among the group.• Having a practice round of discussion and voting prior to the actual rounds can aid in panelist comfort and confidence.• Provide voting results throughout the process to help panelists feel comfortable and confident in the process.• Provide refreshments and lunch if possible. Refreshments and lunch can help to maintain cognitive engagement by reducing fatigue and giving the panelists a natural break period to disengage from the task and return with a renewed focus. Providing refreshments and lunch can also indicate to panelists that they are valued by the research team.

Limitations

There were three main limitations to the study. The first limitation of the study was also one of the identified strengths. All participants were advocates and active in IPECP and likely represented an ideal sample for conducting the NGT aspect of the modified-hybrid Delphi. The participants are experts in teamwork, constructive conflict, and communication, providing a potentially idealized sample for optimal group function, especially within the structure of the NGT. With different samples in different contexts it is possible more of the negative influences would have been apparent.

The second limitation was the lack of physicians in the sample. Physicians are important members of interprofessional teams and their input on developing the competencies was missed. From a process perspective the presence of physicians could have also altered the group dynamics. Physicians tend to have a strong sense of autonomy,⁵³ and with existing hierarchical differences in health care^54,55 the group dynamics may have been altered.

The final limitation was the non-anonymous voting. The rationale and issues of using non-anonymous voting is discussed above and it was determined to be unlikely the non-anonymous voting negatively influenced the outcome of the study. Despite this there is still the potential for some social influence due to the voting and feedback methods used.

Conclusion

Though there were minor concerns such as hierarchies, the modified-hybrid Delphi process was successful and most negative group dynamic issues discussed in the literature were mitigated. The Nominal Group Technique (NGT) created satisfaction among participants and met stakeholders’ needs. The existing literature on the actual function of group dynamics and social influence during an NGT or hybrid-Delphi process is sparse. The present study helps to grow the literature on the psychology of group dynamics during an in-person consensus building process and provides practical recommendations for conducting a successful in-person session. Further research into the psychology of social influence and group dynamics during an in-person consensus building process should be conducted including exploration of different contexts and content areas other than interprofessional education and collaborative practice that may not be as supportive of a collaborative interaction.

Footnotes

ORCID iDs

Melanie Garrison

Sharla King

Efrem Mauro Violato

Charlotte Betty McCartan

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Hsu

Sandford

. The Delphi technique: making sense of consensus. PARE 2007; 12(10): 1–8.

Sinha

Saracho

. The Delphi process. In Saracho

(Ed.) Handbook of research methods in early childhood education. Review of research methodologies. Volume I. IAP: Information Age Processing, 2014.

Dalkey

Helmer

. An experimental application of the DELPHI method to the use of experts. Manag Sci 1963; 9(3): 458–467. https://doi.org/10.1287/mnsc.9.3.458

McMillan

King

Tully

. How to use the nominal group and Delphi techniques. Int J Clin Pharm 2016; 38(3): 655–662. https://doi.org/10.1007/s11096-016-0257-x

Delbecq

Van de Ven

Gustafson

. Guidelines for conducting NGT meetings. In: Group Techniques for Program Planning: A Guide to Nominal Groups and Delphi Process: Scott Foresman Company, 1975, pp. 40–82.

Paré

Cameron

Poba-Nzaou

, et al. A systematic assessment of rigor in information systems ranking-type Delphi studies. Inf Manag 2013; 50(5): 207–217. https://doi.org/10.1016/j.im.2013.03.003

Iqbal

Pipon-Young

. The Delphi method. Psychol 2019; 22(7): 598–601.

Mullen

. Delphi: myths and reality. J Health Organisat Manag 2003; 17(1): 37–52. https://doi.org/10.1108/14777260310469319

Linstone

Turoff

. The Delphi survey: method techniques and applications: Addison-Wesley, 1975.

10.

Van De Ven

Delbecq

. The effectiveness of nominal, Delphi, and interacting group decision making processes. Acad Manag J 1974; 17(4): 605–621. https://doi.org/10.5465/255641

11.

Davies

Romano

Schmidt

, et al. Assessment of a novel hybrid Delphi and nominal groups technique to evaluate quality indicators. J Health Sci.Res 2011; 46(6pt1): 2005–2018. https://doi.org/10.1111/j.1475-6773.2011.01297.x

12.

Landeta

Barrutia

Lertxundi

. Hybrid Delphi: a methodology to facilitate contribution from experts in professional contexts. Technol Forecast Soc Change 2011; 78(9): 1629–1641. https://doi.org/10.1016/j.techfore.2011.03.009

13.

Fitch

Bernstein

Aguilar

, et al. The RAND/UCLA appropriateness method user’s manual: RAND, 2001.

14.

Tasca

. What is group dynamics? Group Dyn: Theory, Research, and Practice 2022; 24(1): 1–5. https://doi.org/10.1037/gdn0000115

15.

Bolger

Wright

. Improving the Delphi process: lessons from social psychological research. Technol Forecast Soc Change 2011; 78(9): 1500–1513. https://doi.org/10.1016/j.techfore.2011.07.007

16.

Humphrey-Murto

Varpio

Wood

, et al. The use of the Delphi and other consensus group methods in medical education research: a review. Acad Med 2017; 92(10): 1491–1498. https://doi.org/10.1097/ACM.0000000000001812

17.

Winkler

Moser

. Biases in future-oriented Delphi studies: a cognitive perspective. Technol Forecast Soc Change 2016; 105: 63–76. https://doi.org/10.1016/j.techfore.2016.01.021

18.

King

Garrison

Violetta

, et al. Getting everyone on the same page: assessing interprofessional competencies during student placements. J Interprof Educ Pract 2024; 37: 1–10. https://doi.org/10.1016/j.xjep.2024.100728

19.

Humphrey-Murto

Varpio

Gonsalves

, et al. Using consensus group methods such as Delphi and nominal group in medical education research. Med Teach 2017; 39(1): 14–19. https://doi.org/10.1080/0142159X.2017.1245856

20.

Barrett

Heale

. What are Delphi studies? Evid Base Nurs 2020; 23(3): 68–69. https://doi.org/10.1136/EBNURS-2020-103303

21.

Dalkey

. An experimental study of group opinion: the Delphi method. Futures 1969; 1(5): 408–426. https://doi.org/10.1016/S0016-3287(69)80025-X

22.

Daniels

. Matter of opinion: the Delphi method in the social sciences. Sage Research Methods Cases, 2017. https://doi.org/10.4135/9781526410603

23.

Janis

. Groupthink. Psychol Today 1971: 84–90.

24.

Kramer

Dougherty

. Groupthink as communication process, not outcome. Comm Soc Chang 2013; 1(1): 44–62. https://doi.org/10.4471/CSC.2013.03

25.

Feeley

. Comment on halo effects in rating and evaluation research. Hum Commun Res 2002; 28(4): 578–586. https://doi.org/10.1093/hcr/28.4.578

26.

Goodman

. The Delphi technique: a critique. J Adv Nurs 1987; 12(6): 729–734. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x

27.

Jacob

Duffield

Jacob

. Development of an Australian nursing critical thinking tool using a Delphi process. J Adv Nurs 2018; 74(9): 2241–2247. https://doi.org/10.1111/jan.13732

28.

Islam

Zyphur

. Power, voice, and hierarchy: exploring the antecedents of speaking up in groups. Group Dyn 2005; 9(2): 93–103. https://doi.org/10.1037/1089-2699.9.2.93

29.

Calhoun

Boone

Porter

, et al. Using simulation to address hierarchy-related errors in medical practice. Perm J. Spring 2014; 18(2): 14–20. https://doi.org/10.7812/TPP/13-124

30.

Green

Oeppen

Smith

, et al. Challenging hierarchy in healthcare teams - ways to flatten gradients to improve teamwork and patient care. Br J Oral Maxillofac Surg 2017; 55(5): 449–453. https://doi.org/10.1016/j.bjoms.2017.02.010

31.

Weiss

Kolbe

Grote

, et al. Agency and communion predict speaking up in acute care teams. Small Group Res 2014; 45(3): 290–313. https://doi.org/10.1177/1046496414531495

32.

Janis

. Victims of groupthink: a psychological study of foreign-policy decisions and fiascoes: Houghton Mifflin, 1972.

33.

Turner

Pratkanis

Turner

. Twenty-five years of groupthink theory and research: lessons from the evaluation of a theory. Organ Behav Hum Decis Process 1998; 73(3): 105–115. https://doi.org/10.1006/obhd.1998.2756

34.

Tsikerdekis

. The effects of perceived anonymity and anonymity states on conformity and groupthink in online communities: a wikipedia study. J Assoc Inf Sci Technol 2013; 64(5): 1001–1015. https://doi.org/10.1002/asi.22795

35.

Thorndike

. A constant error in psychological ratings. J Appl Psychol 1912; 4(1): 25–29. https://doi.org/10.1037/H0071663

36.

Wen

Georgiou

, et al. Reducing the halo effect by stimulating analytic thinking. Soc Psychol 2020; 51(5): 334–340. https://doi.org/10.1027/1864-9335/A000418

37.

Sidanius

Pratto

, et al. Social dominance orientation: revisiting the structure and function of a variable predicting social and political attitudes. Pers Soc Psychol Bull 2012; 38(5): 583–606. https://doi.org/10.1177/0146167211432765

38.

Tesi

Pratto

Pierro

, et al. Group dominance in hierarchy-attenuating and hierarchy-enhancing organizations: the role of social dominance orientation, need for cognitive closure, and power tactics in a person–environment (Mis)Fit perspective. Group Dyn: Theory, Research, and Practice 2020; 24(2): 102–114. https://doi.org/10.1037/gdn0000117

39.

Range

. Case study methodologies. Encyclopedia of health. Salem Press, 2021.

40.

Canadian Interprofessional Health Collaborative . CIHC competency framework for advancing collaboration, 2024. https://cihc-cpis.com/wp-content/uploads/2024/06/CIHC-Competency-Framework.pdf (Accessed 29 October 2025).

41.

Rowe

Wright

McColl

. Judgment change during Delphi-like procedures: the role of majority influence, expertise, and confidence. Technol Forecast Soc Change 2005; 72(4): 377–399. https://doi.org/10.1016/j.techfore.2004.03.004

42.

Woudenberg

. An evaluation of Delphi. Technol Forecast Soc Change 1991; 40(2): 131–150. https://doi.org/10.1016/0040-1625(91)90002-w

43.

Creswell

. 30 essential skills for the qualitative researcher: Sage, 2016.

44.

Hatch

. Doing qualitative research in education settings: SUNY Press, 2002.

45.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3(2): 77–101. https://doi.org/10.1191/1478088706qp063oa

46.

Birt

Scott

Cavers

, et al.

Member checking: a tool to enhance trustworthiness or merely a nod to validation?

Qual Health Res 2016; 26(13): 1802–1811. https://doi.org/10.1177/1049732316654870

47.

Willig

. What can qualitative psychology contribute to psychological knowledge? Psychol Methods 2019; 24(6): 796–804. https://doi.org/10.1037/met0000218

48.

Lincoln

Guba

. Naturalistic inquiry: Sage Publications, 1985.

49.

Cypress

. Rigor or reliability and validity in qualitative research: perspectives, strategies, reconceptualization, and recommendations. Dimens Crit Care Nurs 2017; 36(4): 253–263. https://doi.org/10.1097/DCC.0000000000000253

50.

Hanson

Tong

. Appraisal of qualitative studies. In: Liamputtong

(ed). Handbook of Research Methods in Health Social Sciences: Springer Nature Pte, 2019, pp. 1013–1026. https://doi.org/10.1007/978-981-10-5251-4_119

51.

Doliński

Grzyb

Folwarczny

, et al. Would you deliver an electric shock in 2015? Obedience in the experimental paradigm developed by stanley milgram in the 50 years following the original studies. Soc Psychol Personal Sci 2017; 8(8): 927–933. https://doi.org/10.1177/1948550617693060

52.

Vanstone

Grierson

. Medical student strategies for actively negotiating hierarchy in the clinical environment. Med Educ (Oxf) 2019; 53(10): 1013–1024. https://doi.org/10.1111/medu.13945

53.

Holmes

Harris

Schwartz

, et al. Harnessing the hidden curriculum: a four-step approach to developing and reinforcing reflective competencies in medical clinical clerkship. Adv Health Sci Educ 2014; 20(5): 1355–1370. https://doi.org/10.1007/s10459-014-9558-9

54.

Milligan

Wareing

Preston-Shoot

, et al. Supporting nursing, midwifery and allied health professional students to raise concerns with the quality of care: a review of the research literature. Nurse Educ Today 2017; 57: 29–39. https://doi.org/10.1016/j.nedt.2017.06.006

55.

Violato

. A state-of-the-art review of speaking up in healthcare. Adv Health Sci Educ Theory Pract 2022; 27(4): 1177–1194. https://doi.org/10.1007/s10459-022-10124-8