Measuring shared mental models in healthcare

Abstract

Objective

To review common qualitative and quantitative methods of measuring shared mental models appropriate for use in the healthcare setting.

Background

Shared mental models are the overlap of individuals’ set of knowledge and/or assumptions that act as the basis for understanding and decision making between individuals. Within healthcare, shared mental models facilitate effective teamwork and theorized to influence clinical decision making and performance. With the current rapid growth and expansion of healthcare teams, it is critical that we understand and correctly use shared mental model measurement methods assess optimal team performance. Unfortunately, agreement on the proper measurement of shared mental models within healthcare remains diffuse.

Method

This paper presents methods appropriate to measure shared mental models within healthcare.

Results

Multiple shared mental model measurement methods are discussed with regard to their utility within this setting, ease of use, and difficulties in deploying within the healthcare operational environment. For rigorous analysis of shared mental models, it is recommended that a combination of qualitative and quantitative analyses be employed.

Conclusion

There are multitude of shared mental model measurement methods that can be used in the healthcare domain; although there is no perfect solution for every situation. Researchers can utilize this article to determine the best approach for their needs.

Keywords

Shared mental models assessment healthcare mental models teams or teamwork

Introduction

Defining shared mental models

Mental models are an organized set of knowledge and/or assumptions about reality that give individuals a basis for decision making and understanding.¹ Day et al.² demonstrated that when individuals’ mental models were more similar to experts’ mental models, it was predictive of skill performance. Shared mental models (SMMs) are the overlap or co-understanding between two or more individuals.³ SMMs describe the ability for all members of a team to be thinking in a similar way and performing in accordance with each other. More specifically, team members can have SMMs surrounding tasks, technology, team interactions, and team members.⁴ See Table 1 for examples and definitions of different types of SMMs.

Table 1.

Organization of shared mental models within the context of healthcare teams (adapted from Mathieu et al.³).

Types of SMMs	Definition	Example
Technology	Knowledge of the functional properties and limits of the technology involved in the work, including but not limited to the electronic medical record, ordering systems for drugs or procedures, surgical devices and tools, and patient monitoring equipment	Everyone is knowledgeable of the steps taken to input information into the electronic medical record and how to retrieve that information again.
Tasks	Knowledge of the to-do list of orders, procedures, or treatments that must be delivered to a patient before, during, or after the providers current shift, as well as potential strategies, pitfalls, and contingency plans	Every on the team understands the steps required to place a central line including infection precaution protocols.
Team interaction	Knowledge of the roles and responsibilities of each team member, including the acceptance of patient responsibility, the sources for relevant information, or the expected communication channels for updates/queries (e.g. pagers, phones) or patient status changes	Everyone on a trauma team understands that the first information about a patient will be delivered by the emergency medical personnel.
Team members	Knowledge of other teammate’s team- and task-relevant knowledge, skills, attitudes (SAs), personalities, and preferences including transactive memory systems, expertise, and hierarchies or power gradients based in roles and education	Everyone on the team understands that when a patient is coding, the physician responsible for that patient, will be in charge of the logistics for resuscitating that patient.

Measuring SMMs

The dearth of research on SMMs in healthcare could be due to the fact that they are difficult to assess. Effective measurement of SMMs has proven challenging even in experimental environments, without much consensus among experts.^5,6 For instance, Burtscher and Manser⁵ found that in a review of 33 published studies on SMMs, seven different methods were used for assessment and ten methods were used for analysis. Further, they concluded that “more than a decade after a seminal review by Mohammed et al.⁶ was released, there is still no consensus among the scientific community regarding measurement” (p. 1351). Despite discrepancies in the appropriate form of measurement, research is consistent with regard to the importance of SMMs in excellent teamwork stating that well-established SMMs improves a team’s ability to communicate and coordinate.^2–6

Because appropriate assessment is crucial for better understanding SMMs and their relationship with teamwork and patient safety, this paper seeks to provide a ground level view of how SMMs in healthcare teams can be studied and evaluated. To delineate the most appropriate SMM assessment methods within healthcare, we synthesized meta-analytic evidence⁷ and a seminal systematic review.⁸ The two articles are a review of team SMMs⁶ and cognitive interview techniques.¹⁰ DeChurch’s article focuses on measurement of team SMMs with a particular focus on the various ways the literature has utilized SMM measurement in teams and how, regardless of measurement technique, SMMs are positively related to team performance. This indicates that SMMs play an important role in teamwork across the extant literature even though there is a lot of variance in the way they are measured when looking across studies.

Kohnken et al.’s work instead focuses on one way to assess SMMs, particularly through use of cognitive interviewing techniques compared to standard interviews. This meta-analytic evidence found that cognitive interviews had a strong effect for increased recall of details, although they also led to an increase in errors with regard to recall. Furthermore, effect sizes were larger when participants were involved in staged events compared to video films. This indicates that the cognitive interviewing technique is efficacious for elucidating the way people think, but that positive outcomes are subject to changes as a factor of the method used to interview.

Together these articles demonstrate that SMMs are measurable, but that there is little agreement on how to measure when looking at the current literature base. Further, cognitive interviewing has some tradeoffs, although it is likely a better technique for garnering correct details, it also brings with it more error compared to standard techniques.

In synthesizing this information, we aimed to both foster an understanding regarding the most appropriate methods for assessing SMMs within the healthcare setting, and to aid healthcare researchers in selecting assessment techniques, which empirically demonstrate theorized relationships within the healthcare context. With these aims in mind, we first define mental models as well as SMMs. Next, we describe the methods that can be used to measure team SMMs in the healthcare setting. Finally, we conclude with recommendations for selection of SMM assessment methods based upon the various constraints present in the healthcare setting.

Mental model assessment techniques

Few empirical studies have focused on the concept of SMMs in healthcare. One potential reason is that recommendations for measurement of SMMs do not exist in the healthcare setting. Leveraging work outside the healthcare domain, DeChurch and Mesmer-Magnus⁸ presented three important types of SMM measurement methods. The three techniques can be arranged as follows: (1) elicitation methods, (2) structure representations, and (3) representation of emergence.⁹ Elicitation methods identify the concepts and content within the SMM. Structure representations unite these concepts by organizing them based upon their relationships. Finally, representation of emergence allows multiple SMM “maps” to be compared and contrasted. As an example, the elicitation method would demonstrate one’s family by including content like “mother”, “father”, “brother”, and “sister”, The structure of representation would be demonstrated by a family tree. Finally, the representation of emergence would be determined by the overlap between your family tree and a cousin’s family tree. Refer to Table 2 for a summary and description of these techniques.

Table 2.

Categories of shared mental model elicitation techniques.

Characteristics of SMM	Description	Commonly used techniques
Elicitation method^1,6	Technique used to determine the knowledge content or components of the model	Concept maps
		Card sorting
		Cognitive interviewing
		Pairwise ratings
Structure representation^4,6	Technique used to determine the degree of correspondence between how the knowledge contained in the model is represented in the mind and how the knowledge representation is modeled	Concept maps
		Pathfinder
		MDS
		Distance ratio formula
Representation of emergence⁹	Technique used to determine how individuals’ mental models are considered as components of a team mental model	Concept maps
Representation of emergence⁹		Pathfinder

The remainder of this paper will review common assessment methods that could be utilized in healthcare research to garner insights regarding SMMs. The methods have been arranged according to type of data analysis: qualitative and quantitative. Qualitative methods of data collection and analysis commonly focus on the use of words, gathered through storytelling, theory development, and case study. Conversely, quantitative methods focus on objective measurements commonly by conducting polls, surveys, questionnaires, or direct measurement.

Cognitive interviewing

Cognitive interviewing is a systematic approach to elicit knowledge with the aim of increasing relevant and accurate information.¹⁰ Interviews are transcribed and analyzed using propositional or discourse analysis.⁷ Typically, this involves aggregating interviewer notes and respondent answers, establishing common themes, and determining key findings.¹¹ Meta-analytic evidence suggests this method is particularly beneficial for improving correct detail recall.¹² The healthcare industry has leveraged cognitive interviewing for survey development,^13–16 root-cause analyses,¹⁷ and design guidelines.¹⁸

Cognitive interviewing includes two dominant approaches: think-aloud interviewing and verbal probing.^19,20 Think-aloud interviewing entails the interviewer reading a question, with minimal interjection, and the respondent “thinking aloud” as (s)he answers each question. For example, an interviewer might ask the participant to describe a process as if they were doing it, essentially having the participant “think aloud” all of the steps in first person. Because interviewers solicit information and ask only clarifying questions beyond the scripted interview, minimal training is required, and interviewer bias is limited.²¹ Additionally, the open-ended format of the think-aloud interviewing is conducive for respondents who have a wealth of knowledge.²² Conversely, respondents may have difficulty articulating their processes eloquently; therefore, some respondents may require training on how to effectively “think-aloud”.¹⁹

Verbal probing consists of an interviewer following-up broader questions with more specific questions (i.e. probes) to gain additional insights on a particular topic. Probes can either be asked concurrently or retrospectively. Concurrent probing involves a question–answer exchange between the interviewer and respondent, while retrospective probing involves probes asked after the entire interview has been administered. Regardless of the probing technique, there are two types of probing questions: scripted and spontaneous. Scripted probes are developed a priori, and spontaneous probes are created impromptu. Overall, the advantage of using verbal probing (as opposed to think-aloud interviewing) is that the interviewer can tailor the interview, thus resulting in a more relevant, productive, and consistent interviews.¹⁹ Unfortunately, the active role of the interviewer potentially invokes bias and leading respondents.¹⁹

Within healthcare, these elicitation methods may illustrate differences in procedural expectations between disciplines, professions, or expertise. For example, an expert nurse and novice surgical resident may have different understandings of pertinent information to include in a handoff. The differences and similarities between these two conceptualizations may be used to adjust protocol, create useful cognitive aids, and ensure that all stakeholders are receiving the information necessary to complete their tasks safely and efficiently. However, cognitive interviewing is limited to information that a respondent can verbalize. Additionally, since interviews are typically conducted after performance, bias arises as respondents may distort information or memories may have decayed.⁷ These errors in recall or verbalization may result in crucial minor details being overlooked and thus left out of any future interventions.

Although cognitive interviewing is traditionally conceptualized as one-on-one interviews to establish individual mental models, they can also be conducted in groups to conceptualize team mental models, adding a level of flexibility not available to all SMM techniques. Group discussions can generate important linkages; however, the views of the most extroverted and opinionated may dominate the discussion and misrepresent the team’s mental model. Despite this potential limitation, group interviewing may encourage teams to engage in self-correction, strengthening the quality of the SMM, not only allowing for researchers to measure the SMM but practitioners to enhance it.

Card sorting

Card sorting refers to a diverse group of techniques that involves the naming, arrangement, and ordering of concepts in accordance to a set of criteria. Card sorting has been used to study cognitive function²³ and has provided insight to participants’ mental organization of information.

Card sorting can be an effective method in eliciting important concepts of a mental model as well as provide insight into how concepts are cognitively associated.⁸ In its most basic form, the card sort involves concepts written on physical cards where participants arrange and order the cards according to their own understanding and perception of the topic.²⁴ In general, there are two approaches labeling the groupings.²⁵ In open card sorting, the labels are generated by the participants as they group the concepts. In closed card sorting, the labels are provided by the investigators and are typically generated from prior research. For example, in an open card sort, participants might organize a standard deck of playing cards by color, face, number, or any other chosen manner. In a closed card sort, participants would be asked to sort the cards strictly by suit.

After the groupings are generated, they can be compared across individuals to derive general principles. However, more modern forms of card sorting employ virtual cards on computers and utilize cluster analysis to construct tree diagrams (i.e. dendrograms). These provide a visual depiction of how concepts and terms are grouped by an individual or by a set of individuals and allows for comparisons and calculation of similarities between different models. Further, online card sorting software permits card sorting be accomplished remotely with little input from the investigator, making it flexible with regard to where it can be implemented.

In healthcare, there has been a growing interest in measuring SMMs through methods like card sorting to understand their impact on areas such as the integration of healthcare organizations,²⁶ collaboration of clinical teams,²⁷ and implementation of clinical guidelines.²⁸ Card sorting has been employed to show significant differences between mental models among healthcare team members,^29,30 as well as demonstrated how disparate mental models impact computerized decision aids design,³¹ clinical team performance,³² and the development of clinical handoff tools.³³ This particular method of elicitation may be helpful within the context of healthcare for understanding differences in role expectations or policy perceptions between disciplines, professions, or levels of tenure.

Card sorting can be a simple, time-efficient technique that allows for a large number of participants. However, the insight into the structure of concepts is relatively crude and only provides information on what ideas are related, not the relative strength of the relationships. Therefore, card sorting is effective at eliciting the content of mental models with minimal insight into the structure of the information.

Finally, card sorting specifically focuses on individual mental models, which allows the comparison of each team member’s mental model with other members but its main limitation is that it does not evaluate how those mental models interact within the team.⁸ Nevertheless, it is an important tool to have in one’s arsenal, as the team SMM is directly influenced by how individual mental models overlap. Future applications may examine using the card sorting task at the group level as a method of training and solidifying team conceptualization and strengthening SMMs.

Concept mapping

Concept maps are visual hierarchical representations that examine the organization and structure of knowledge.^34,35 Originally developed to elicit the knowledge and understanding of basic scientific principles in children, concept mapping has evolved into both qualitative and complex quantitative modeling techniques. By understanding an individual’s organization of knowledge, gaps of understanding and mismatches between team members’ knowledge representations can be identified and addressed.

In terms of elicitation, every map is guided by its most general concept, typically the item which you are trying to elicit the information about (i.e. an event, object, or process).³⁵ In concept mapping, each concept is connected by branches or lines, which describe the relationship between them. Each layer beneath this concept branches out to include other sub-concepts within this category. For example, consider the concept of teamwork. See Figure 1 for an example of concept mapping. Concept mapping is useful not only for knowledge elicitation but also as a comparison method between experts and novices, individuals within the same team, and those completing the same or similar tasks.^35–38

Figure 1.

Example of concept map of “teamwork”.

There are several quantitative concept mapping methodologies that have been identified for measuring SMMs including pairwise ratings, distance ratio formulas, multidimensional scaling (MDS), and Pathfinder.^7,8 Each of these will be discussed in detail below.

Pairwise ratings

Pairwise ratings are used to identify if two paired concepts are similar, dissimilar, or identical. Participants are asked to rate pairs of concepts based upon how similar/related they are on a scale that commonly ranges from “not at all related” to “highly related”. These data are then transferred into a matrix for analysis (see Figure 2.).³⁹ For SMMs, the pairwise rating method is based on relationships and organization of words held in the memory,³⁹ connected through a hierarchical and stereotypical relationship.³⁹

Figure 2.

Example of a rater response matrix for the concept map of “teamwork”.

Pairwise comparison has been used to measure patients, healthcare teams, healthcare education/training, and medical sciences and pharmaceutical research.^40–43 Pairwise ratings have four advantages: time efficiency, ease of use, the ability to provide a more robust data set, and the ability to create a statistically rendered mental model. Given these advantages, pairwise ratings can be particularly useful in the context of healthcare where patients are incorporated as a valued member of the team, or at the very least, recognized as the key informant that may prevent errors.⁴⁴

Because pairs can be read to the participant and the participant can respond verbally, pairwise ratings offer a convenient method of eliciting patient mental models regardless of the education/literacy levels. Last, rather than the participant expressing or expounding upon the SMM, statistical analysis creates the data-based models.⁷ The disadvantage of collecting data in this manner is that as the list of pairs is increased, the risk of the task becoming arduous to the participant increases.⁷

Distance ratio formula

The distance ratio formula is an algorithm employed when using causal mapping measures (visualizing the influence or causality between variables). It can be used to study complex networks, even when variables interact with themselves directly (i.e. self-loop).^45,46

The distance ratio formula builds upon the pairwise rating by requiring participants to comment on the relationship of the concept pairs as having a positive or negative interaction and the strength of that interaction. The raters’ matrices are then analyzed using simple calculations to produce a convergence ranging between 0 and 1, 0 being most similar and 1 being least similar.⁴⁷

This technique allows for the comparison of complex networks between individuals or within a group.^39,45 The data set that is rendered using this technique is robust and rich, providing not just the relationship between concepts but the direction of that relationship. A broad sample is required to achieve a rich data, which becomes increasingly difficult to analyze as the matrices expand.^45,47 Additionally, if a link is absent between two concepts, the formula treats this absence as if the concept itself were missing.³⁹

Distance ratio formula has been applied across multiple fields of study including social relations, power grids, genes, and text analysis.^48–52 Specific to healthcare, this method can be applied to measure providers’ (individual or team) SMMs throughout a variety of tasks, both qualitatively and quantitatively.⁷ One particularly useful domain of application for this method of elicitation is the potential causes of sentinel events. Using this method with interprofessional healthcare teams not only aids in understanding what potential causes lead to errors or near-misses, but also allows researchers and practitioners to identify differing conceptualizations of risky behaviors between professions and disciplines.

Multidimensional scaling

MDS refers to a method of creating a visual representation of similarity between individual cases by using proximity data. The main difference with MDS and distance mapping is that it requires data-driven, a priori categories, compared to post hoc categories. This indicates that a theoretical basis must be established ahead of measurement to ensure that the method is being used appropriately. In this manner, this method can be used as an alternative to factor analysis, allowing the researcher to “discover how and why variables are related”.⁵³ MDS is flexible in that it can be used to analyze multiple types of information, including interval, ratio, and ordinal level metrics. MDS has been applied to a multitude of differing fields including anthropology, economics, organizational behavior, political science, psychology, and sociology.⁵³

Data are mapped upon a two-dimensional space with each concept being represented by a marker, usually a dot. Similar concepts have smaller spaces between them and dissimilar concepts have larger spaces between them. The more information you use to create an MDS, the more reliable the model becomes as dots cluster together into related themes. However, higher densities of markers create complexity and can be difficult to interpret.

The advantage of MDS is that one can manipulate the data set so that the convergence becomes refined, expressing more detail regardless of the scale applied. Further, this method provides the ability to create a pictorial representation by clustering more similarly related concepts. MDS also allows the researcher to identify the dimensions used by the respondent to judge similarity and dominance. MDS can also compensate for missing data within a dataset. The only requirement is that the data entries must be relatively large compared to the number of relationships between them.⁵³

MDS does have some methodological issues, including the fact that once the data are refined to a small section to show intricacy, it becomes easy to lose the large-scale view of the data. For instance, when a researcher is scaling the data, the best technique to use is not always easily identifiable.⁷ Another disadvantage of MDS is the requirement of a large number of respondents to ensure there are more data points than relationships between those points.⁵⁴ This can lead to an extraordinarily large number of ratings for even small set of concepts, leading to a time-consuming data collection process.

Within healthcare, MDS has been used to extend patient-centered marketing research by addressing patient satisfaction, patient loyalty, and hospital image with each measure being a coalescence of other factors like physical environment quality, interaction quality, outcome quality, and effect on image.⁵⁵ MDS has also been used for complex concepts such as psychiatric disorders, quality of life and quality of care, end of life care, health planning and evaluation, and development of guidelines for public health management of lower prevalence chronic conditions such as epilepsy, communication patterns, and doubt/uncertainty in clinical settings.^56,57

Potential applications of measuring SMMs in healthcare can range from understanding what medicines are confused with one another, to creating associations between similar patient cases and disease states, procedure processes and anticipated complications, patient handoffs, and more. MDS allows for a very large system, like healthcare, to be analyzed in small pieces without losing site of the “big picture.”

Pathfinder

Pathfinder is a structural assessment technique that represents each concept within a map as a node, with the distance between nodes acting as a representation of the strength of the relationship between those two concepts. The more similar a concept, the more closer together the nodes. This technique visually generates each person’s mental model by comparing judgments given between raters.⁵⁸ Using Pathfinder, similarities can be represented in a network structure, where these similarities are shown as nodes. Pathfinder applies a numerical weight based upon the closeness and the similarity between concepts. Because each person has a different perception of how concepts may relate to one another, Pathfinder must compare the models created by all individuals. This is accomplished by assessing “closeness”. “Closeness” refers to whether a node is present in structures for two individuals.⁵⁹ Closeness is computed for each node, and the results are averaged, which then indicates the overall similarity of the mental models. The range of similarity is from 0 to 1, 0 being unrelated and 1 indicating identicalness. As an example, Pathfinder can be implemented when analyzing SMMs with regard to handoff protocols. Items that are passed during the handoff can be grouped together according to relatedness. Terms like “patient identity” and “name” would be placed close together by participants and most likely assigned a “closeness” rating of 1, because the terms are synonymous. However, terms such as “blood pressure” and “temperature” would be placed near each other because they are commonly understood to be vital signs, but would not have a high closeness rating because they are independent concepts.

Although Pathfinder represents the similarity of perceived relatedness, it does not depict causality of the perceived content.⁶⁰ Much like other measurement tools of mental models, there are prescribed concepts that are to be compared between individuals; therefore, Pathfinder may not be applicable to all scenarios.⁶⁰ Pathfinder differs from other assessment techniques because it represents better comparisons of judgments in each structure.⁶¹

Advantages of using Pathfinder include the ability to compare the similarity of concepts of different networks through an algorithm, reducing bias present when scored by people. It organizes the concepts through networks or trees, making it easier to use and understand.⁷ It also provides a more predictive advantage over other measurement tools because it exemplifies the comprehensive relationships of concepts rather than the direct relationships.⁶² Pathfinder’s global approach, which analyzes similarity between networks, has been recognized as more reliable and valid over MDS or simple representation models.^62–64

Pathfinder’s simplicity, conceptual approach, and overall comprehensive technique has been applied in a broad range of industries, including: aviation, human–computer interaction, healthcare, and education.^5,7,65–67 There are also disadvantages of the Pathfinder technique in measurement of SMMs. Pathfinder does not adequately provide semantic information behind the nodes or concepts, rather it just provides the general ideas.⁷ Therefore, the causality is not known between the nodes but just understood that there is similarity.⁶⁰ The Pathfinder technique is also very repetitive in nature for the rating of concepts; therefore, there is a possibility of the responses to lack continuous reliability or validity.⁷

Overall, there is a lack of recent published research or work for Pathfinder and mental model research. Despite this lack of published work, the recommended use of Pathfinder is for individuals working interdependently under time pressure in a dynamic setting where performance influences serve as an input and output, which lends itself well to healthcare.^7,68

Conclusion

This paper reviews multiple techniques, each with their own advantages and disadvantages that could be applied to better understanding of SMMs in the healthcare context. Although it is well established that SMMs are important for excellent teamwork, little research in healthcare effectively assesses SMMs and their relationship to relevant healthcare outcomes such as patient safety. Therefore, we aimed to provide a basis for future research in this area through summarizing the current practices used to assess SMMs and the various ways they could each be utilized in the healthcare setting. A summary of the techniques along with their advantages and disadvantages is presented in Table 3.

Table 3.

Summary of SMM measurement techniques.

Measurement technique		Definition/Method	Advantages	Disadvantages	Example
Cognitive interviews¹⁰		Systematic approach to elicit knowledge with the aim of increasing relevant and accurate information. Interviews are transcribed and analyzed using propositional or discourse analysis.	Level of flexibility that is not available to most of the SMM techniques	Limited to information that a respondent can verbalizeParticipants may unknowingly distort informationViews of the most extraverted in the group discussions may misrepresent teams mental model	Has been used for survey development
	Verbal probe¹⁹	Consists of an interviewer following-up broader questions with more specific questions (i.e. probes) to gain additional insights on a particular topic	Interviewer can tailor/guide the interview to get relevant insight into a topic	Active role of interviewer does invoke potential for bias and leading questions Limited to information that the respondent can verbalizeProne to distortions in memory	___________
	Think-aloud¹⁹	Entails the interviewer to read a question with little additional interjection and the respondent “thinking aloud” as (s)he answers the question. While the respondent is answering, the interviewer records the processes the respondent used to arrive at their answer	Open-ended questions allow for a lot of information to be gatheredMinimal training is required for interviewer	Burden is on respondentProne to distortions in memoryLimited to information that the respondent can verbalizeMay have to train participants how to effectively think-aloud	___________
Card sorting⁸		Diverse group of techniques that involve the naming, arrangement, and ordering of concepts in accordance to a set of pre-determined criteria. Concepts that are either written on physical cards, or are virtual cards that a participant then arranges and orders according to their own understanding and perception of the topic. Those groupings then can be compared to derive a visual depiction on how the concepts and terms are grouped by a team. This method allows for the similarities and comparisons between models to be calculated.	Gives insight into how the concepts are cognitively associatedCan be done remotelyHas some flexibilityCan provide subjective detail when paired with questions from investigatorTime efficient	Does not provide the relative strength of the relationshipsDoes not evaluate how mental models interact within the team	Used to significant differences between mental models among healthcare team members as well as demonstrate how disparate mental models impact computerized decision aids design, clinical team performance and the development of clinical handoff tools
	Open card sort²⁵	Participants sort cards into as many groups as desired and then name the groups	–	–	_________
	Closed card sort²⁵	Participants are asked to sort cards into pre‐defined groups generated from previous research	–	–	_________
Concept mapping^34,35		Concept maps are visual hierarchical representations of knowledge. Every map is guided by its most general concept, typically the item from which you are trying to elicit the information about, whether that is an event, an object, a process, etc. Then each layer beneath that concept, branches out to include other sub-concepts that fall within this category.	Able to compare between experts and novices, individuals completing the same task, and individuals on the same team	ComplexMany ways to analyze data	Has been used for complex constructs such as “coping, depression, anxiety, a theory of care, quality of life and quality of care, end of life care, etc.
	Pairwise ratings^39–42	Pairwise ratings are used to identify if two paired concepts are similar, dissimilar, or identical. Participants rate pairs of concepts based upon how similar/related they are on a scale that commonly ranges from “not at all related” to “highly related.” These data are then transferred into a matrix for analysis.	Easy to administerParticipants with little to no education can still participateTime efficient	As the list of pairs is increased, the risk of the task becoming arduous to the participant also increases	Has been used to measure patients, teams within medicine, education/training within medicine, and medical sciences/pharmaceutical research
	Distance ratio formula^39,45	Algorithm employed when using causal mapping measures that is used to study complex networks	Provides a rich data set about the interactions of concepts	Can become difficult to understand as the matrices expand	Has been used to measure providers (both individually or as a team) shared mental models
	Multi-dimensional scaling (MDS)^53,55	Method of creating a visual representation of similarity between individual cases by using proximity data.Requires creation of a priori boundaries and parameters to explore differences between data sets.Wide boundaries are set and redefined for more detailed analysis.	Maintains the integrity of the data by preserving the relationships between the data points, while enhancing the view of the relationships	Can lose sight of the large scale view of the dataLarge number of participants are needed	Addresses patient satisfaction, patient loyalty, and hospital image with each measure being a coalescence of other factors like physical environment quality, interaction quality, outcome quality, and effect on image
	Pathfinder^7,58	Structural assessment technique, which represents each concept within the map as a node and distance closer or farther apart from one another depending on the strength of the rater’s judgment of the relationship between those two concepts. Generates mental model of each person based on comparing judgements given by each member	Able to show the degree of similarity in teamsSimpleMore reliable and valid tool	Does not adequately provide semantic information behind the nodes or concepts, rather it just provides the general ideasCausality is not known between nodesRepetitive	Has been used in a broad range of industries, such as, aviation, HCI, healthcare, and education

Those wishing to conduct SMM research within the healthcare industry must take into consideration the variety of factors affecting the appropriate selection of method(s) used. Time pressures, availability of resources and technology, and access to participants can severely affect the ability to implement some of these assessment methods. Some of these methods depend on the use of software, requiring financial and budget approval/support. For others, it may be difficult to obtain time with participants to complete a cognitive interview or a lengthy pairwise rating session. In this instance, a card sort may be the easiest solution. While it is the recommendation of the authors to pair qualitative and quantitative methods of analysis in order to provide a robust and rigorous analysis of conceptual relationships, the previously mentioned factors and others can prevent this possibility of using multiple measures. Researchers must select the most appropriate measurement technique for each unique healthcare setting and study goal. A summary of assessment techniques in relation to possible setting constraints is presented in Table 4.

Table 4.

Preferred measurement techniques based on provider circumstance.

Measurement technique	Data gained (content vs. structure)	Quantitative vs. Qualitative	Technology burden	Effort to use	Time needed to complete
Closed card sorting	Structure	Qualitative (quantitative is using software)	Manual (software available)	Easy	Minimal if emailed, moderate if performed by hand.
Open card sorting	Structure and content	Qualitative (quantitative is using software)	Manual (software available)	Easy	Minimal if emailed, moderate if performed by hand.
Cognitive interviews	Content	Qualitative	Manual (software available)	Easy	Maximum
Concept mapping	Structure	Both	Manual (software available)	Medium	Moderate
Pathfinder	Structure	Quantitative	Software needed	Medium	Minimal if familiar with software, maximum if learning the software
Distance ratio formula	Structure	Both	Manual (software available)	Medium	Moderate
Pairwise ratings	Structure	Quantitative	Manual (software available)	Medium–Hard	Moderate–Maximum
Multidimensional scaling	Structure	Quantitative	Software needed	Hard	Minimal if familiar with software, maximum if learning the software

Easy – Can perform this technique with minimal training.

Medium – Simple mathematical understanding of the concept or software needed for analysis.

Hard – Complex mathematics required or use of software required.

Even though SMMs are believed to be an important theoretical aspect of clinical decision-making and performance,⁶⁹ there is insufficient empirical assessment of SMMs in healthcare. This paper provides a comprehensive list of assessment techniques, strengths and weakness of each technique, and a foundation for clinicians and researchers alike to utilize the correct assessment techniques for their specific healthcare setting. Future work will need to address techniques that provide empirical support for the relationship between SMMs and the complex outcomes associated with healthcare work.

Footnotes

Application

Although mental models are often mentioned as an important component of teamwork in medicine, they are measured infrequently in the healthcare environment. This article provides guidance for those interested in measuring shared mental models in healthcare.

Precis

This article reviews multiple methods for measuring shared mental models and applies them to the context of healthcare teamwork.

Authors’ contribution

Logan Gisick: Drafting of manuscript and critical revision; Kristen Webster: Conception and design, drafting of manuscript, critical revision; Joseph Keebler: Conception and design, drafting of manuscript, critical revision, guarantor; Elizabeth Lazzara: Conception and design, drafting of manuscript, critical revision; Sarah Fouquet: Drafting of manuscript and critical revision; Keaton Fletcher: Drafting of manuscript and critical revision; Agnes Fagerlund: Drafting of manuscript; Victoria Lew: Drafting of manuscript; Raymond Chan: Drafting of manuscript and critical revision.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Johnson-Laird

PN.

A computational analysis of consciousness. Cogn Brain Theory 1983; 6: 499–508.

Day

Arthur

Jr Gettman

Knowledge structures and the acquisition of a complex skill. J Appl Psychol 2001; 86: 1022.

Mathieu

Heffner

Goodwin

et al . The influence of shared mental models on team process and performance. J Appl Psychol 2000; 85: 273–283.

Klimoski

Mohammed

Team mental model: Construct or metaphor?

J Manage 1994; 20: 403–437.

Burtscher

Manser

Team mental models and their potential to improve teamwork and safety: A review and implications for future research in healthcare. Safety Sci 2012; 50: 1344–1354.

Mohammed

Klimoski

Rentsch

JR.

The measurement of team mental models: We have no shared schema. Organ Res Methods 2000; 3: 123–165.

Langan-Fox

Code

Langfield-Smith

Team mental models: Techniques, methods, and analytic approaches. Hum Factors. 2000; 42: 242–271.

DeChurch

Mesmer-Magnus

JR.

Measuring shared team mental models: A meta-analysis. Group Dyn: Theory Res Pract 2010; 14: 1–14..

Kozlowski

Klein

KJ.

A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In: Klein KJ and Kozlowski SWJ (eds) Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. San Francisco, CA: Jossey-Bass, pp. 3–90.

10.

Geiselman

Fisher

RP.

Interviewing witnesses and victims. In: Investigative interviewing: Handbook of best practices. Toronto, ON: Thomson Reuters Publishers, 2014; 29–53.

11.

Willis

Artino

Jr. What do our respondents think we’re asking? Using cognitive interviewing to Improve Medical Education surveys. J Grad Med Educ 2013; 353–356.

12.

Köhnken

Milne

Memon

et al . The cognitive interview: A meta-analysis. Psychol Crime Law 1999; 5: 3–27.

13.

Chernyak

Ernsting

Icks

Pre-test of questions on health-related resource use and expenditure, using behaviour coding and cognitive interviewing techniques. BMC Health Serv Res 2012; 12: 303.

14.

DiBenedetti

Price

Andrews

EB.

Cognitive interviewing in risk minimization survey development: Patient and healthcare professional surveys. Expert Rev Clin Pharmacol 2013; 16: 369–373.

15.

Drennan

Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. J Adv Nurs 2003; 142: 57–63.

16.

Irwin

Varni

Yeatts

et al . Cognitive interviewing methodology in the development of a pediatric item bank: A patient reported outcomes measurement information system (PROMIS) study. Health Qual Life Outcomes 2009; 7: 3.

17.

Black

Vernetti

BJ.

Root-cause analysis: Creating and utilizing a functional database. Professional Safety 2015; 60: 60–62.

18.

Moon

Miller

Exploring macrocognitive healthcare work: Discovering seeds for design guidelines for clinical decision support. In: Proceedings of the 5th international conference on applied human factors and ergonomics AHFE 2014, Kraków, Poland, 19–23 July 2014.

19.

Beatty

Willis

GB.

Research synthesis: The practice of cognitive interviewing. Public Opin Quart 2007; 1: 287–311.

20.

Willis

DeMaio

Harris-Kojetin

Is the bandwagon headed to the methodological promised land? Evaluating the validity of cognitive interviewing techniques. In: Sirken

Herrmann

Schechter

et al . (eds) Cognition and survey research. New York: John Wiley, 1999, pp. 133–153.

21.

Bolton

Bronkhorst

TM.

Questionnaire pretesting: Computer-assisted coding of concurrent protocols. In: Schwarz

Sudman

(eds) Answering questions: Methodology for determining cognitive and communicative processes in survey research. San Francisco, CA: Jossey-Bass, 1996, pp. 37–64.

22.

Willis

GB.

Cognitive interviewing: A tool for improving questionnaire design. London: Sage Publications, 2004.

23.

Grant

Berg

A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. J Exp Psychol 1948; 38: 404.

24.

Hudson

Card sorting. The Interaction Design Foundation, https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/card-sorting (accessed 22 February 2018).

25.

Wood

LE.

Card sorting: Current practices and beyond. J Usab Stud 2008; 14: 1–6.

26.

Evans

Ross Baker

Shared mental models of integrated care: Aligning multiple stakeholder perspectives. J Health Organ Manag 2012; 26: 713–736.

27.

McComb

Simpson

The concept of shared mental models in healthcare collaboration. J Adv Nurs 2014; 70: 1479–1488.

28.

Hysong

Best

Pugh

et al . Not of one mind: Mental models of clinical practice guidelines in the Veterans Health Administration. Health Serv Res 2005; 40: 829–848.

29.

McComb

Lemaster

Henneman

et al . An evaluation of shared mental models and mutual trust on general medical units: Implications for collaboration, teamwork, and patient safety. J Patient Saf 2017; 13: 237–242.

30.

Nakarada-Kordic

Weller

Webster

et al . Assessing the similarity of mental models of operating room team members and implications for patient safety: A prospective, replicated study. BMC Med Educ 2016; 16: 229.

31.

Freer

Ferguson

Ewing

et al . Mismatched concepts in a neonatal intensive care unit (NICU): Further issues for computer decision support? J Clin Monitor Comput 2002; 17: 441–447.

32.

Lazzara

Riss

Patzer

et al . Directly comparing handoff protocols for pediatric hospitalists. Hosp Pediatr 2016; 16: 722–729.

33.

Novak

JD.

An alternative to Piagetian psychology for science and mathematics education. Sci Educ 1977; 61: 453–477.

34.

Novak

Cañas

AJ.

The theory underlying concept maps and how to construct and use them. Technical Report IHMC CmapTools, Florida Institute for Human and Machine Cognition, 20008.

35.

Delugach

Etzkorn

Carpenter

et al . A knowledge capture approach for directly acquiring team mental models. Int J Hum-Comput Stud 2016; 96: 12–21.

36.

Ferrario

CG.

Developing nurses’ critical thinking skills with concept mapping. J Nurses Prof Dev 2004; 20: 1261–1267.

37.

Gul

Boman

JA.

Concept mapping: A strategy for teaching and evaluation in nursing education. Nurse Educ Pract 2006; 6: 199–206.

38.

Langan ‐ Fox

Anglim

Wilson

JR.

Mental models, team mental models, and performance: Process, development, and future directions. Hum Factors Ergon Manuf Serv Ind 2004; 14: 331–352.

39.

Lin

Williamson

JM.

A simple approach for sample size calculation for comparing two concordance correlation coefficients estimated on the same subjects. J Biopharm Stat 2015; 25: 1145–1160.

40.

Pouchot

Kherani

Brant

et al . Determination of the minimal clinically important difference for seven fatigue measures in rheumatoid arthritis. J Clin Epidemiol 2008; 61: 705–113.

41.

Qian

Wang

KW.

Medical waste disposal method selection based on a hierarchical decision model with intuitionistic fuzzy relations. Int J Environ Res Public Health 2016; 13: 896.

42.

Tractenberg

Gushta

Weinfeld

JM.

The mastery rubric for evidence-based medicine: Institutional validation via multidimensional scaling. Teach Learn Med 2016; 28: 152–165.

43.

Longtin

Sax

Leape

et al . Patient participation: Current knowledge and applicability to patient safety. Mayo Clin Proc 2010; 85: 53–62.

44.

Caseiro

Trigo

Distance ratio: An exploratory application to compare complex networks. In: Proceedings of the European conference on complex systems, 2012, pp. 123–130). Cham: Springer.

45.

Schaffernicht

Groesser

SN.

A comprehensive method for comparing mental models of dynamic systems. Eur J Oper Res 2011; 210: 57–67.

46.

Ross

Allen

Examining the convergent validity of shared mental model measures. Behav Res Methods 2012; 44: 1052–1062.

47.

Boccaletti

Latora

Moreno

et al . Complex networks: Structure and dynamics. Phys Rep 2006; 424: 175–308.

48.

Butts

CT.

Revisiting the foundations of network analysis. Science 2009; 325: 414–416.

49.

Newman

ME.

The structure and function of complex networks. SIAM Rev 2003; 45: 167–256.

50.

Dorogovtsev

Mendes

JF.

Language as an evolving word web. Proc R Soc Lond B: Biol Sci 2001; 268: 2603–2606.

51.

Rubinov

Sporns

Complex network measures of brain connectivity: Uses and interpretations. Neuroimage 2010; 52: 1059–1069.

52.

Pinkley

Gelfand

Duan

When, where and how: The use of multidimensional scaling methods in the study of negotiation and social conflict. Int Negot 2005; 10: 79–96.

53.

Gelfand

Triandis

Chan

DK.

Individualism versus collectivism or versus authoritarianism?

Eur J Soc Psychol 1996; 26: 397–410.

54.

Chahal

Kumari

Development of multidimensional scale for healthcare service quality (HCSQ) in Indian context. J Indian Business Res 2010; 2: 230–255.

55.

Trochim

Kane

Concept mapping: An introduction to structured conceptualization in health care. Int J Qual Health Care 2005; 17: 187–191.

56.

Segalowitz

Doucerain

Meuter

et al . Comprehending adverbs of doubt and certainty in health communication: A multidimensional scaling approach. Front Psychol 2016; 7: 558.

57.

Stout

Cannon-Bowers

Salas

et al . Planning, shared mental models, and coordinated performance: An empirical link is established. Hum Factors 1999; 41: 61–71.

58.

Davis

Curtis

Tschetter

JD.

Evaluating cognitive training outcomes: Validity and utility of structural knowledge assessment. J Business Psychol 2003; 18: 191–206.

59.

Lim

Klein

KJ.

Team mental models and team performance: A field study of the effects of team mental model similarity and accuracy. J Organ Behav. 2006; 27: 403–418.

60.

Acton

Johnson

Goldsmith

TE.

Structural knowledge assessment: Comparison of referent structures. J Educ Psychol 1994; 86: 303.

61.

Kraiger K and Wenzel LH. Conceptual development and empirical evaluation of measures of shared mental models as indicators of team effectiveness. In: Brannick MT, Salas E and Prince CW (eds) Team performance assessment and measurement: Theory, methods, and applications. 1997.

62.

Rye

Rubba

PA.

An exploration of the concept map as an interview tool to facilitate the externalization of students’ understandings about global atmospheric change. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching 1998; 35(5): 521–546.

63.

Gillan

Breedin

Cooke

NJ.

Network and multidimensional representations of the declarative knowledge of human-computer interface design experts. Int J Man-Mach Stud 1992; 36: 587–615.

64.

McDonald

Paap

McDonald

DR.

Hypertext perspectives: Using Pathfinder to build hypertext systems. In: Schvaneveldt RW (eds) Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex Publishing Corp., 1990, pp. 197–211.

65.

Vortac

Edwards

Manning

CA.

Sequences of actions for individual and teams of air traffic controllers. Hum-Comput Interact 1994; 1 9: 319–343.

66.

Burke

Salas

Wilson

et al . How to turn a team of experts into an expert medical team: guidance from the aviation and military communities. BMJ Qual Saf 2004; 13: i96–104.

67.

Haig

Sutton

Whittington

SBAR: A shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf 2006; 32: 167–175.

68.

Langfield-Smith

Wirth

Measuring differences between cognitive maps. J Oper Res Soc 1992; 43: 1135–1150.

69.

Markíczy

Goldberg

A method for eliciting and comparing causal maps. J Manage 1995; 21: 305–333.