The Gap Between Perceived and Actual Learning From Group Reflection

Abstract

Group reflection is often used as an intervention to facilitate group performance, but reflecting in groups may also affect individual learning. In this article, we compare the effects of individual and group reflection on individuals’ learning in two pairs of decision-making tasks. In two studies, we found that individuals who reflected in groups improved their performance from Task 1 to 2. However, individuals who reflected in groups did not realize greater performance improvements than individuals who reflected alone. Furthermore, individuals who reflected alone perceived that they learned more than individuals who reflected in groups. We discuss the implications of the gap between perceived and actual learning and describe the implications of our findings for group research, as well as recommendations for future research.

Keywords

group reflection reflexivity subjective assessments of learning decision making

There is an increasing interest in the value of working in groups to facilitate the learning of individuals (Olivera & Straus, 2004). Individual learning is often indicated as one of the desired outcomes of working in groups (Borredon, Deffayet, Baker, & Kolb, 2011; Hackman & Wageman, 2005) and, under the right conditions, the lessons learned while working in groups can translate into individual performance improvements (Laughlin, Carey, & Kerr, 2008; Littlepage, Robison, & Reddington, 1997). However, group experiences can also frustrate participants and hinder their ability to learn (Bacon, 2005; Hollingshead, 1998).

One intervention that may facilitate individual learning from group experiences is group reflection. Reflection is a cognitive process in which individuals actively consider what they can learn from their experiences (Gordon & Smith Hullfish, 1961; Raelin, 2002). Ideally, reflecting in groups should provide individuals with opportunities to break out of embedded assumptions, learn from past experiences, and consider alternative viewpoints that may facilitate future decision-making performance (Anseel, Lievens, & Schollaert, 2009; Foldy & Buckley, 2010). Organizational researchers have explored the effects of variance in “naturally occurring” reflection on group performance (e.g., Schippers, Den Hartog, Koopman, & Wienk, 2003), as well as individual innovation (Urbach, Fay, & Goral, 2010). Informal opportunities for collective reflection in organizations (i.e., huddles) have also been associated with individual learning (Quinn & Bunderson, 2013). However, research examining the effects of formal group reflection interventions on individual learning has been rare, and the inconclusive results of group reflection interventions in facilitating individual outcomes suggest that this literature remains an area of active debate (e.g., Gurtner, Tschan, Semmer, & Nägele, 2007; Moreland & McMinn, 2010).

Literatures in psychology and education speak to the effects of cooperative learning (e.g., Kyndt et al., 2013), decentering (Piaget, 1962), and perspective-taking (Mezirow, 2003) on individual learning outcomes. However, these studies seldom examine the effects of group reflection interventions on individual learning. In this study, we explore the effects of a formal group reflection intervention on individual learning in decision-making tasks that reflect the uncertainty and complexity of the decisions made in organizations (e.g., sales estimates or strategic planning with multiple decision points). In addition, this research addresses an ongoing concern for researchers and organizations (e.g., Association to Advance Collegiate Schools of Business [AACSB] International, 2007; Sitzmann, Ely, Brown, & Bauer, 2010) by simultaneously including and comparing objective measures of learning (i.e., performance improvement) and individuals’ subjective evaluations of their learning.

Reflecting in groups may not only affect whether individuals learn but also whether individuals perceive that they have learned. Sitzmann et al. (2010) distinguish self-assessments, or individuals’ subjective evaluations of learning, from cognitive learning (i.e., objective measures of performance improvements or demonstrations of knowledge acquisition). Self-assessments are often used as a primary measure of learning, both in organizations and by researchers (Benbunan-Fich & Hiltz, 2003; Lim & Morris, 2006; Morgan & Casper, 2000). However, researchers have found discrepancies between individuals’ perceptions of their learning and objective measures of learning (Bowman, 2010; Witt & Wheeless, 2001). Thus, self-assessments may not always be accurate proxies for cognitive learning, but our understanding of how self-assessments of learning differ from measures of cognitive learning is limited because self-assessments and measures of cognitive learning are seldom compared in the same studies (Dunning, Heath, & Suls, 2004).

In this article, we explore the possibility that individuals who reflect in a group learn more than individuals who reflect alone but perceive that they have learned less. Research in both educational and organizational settings suggests that, for many reasons, individuals frequently have negative attitudes toward working in groups (e.g., Bacon, Stewart, & Silver, 1999; Chapman, Meuter, Toy, & Wright, 2009; Kirkman, Jones, & Shapiro, 2000). For instance, frustration with working in groups may result from group members who engage in off-topic conversations (Rasker, Post, & Schraagen, 2000) or social loafing (Kerr & Bruun, 1983), from interpersonal conflicts (Erbert, Mearns, & Dena, 2005), or from other dysfunctional team behaviors of other members (Cole, Walter, & Bruch, 2008). Thus, even if individuals learn from reflecting in groups, their beliefs about whether they have learned may be negatively affected by their experiences in the group.

We report the findings of two studies that explore the effects of reflecting alone versus in a group on individuals’ self-assessed and objectively measured (i.e., cognitive) learning. In Study 1, we use two parallel estimation tasks (Bonner & Baumann, 2012; Bonner & Sillito, 2011) to measure differences in cognitive learning and participants’ perceptions of learning from Task 1 to Task 2. As a cognitive exercise, estimation occurs commonly both in normal life contexts (e.g., estimating how long it will take to get a table at a favorite restaurant on a Saturday night) and in organizations (e.g., forecasting future earnings). Study 2 builds on the design of Study 1 by using a pair of decision-making tasks that incorporate the complexity of survival scenarios in extreme environments (Ferrin & Dirks, 2003; Littlepage, Schmidt, Whisler, & Frost, 1995). We conclude by discussing the implications of our findings for research and practice.

Group Reflection and Cognitive Learning

Reflection describes a process whereby individuals or group members intentionally think about existing “objectives, strategies, and processes and adapt them to current or anticipated endogenous or environmental circumstances” (West, 1996, p. 559). Reflection relates to the idea of meta-cognition in learning, whereby individuals think about and monitor their own cognitive processes and, if necessary, make adjustments in pursuit of relevant goals (Mayer, 2001). The content of reflection can vary, depending on the goals of the individual. For example, individuals can reflect on alternative courses of action to meet task requirements (e.g., Marks, Mathieu, & Zaccaro, 2001), as well as how to implement them (Frese & Zapf, 1994). Reflection involves cognitive benchmarking such that individuals consider the extent to which their mental models about the nature of the task and strategies to cope with it differ from those of colleagues or teammates (Mathieu, Heffner, Goodwin, Cannon-Bowers, & Salas, 2005).

Evidence across multiple literatures suggests numerous learning benefits associated with systematic collective reflection. Ellis and Davidi (2005) find that Israeli soldiers who engaged in formal reflection (i.e., after-event reviews) following both successful and unsuccessful navigation training exercises improved significantly in subsequent exercises. Reflection, particularly when combined with feedback, has been found to enhance objective learning outcomes among employees on a web-based work simulation training module (Anseel et al., 2009). According to a recent meta-analysis, multiple studies of learning interventions find that cooperative learning is positively associated with improvements in students’ academic achievement (Kyndt et al., 2013). The value of collective reflection has been examined in several other contexts, including TV production crews (Carter & West, 1998), surgical teams of doctors and nurses (Edmondson, 1999), and Irish top management teams (MacCurtain, Flood, Ramamoorthy, West, & Dawson, 2010).

Reflecting in a group may benefit individuals’ cognitive learning in multiple ways. For instance, individuals who reflect in a group can gain access to the information held by multiple group members rather than being limited to only the information they alone know (Ellis & Bell, 2005). Groups are generally more capable than individuals at processing and storing a greater volume of information (Hinsz, Tindale, & Vollrath, 1997), which suggests that group reflection should offer access to a broader array of pertinent lessons than would be available to individuals reflecting alone. Another potential advantage of group reflection involves the generative effects of discussing problems and potential solutions collectively. For instance, research on cognitive stimulation in brainstorming groups suggests that an idea discussed by one group member can prime a related idea for another person, which in turn sets off a chain reaction that may facilitate more optimal solutions (Osborn, 1957; Paulus & Brown, 2003). As individuals engage in cooperative learning, they gain rich opportunities for cognitive growth from confronting and understanding ideas that are different from their own (Rogoff, Goodman Turkanis, & Bartlett, 2001). The process of actively taking others’ perspectives and reconciling the multiple, sometimes competing perspectives of one’s peers is a central element in learning that is transformational (Mezirow, 2003).

The ability to bring a greater variety of problem-solving strategies to bear on a problem is another potential advantage of reflecting in groups. Reflecting in groups may facilitate cognitive learning by encouraging individuals to share and elaborate on task strategies (van Ginkel, Tindale, & van Knippenberg, 2009). When individuals reflect in groups, they have opportunities to learn how other members approach a problem and integrate the problem-solving strategies described by others to facilitate their own performance. For instance, a study of unstructured collective reflection suggests that students compare their representations of the problem (e.g., sharing their handwritten notes) to learn from each other’s interpretations of the course content (Sawyer & Berson, 2004). Research on the effects of collaborative student groups on cognitive learning supports the idea that reflecting in a group facilitates the development of a wider array of problem-solving strategies (e.g., Cooper, Cox, Nammouz, Case, & Stevens, 2008).

Based on the superior processing capacity of groups over individuals, the potential for opportunities to compare and learn from each other’s problem-solving strategies, and the generative effects of building on the ideas and problem-solving strategies of others, we expect that individuals who reflect in groups will experience greater cognitive learning (i.e., significant improvements in performance from Task 1 to Task 2) than individuals who reflect alone. Although the efficacy of group reflection is an area of active debate in the literature (e.g., Moreland & McMinn, 2010), our expectation that group reflection will facilitate individuals’ cognitive learning is based on the assumption that individuals who reflect in groups will have access to a wider repertoire of task strategies, a greater amount of information to draw on, and the benefits of cognitive stimulation from listening to and building on the ideas discussed by others:

Hypothesis 1 (H1): Individuals who reflect in groups will experience cognitive learning, measured by significant improvements in performance from Task 1 to Task 2.

Hypothesis 2 (H2): Individuals who reflect in groups will experience greater cognitive learning, measured by improvements in task performance, than individuals who reflect alone.

Group Reflection and Self-Assessments of Learning

Individuals’ self-assessments are often used as a primary measure of whether learning has occurred in both individual (Arbaugh, 2005) and cooperative (Peterson & Miller, 2004) learning contexts. However, self-assessments of learning have been criticized for only modestly correlating with more objective measures of learning (Dunning et al., 2004; Witt & Wheeless, 2001). Thus, stakeholders ranging from researchers (e.g., Sitzmann et al., 2010) to higher education accrediting bodies (AACSB International, 2007) have called for approaches that measure both self-assessments of learning as well as cognitive learning.

Individuals’ subjective evaluations of whether and how much they have learned in groups are useful to examine as an outcome in their own right (Hackman, 1987; Hackman & Wageman, 2005). For instance, with recent trends in organizations toward individuals working in multiple, temporary groups with fluid membership, there is an increasing interest in the learning that individuals take away from each of their group experiences (O’Leary, Mortensen, & Woolley, 2011). Similarly, organizations often evaluate the efficacy of training interventions by assessing employees’ subjective reactions to the training (Aguinis & Kraiger, 2009; Morgan & Casper, 2000). Thus, in this study, we incorporate both measures of self-assessed and cognitive learning to more fully capture the effects of group reflection on individual learning.

Even as reflecting in groups may facilitate individuals’ cognitive learning, we suggest that individuals may perceive that they are learning less from reflecting in a group than do individuals who reflect alone. Previous research on group reflection describes several problems that can make group reflection a frustrating experience for participants. Some of the problems with group reflection stem from collective shortcomings in motivation. For instance, groups may be less likely to adhere to the guidelines for reflection than individuals reflecting alone (Daudelin, 1996). Even when groups follow the rules, they may not utilize the full-time period available for reflection (Moreland & McMinn, 2010). Furthermore, individuals reflecting collectively may have to cope with other demotivating aspects of group work, such as social loafing (George, 2011; Karau & Williams, 1993) or individual members who engage in free riding (Kerr & Bruun, 1983; Stroebe & Frey, 2011).

A group’s social and interactional dynamics may also discourage individuals. For instance, several studies have found that individuals experience discomfort when asked to reflect on a task in groups (e.g., Barry, Britten, Barber, Bradley, & Stevenson, 1999; Moreland & McMinn, 2010). Other research finds that individuals reflecting in groups may spend substantial amounts of time discussing topics unrelated to the task (Gurtner et al., 2007) or engage in discussions that are overly shallow or superficial (Daudelin, 1996). Many individuals perceive group work to be counterproductive or a waste of time (Bacon et al., 1999). Thus, we predict that individuals who reflect alone will have greater self-assessed perceptions of learning than individuals who reflect in a group:

Hypothesis 3 (H3): Individuals who reflect alone will rate their self-assessed perceptions of learning more highly than individuals reflecting in groups.

Overview of Decision-Making Tasks

In the following sections, we describe a pair of studies that examine the effects of individuals reflecting in groups versus reflecting alone on self-assessments of perceived learning and objective measures of cognitive learning. Each study employs a pair of structurally equivalent decision-making tasks, with a break between tasks for participants to reflect on their performance in the first task and to think about strategies for how they could improve on the second task. We begin by describing our research strategy and the decision-making tasks used in each study.

Reflection is thought to play a central role in various types of decision making (e.g., Facchin & Tschan, 2006; Reynolds, 2006). For the purposes of this research, we examined reflection in the context of two types of decision-making tasks: estimation tasks (Bonner & Sillito, 2011) and survival tasks (Ashburn-Nardo & Johnson, 2008). We selected tasks that have been used in previous research to replicate the complexity and structure of real-world decisions in both organizational and non-organization settings. In particular, we narrowed our search to tasks where the correct answer can be compared with an objective standard, such as a quantitative value or an expert’s ranking, where there are differences in the level of task complexity, and where formal reflection by participants is likely to influence performance.

First, we selected tasks with demonstrably correct answers (Bonner & Baumann, 2012; Bonner, Baumann, & Dalal, 2002). The tasks in Study 1 require participants to make point estimates of general knowledge questions, such as minimum driving distances between major U.S. cities, average male and female weights, and university enrollments. (The estimation questions used in Tasks 1 and 2 for Study 1 are adapted from Bonner, Sillito, and Baumann (2007) and are listed in Figure 1.) The questions were informally pre-tested with students, yielding feedback that the questions were difficult (i.e., not something that most people would know off the top of their head) but not unreasonable to estimate. Study 2 involves a pair of survival tasks used in previous research (e.g., Bottger & Yetton, 1988; Ferrin & Dirks, 2003; Littlepage et al., 1995) where individuals’ prioritization of items for survival in an extreme environment is compared with ratings of a survival expert.

Figure 1.

Estimation questions used in each experimental task for Study 1.

A second consideration is that in some decision-making tasks (e.g., tasks that benefit from intuition or subconscious processing), formal reflection may have little effect on participants’ performance or may, in some cases, actually hurt performance (Dane & Pratt, 2007; Dijksterhuis & van Knippenberg, 1998). Thus, we selected tasks where conscious, deliberate reflection is likely to positively influence learning from group discussion (Littlepage et al., 1995; Schultze, Mojzisch, & Schulz-Hardt, 2012). Success in estimation tasks, for instance, depends on the degree to which individuals are able to build bridges between what they already know and the question at hand (Bonner & Sillito, 2011). For instance, individuals may not know the minimum freeway driving distance from Philadelphia to Chicago, but they may know that it takes about 6 hr to drive from Philadelphia to Pittsburgh, which they in turn estimate to be roughly halfway to Chicago. By reflecting on the information that they already know, individuals can improve the accuracy of their estimates (Bonner & Bolinger, 2013). Similarly, reflection can facilitate performance in survival tasks by stimulating thought about broader strategic decisions about what course of action to pursue (e.g., whether to stay at the site of an accident and wait for help or strike out on foot for the nearest town) before prioritizing each of the individual items (Ferrin & Dirks, 2003; Marcic, 1995).

In addition to seeking tasks that have objective standards and offer potential value from reflection, we also sought greater generalizability of our results by selecting tasks that complemented each other with different levels of complexity. Different levels of complexity are important because researchers have suggested that reflection may be more beneficial when the task is more complex or less well-defined (Facchin & Tschan, 2006; West, 1996). Campbell (1984, 1988) identifies several elements that make a task more complex to participants, including multiple decision alternatives and interrelationships among subtasks. Whereas estimation tasks involve educated guesses about a single point estimate, survival tasks are more complex because the decisions that individuals make about prioritizing the relative importance to their survival of any given item influence the rankings of all other items (e.g., “Is a can of Crisco more important than a bar of chocolate?”), requiring that participants coordinate strategic priorities across a series of individual decisions (Ferrin & Dirks, 2003). Our research strategy in Studies 1 and 2 proceeds, then, from examining the effects of group reflection on individual learning in decision-making tasks that progress from less to more complex and from subject matter more closely related to individuals’ day-to-day experiences (e.g., driving distances, university enrollments) to decisions in more extreme, less familiar survival contexts.

Study 1

Participants

A total of 150 students enrolled in undergraduate courses at an East Coast university participated in Study 1 as part of their coursework. The mean age of the participants was 21.6 years (SD = 5.0 years) and the sample was 38.5% female. Participants were told that they would be taking part in a decision-making activity and were provided with an informed consent form before deciding whether to participate in this research.

Procedure

Participants were randomly assigned to experimental conditions. They began by completing Task 1, the first set of five estimation questions (e.g., “What is the minimum freeway driving distance from Philadelphia to Chicago?”). After answering the first set of questions, participants were given the correct answers (i.e., feedback) and were asked to take 10 min to reflect on Task 1. It was expected that the feedback would prompt reflection on the estimation strategies used by the individual and thus a reason to reconsider estimation strategies for Task 2. As part of the reflection, they were asked about the question(s) they felt most confident answering and the question(s) they felt least confident answering, and why. Participants in the reflect in groups condition engaged in the 10-min period of reflection in randomly assigned groups of three, whereas participants in the reflect alone condition spent the 10-min period reflecting alone. Participants took part in the study in sessions of up to 35 people, with each session consisting of only one of the experimental conditions.

Following the 10-min period for reflection, participants were given a second set of five estimation questions to answer individually. Modeled on the format used in previous estimation research (e.g., Bonner & Bolinger, 2013; Bonner & Sillito, 2011), the questions in Task 2 were different in content but parallel in form to the questions in Task 1 (e.g., “What is the minimum freeway driving distance from Philadelphia to Atlanta?”), as illustrated in Figure 1. After completing the second set of five questions, participants were asked whether they had learned anything from Task 1 that they had been able to apply to Task 2. The exercise concluded with a participant debriefing.

We were sensitive to the possibility that participants in the group reflection condition might fail to interact with others or engage in meaningful reflection. Accordingly, we limited the number of participants per session so that we could actively observe the groups and monitor their conversations. Similarly, individuals reflecting alone could fail to stay on topic and not adequately reflect on the task. Our presence in and around the room was used as a visual reminder of their immediate task. The post-task questionnaire explored the nature of their thought processes as well, including questions about whether their answers were altered by the group interaction and discussion, and provided additional evidence of effort to learn from Task 1.

Measures

Cognitive learning

Cognitive learning was measured in terms of improvements in performance from Task 1 to Task 2. This metric is consistent with conceptualizations of cognitive learning as a measure of gains in knowledge that is assessed by an outside source (e.g., grades or corporate performance metrics and ratings; Kraiger, Ford, & Salas, 1993; Sitzmann et al., 2010).

Measuring improvements in task performance involved three steps. First, the absolute percentage difference of the participant’s estimate from the true answer was calculated for each of the five questions separately for Tasks 1 and 2. This approach controls for differences in scale among the five estimation questions. As an example, if the participant estimated the distance between two cities to be 110 miles and the correct answer is 100 miles, then the percentage difference is the absolute value of 110 − 100 divided by the correct answer (in this case, 100), for an outcome of .10. (Because participants’ estimates can be greatly different from the correct answer, there is no preconceived upper bound.) Second, the percentage difference between the participant’s answer and the correct answer for the first question in Task 1 was subtracted from the percentage difference for the first question in Task 2 to measure the improvement (or decline) in accuracy between Task 1 and Task 2. The improvement scores were calculated for each of the five questions. Finally, the average difference in the participant’s accuracy from Task 1 to Task 2 across the five questions was calculated as the overall measure of cognitive learning.

Self-assessment of learning

The second dependent variable, participants’ own perception of learning, was measured with a categorical (i.e., yes or no) response to whether they had learned anything from Task 1 that they had been able to apply to Task 2. Evaluating subjective learning in this dichotomous form was intended to capture the participant’s gut-level assessment of whether he or she had learned, a valuable initial impression that can be lost in a more complex question.

Results

We began by assessing whether there were significant improvements in cognitive learning between Task 1 and Task 2 for the individuals in the reflect in group condition. A one-sample t test, in which we compare improvement level to a point value of 0, showed that the mean improvements in the performance of individuals who reflected in a group were marginally significantly greater than 0, t(71) = 1.83, p = .07, suggesting marginally significant support for H1. For individuals reflecting alone, there was no evidence of significant change in performance, t(80) = −0.58, p > .05. (That is, the distribution functions of performance in the two tasks were too similar to claim any change from Task 1 to Task 2.)

For H2, we compared the change in performance for two separate groups of respondents, that is, whether cognitive learning for those who reflected in groups was indeed greater than for those who reflected alone. Hence, a one-way ANOVA was conducted. Individuals who reflected in groups experienced a small improvement in performance from the first task to the second task (M = 0.37, SD = 0.67) and individuals who reflected alone had no improvement in performance (M = −0.52, SD = 0.71). The ANOVA results show that the difference between the two groups was not significant, F(1, 149) = .826, p > .05, so H2 was not supported. The distribution of our cognitive learning measure for individuals who reflected alone was sufficiently wide so as to encompass the mean of the distribution of those who reflected in groups. We discuss this finding and its implications further in the “General Discussion” section.

Because our measure of perceived learning was dichotomous (1 = yes, 0 = no), we ran a binary logistic regression to determine the effect of the reflection condition on self-assessed learning. The results of our analysis demonstrated that individuals who reflected alone were significantly more likely to perceive that they had learned from the first task (M = 0.75, SD = 0.05) than were individuals who reflected in groups (M = 0.24, SD = 0.05), β = −2.254, p < .01, odds ratio (OR) = 0.105. Specifically, the OR indicates that individuals who reflected alone were approximately 10% more likely to perceive that they had learned from reflecting on the first task than were individuals who reflected in groups, which supported our prediction in H3.

Discussion

Although the results for two of our hypotheses in Study 1 were significant or marginally significant, our hypothesis that individuals reflecting in groups will experience greater cognitive learning than individuals who reflect alone was not supported. One possibility for explaining this finding is that the task used in Study 1 did not stimulate participants to reflect as much as they would with a task that is more complex. Previous research suggests that more complex tasks are likely to stimulate greater reflection (Facchin & Tschan, 2006; West, 1996). When confronted with multiple decisions that require broader decisions about overall strategic direction in an unfamiliar context, individuals may be less confident in their own ability and more likely to engage in sustained reflection to make sense of what is required of them (Louis, 1980). Thus, we used survival tasks (Marcic, 1995) in Study 2 to investigate the effects of reflecting alone or in a group on cognitive learning and self-assessed learning.

Study 2

Participants

A total of 146 students enrolled in undergraduate courses at an East Coast university participated in Study 2 as part of their coursework. The mean age of participants was 21.6 years old (SD = 4.1 years). The sample was 33.3% female.

Procedure

Study 2 followed the same experimental procedure described in Study 1, but used a pair of survival decision scenarios drawn from previous research (Ashburn-Nardo & Johnson, 2008; Dovidio, Gaertner, & Validzic, 1998). Task 1 involves a Winter Survival decision-making scenario (Johnson & Johnson, 1987), in which participants are stranded in the tundra of northern Canada in the middle of January as survivors of the crash of a small airplane. They are 20 miles from the nearest town and they are able to recover 12 items from their plane. Their task is to rank the items in order of usefulness to their survival. Task 2 involves the NASA Moon Survival exercise (Hall & Watson, 1970), where participants are stranded on the moon and must decide how to rank, in order of usefulness to their survival, the items available to them. We did not pre-test the equivalency of the two tasks in Study 2, but previous research has used the Winter Survival exercise and the NASA Moon Survival activity as equivalent decision-making tasks (e.g., Mayer, Sonodas, & Gudykunst, 1997). The instructions for these two survival decision-making tasks are included in Figure 2.

Figure 2.

Text of each experimental task used in Study 2.

After completing Task 1, participants were given feedback in the form of the rankings made by a survival expert and were asked to take 10 min to reflect on their answers. Participants in the reflect in groups condition were asked to engage in the 10-min period of reflection in randomly assigned groups of three, whereas participants in the reflect alone condition were asked to reflect alone. Specifically, participants were asked to reflect about the items they felt most confident in ranking and the items they felt least confident ranking and why. Following the period for reflection, participants completed Task 2 individually and then were debriefed.

As in Study 1, we checked that participants in the group reflection conditions were engaging in interactive reflection through the observations of the experimenters during the activity and from informal feedback during the post-exercise debrief. In addition, a subset of groups from one session of group reflections was audio recorded. (There were no significant differences in performance for individuals in groups who were recorded versus those who were not, F = .121, p > .05, nor in their incidence of perceived learning, χ² = .056, p > .05.) We observed that the groups engaged in on-topic reflection during the majority of the reflection period. The typical conversational pattern from our analysis of the recordings began with participants sharing their individual rankings of survival items and then collectively reviewing the expert’s answers, commenting on and discussing the (often surprising) reasons for those answers as they went. We discuss the content of the group reflections in further detail in the “Reflection Recordings” section below.

Measures

The same independent and dependent variables used in Study 1 were assessed in Study 2. Perceived learning is once again measured by the categorical response to the question of strategies learned from Task 1 for Task 2. Due to the nature of the survival decision-making task, however, cognitive learning was measured differently in Study 2. Survival tasks involve generating rankings rather than answering five discrete questions, and the participants’ responses were compared with those of a wilderness expert (Task 1) or NASA expert (Task 2). Performance was operationalized as the degree of consistency between the rankings of the participant and the expert using Krippendorff’s alpha (Hayes & Krippendorff, 2007). Consistency (i.e., match) between rankings reflects the same concerns as in content analysis where inter-coder (or inter-rater) reliability is sought. In that context, two or more individuals record their assessment of text or visual content using a pre-determined and understood set of instructions and coding guide, much like the problem definition in our case.

Krippendorff’s alpha is widely used as a measure of inter-coder reliability in content analysis (e.g., Lombard, Snyder-Duch, & Bracken, 2002) due to its rigor and flexibility. Compared with other measures such as Cohen’s kappa or percent agreement, Krippendorff’s alpha is suitable for any number of participants, any type of data, and has a clear meaning to its own scale (Hayes & Krippendorff, 2007). It also takes into account the possibility of chance agreement. As a result, it is regarded as a relatively conservative measure of consistency. In our case, this means that a finding of significant differences in performance is even more meaningful. We calculated Krippendorff’s alpha for each survival exercise separately as a measure of how well the participants did compared with the expert’s rankings of the items. Higher values indicate a higher degree of consistency between participant and expert, with a score of 1.0 indicating perfect consistency. Then, we used the difference in scores between Task 1 and Task 2 as the measure of cognitive learning, with a positive difference indicating an improvement from Task 1 to Task 2.

Results

As in Study 1, to assess whether there were significant improvements in cognitive learning between Task 1 and Task 2 for individuals who reflected in a group, we used a one-sample t test (to compare their level of improvement to a point value of zero). The result showed that the differences in the performance of individuals who reflected in a group were significantly greater than 0, t(72) = 6.20, p < .01, supporting H1. Individuals who reflected alone also realized improvements from Task 1 to Task 2 that were significantly different, t(72) = 3.82, p < .01.

We again conducted an ANOVA to determine the effect of the reflection condition on cognitive learning (H2), that is, whether group reflection yielded a different change in learning than did individual reflection. There was not a significantly greater improvement in learning among those who reflected in groups (M = 0.12, SD = 0.02) than among individuals who reflected alone (M = 0.08, SD = 0.02), F(1, 145) = 1.30, p > .05, so H2 was not supported. We then examined whether the perceptions of learning of individuals who reflected alone (M = 0.62, SD = 0.06) differed significantly from individuals who reflected in groups (M = 0.33, SD = 0.06). A binary logistic regression to determine the effect of the reflection condition on individuals’ perceptions of learning showed a statistically significant difference (β = −1.181, p < .01, OR = 0.307). Specifically, individuals who reflected alone were nearly 31% more likely to perceive that they had learned from reflecting on the first task than were individuals who reflected in groups, which supported H3.

Reflection Recordings

As described earlier, we audio recorded a subset of the group reflection sessions as an informal check to ensure that participants in the group reflection condition were actually engaging in on-topic discussions about the task at hand. In this section, we explore the content of those discussions to characterize the process of reflecting with a group and thereby view our findings through a different lens. The content of the recordings (described below) indicates that the participants in the group reflection sessions were generally on-task.

A hermeneutic approach (Thompson, Locander, & Pollio, 1989) was used to evaluate the content of the recordings. Accordingly, themes were developed both within and across participants’ texts. The process was iterative, yielding both expansion and contraction of the theme set until no additional insights were obtained. Although we did not make any a priori hypotheses about the effects of differences in content on participants’ cognitive and subjective perceptions of learning, we were able to deduce three distinct levels of content in participants’ reflection in groups, which are summarized in Figure 3.

Figure 3.

Overview of the different levels of content observed in participants’ reflections on the experimental tasks.

The first level of content we observed was descriptive in nature, where participants simply shared their individual perspectives without further analysis or comparison with one another. In some cases, this sharing could be described as talking at rather than talking with others in the group. For the subset of group reflections that we recorded, descriptive reflection occurred in every group but for varying amounts of time, sometimes as isolated statements by an individual (e.g., “I put the empty cigarette lighter last”) and sometimes in conversation with others (e.g., Person 1: “I put the extra pair of clothes first.” Person 2: “Oh, I put the pistol first.”). Descriptive statements involved little analytical depth and only shallow explanations for why participants chose to rank items in a particular way.

The second level of reflection we observed was benchmarking, in which individuals used the expert’s answers or the answers of others in the group to evaluate their own perspectives. We observed benchmarking in all but one group. Benchmarking was usually preceded by descriptive reflection, where participants first became familiar with each other’s rationale and then began to question why theirs was different. However, benchmarking began in earnest when participants read the expert’s answers to the survival scenario. The expert’s credibility appeared to anchor the participants’ thoughts and provide them grounds for reexamining their own perspectives (e.g., “Wow, I didn’t think of that. The whiskey would be too cold to drink. I would’ve died.”). Benchmarking reflection involved greater depth of analysis than descriptive reflection because participants who engaged in benchmarking actively compared and contrasted their rankings with those of other group members or the expert. However, groups whose members engaged in only descriptive and benchmarking reflection did not discuss take-away lessons that could be applied to the second task.

The third level of reflection we observed was meta-strategic reflection, whereby participants generated overall strategies or lessons that they learned from the first task that could then be applied in a future task. This big picture thinking occurs when participants step back from the rankings of individual items to consider why certain groups of items are ranked higher than others. In the survival scenario, this generally involved a broad decision about the top strategic priority (i.e., whether to build a fire and wait for help vs. striking out for the nearest town). Although groups of individuals who engaged in meta-strategic reflection demonstrated the greatest depth of analysis of the three levels of reflection that we observed, only some of the groups of participants reached this level. This is consistent with -Mayer’s (2009) suggestion that meta-strategies represent higher level forms of knowledge that should be considered in evaluating learning in cognitively complex tasks.

The qualitative data drawn from the audio recordings provide some evidence of how the content of reflection can vary in interacting groups. Based on our preliminary analyses, we agree with others (e.g., Gurtner et al., 2007; Moreland & McMinn, 2010) that additional research is needed that not only investigates whether or how much groups engage in reflection but also explores the content (i.e., what group members discuss) of group reflection. In particular, additional research could build on the levels of reflection we have identified to examine how the level of reflection a group reaches (e.g., whether group members discuss meta-strategies) influences both objective and subjective measures of individual group members’ learning. We discuss specific suggestions for future research in the next section.

General Discussion

The Value of Group Reflection

The purpose of this research was to investigate how reflecting in a group affects both subjective perceptions and objective measures of individual learning. In both Studies 1 and 2 and consistent with our predictions, we found that individuals who reflected in a group experienced improvements in cognitive learning from Task 1 to Task 2, in support of H1. However, individuals who reflected in a group did not experience greater improvements in cognitive learning than individuals who reflected alone (H2). In addition, the individuals who reflected alone perceived that they learned more than the individuals who reflected in groups in support of H3. Analysis of the content of group reflection suggests that these findings may stem from differences in the quality and extent of cognitive effort undertaken by group members.

Our results speak to the literature on collective reflection that spans multiple disciplines, ranging from education and psychology to health care and organizational studies (e.g., Delany & Watkin, 2009; Quinn & Bunderson, 2013; Sawyer & Berson, 2004). In particular, we examined the effect of a formal, open-ended group reflection intervention relative to an individual reflection intervention on two types of individual learning outcomes: objective performance improvements and subjective evaluations of learning. Our findings highlight how individuals’ subjective evaluations of whether they learned can deviate from objective indicators of their learning.

The potential misalignment between subjective evaluations and objective measures of learning has consequences for both individuals and organizations. On one hand, individuals who underestimate the actual value of reflecting in groups may prematurely disengage from or leave the group, thereby forfeiting valuable opportunities to improve their performance. Conversely, individuals who believe that they have learned more from group reflection than objective assessments of their learning warrant may overestimate their capabilities unless confronted by objective evidence (Kruger & Dunning, 1999). Overestimation of self-assessed knowledge is a potential barrier to subsequent learning and improvement because individuals do not know how much they do not know (Dunning, Johnson, Ehrlinger, & Kruger, 2003). Our results offer further evidence of how subjective measures and objective measures of learning may differ, which underscores the importance of assessing subjective and objective measures of learning concurrently rather than assuming that one is an accurate proxy of the other (Sitzmann et al., 2010).

The finding that individuals who reflected in groups did not demonstrate significantly greater performance improvements than individuals who reflected alone was contrary to our predictions. However, it is also not entirely inconsistent with some recent studies that have found no effect or even negative effects associated with group reflection (Moreland & McMinn, 2010). Contrary to their expectation that reflection in groups would be superior to individual reflection, Gurtner et al. (2007) find that it was individual reflection that facilitated performance on a computer-simulated military air-surveillance task. De Dreu (2007) finds that collective reflection actually has a marginally negative correlation with team effectiveness as rated by supervisors. Understanding the conditions that moderate the effects of group reflection on learning outcomes is a priority for future research.

The Content of Group Reflection

As indicated above, one potential moderator of the relationship between group reflection and learning outcomes is the content of the reflection itself (Sutton & Dalley, 2008). Many of the earlier studies of group reflection (or group reflexivity; West, 1996) involve case studies in which group reflection is self-reported after the fact, so the actual content of the reflection cannot be observed (Moreland & McMinn, 2010). Among more recent studies where group reflection has been found to facilitate performance (e.g., S. Ellis & Davidi, 2005), the process of reflection is often carefully structured such that participants must respond to a series of specific questions rather than engage in open-ended reflection. In fact, post hoc analysis by Gurtner et al. (2007) suggests that it is what participants fail to talk about in unstructured reflection that hinders the effectiveness of group reflection. Specifically, they find that the task strategies discussed among participants in the groups in their study were so general that they hindered individual learning.

In our case, we can point to three distinct levels of reflection occurring among individuals reflecting in groups: descriptive, benchmarking, and meta-strategic. Although additional research is needed to verify and expand on these levels, they illustrate differences in reflection content that can occur across groups, even when all of those groups are engaged in interactive reflection, which may influence the extent of individual learning from reflection. For instance, groups of participants that engaged in only descriptive reflection did not generate lessons for use in future tasks. Groups that engaged in benchmarking, in contrast, compared their knowledge with an external source (i.e., the correct answer in Study 1 or the survival expert in Study 2), which provided a standard against which they could recalibrate their thinking and anchor their understanding. Meta-strategic reflection involved searching for broader lessons from the feedback that participants received, which they could then generalize and apply to a subsequent, related task. We suggest that systematically investigating the variance in the content of the group reflection, and not just examining whether reflection occurs, has potential to facilitate understanding of the inconsistencies in learning from group reflection.

Directions for Future Research

The discrepancy between individuals’ subjective perceptions of whether they learned from reflecting in a group and measures of their cognitive learning has implications for the performance of not only individuals but also the groups with whom they work. For example, individuals’ subjective perceptions of the efficacy of their experiences may influence the amount of time and effort they invest in working with others, both presently and in the future (Cameron & Whetten, 1981). Specifically, individuals who have negative experiences with a group may carry their negative perceptions forward to future group experiences, which may come at the expense of groups with whom they work later on (Hackman, 1987; O’Leary et al., 2011). Thus, there may be some circumstances where reflecting in groups could be harmful rather than productive (Moreland & McMinn, 2010).

An important avenue for future research, then, is to explore potential modifications to group reflection interventions to narrow the gap between individuals’ perceived and actual learning from reflecting in groups. In particular, we would suggest that exploring variables associated with the structure of the intervention used to stimulate group reflection, along with providing groups with guidance from a trained facilitator, may substantially influence group outcomes. For instance, it may be that the timing of reflection (i.e., during a task or after the task is completed) influences individual learning (Rasker et al., 2000). Future research could also explore how the degree of familiarity group members have with one another may influence the volume and quality of information discussed in group reflection (Okhuysen, 2001). The studies reported in this article involved groups without prior history and with little expectation of future interaction, but individuals may place different value on reflection in groups with individuals whom they know or identify with.

Systematic consideration should also be given to the questions used to elicit individual and group reflections. Although most reflection interventions ask individuals to think about strategies for improving their performance (e.g., Gurtner et al., 2007), little research has considered whether the types of questions are likely to yield different forms of reflection and whether different types of reflection (e.g., reflection on task strategies versus reflecting on the quality of the group’s interactions) are more or less valuable in facilitating performance (Moreland & McMinn, 2010; Sutton & Dalley, 2008). It may be that more specific, concrete questions (“What were two specific strategies that you could use to improve in the future?”) are more useful than open-ended, general questions (e.g., “What are some suggestions for how you could improve task performance in the future?”). Researchers could potentially use counterfactual reflection (Kray, Galinsky, & Markman, 2009), for example, to engage participants in cognitively elaborating on what would have happened had they used a different strategy.

Finally, future research could also examine matching the questions used in a reflection intervention with different types of tasks. For example, specific reflection interventions may be more or less relevant for tasks of different degrees of complexity and ambiguity (e.g., Facchin & Tschan, 2006) or based on the degree to which the task requires innovativeness and creativity (Tjosvold, Tang, & West, 2004). The questions we have outlined constitute an ambitious agenda for future research. However, given the potential value of reflection as an intervention for facilitating individual learning, we believe that such efforts are warranted.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Alexander R. Bolinger is an assistant professor of management at Idaho State University, USA. His research interests include group work and its effects on group members, negotiation, and service work. His research has also appeared in Organizational Behavior and Human Decision Processes and Cornell Hospitality Quarterly.

Julie V. Stanton is an associate professor of business at The Pennsylvania State University, USA. Her research explores the grouping mechanisms that facilitate small producer participation in export-oriented distribution channels, including cooperatives. Her research has also appeared in Supply Chain Management and World Development, among other outlets.

References

Association to Advance Collegiate Schools of Business International. (2007). AACSB assurance of learning standards: An interpretation. Retrieved from http://www.aacsb.edu/resource_centers/assessment

Aguinis

Kraiger

(2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451-474. doi:10.1146/annurev.psych.60.110707.163505

Anseel

Lievens

Schollaert

(2009). Reflection as a strategy to enhance task performance after feedback. Organizational Behavior and Human Decision Processes, 110, 23-35. doi:10.1016/j.obhdp.2009.05.003

Arbaugh

J. B.

(2005). How much does “subject matter” matter? A study of disciplinary effects in on-line MBA courses. Academy of Management Learning & Education, 4, 57-73. doi:10.5465/AMLE.2005.16132549

Ashburn-Nardo

Johnson

N. J.

(2008). Implicit outgroup favoritism and intergroup judgment: The moderating role of stereotypic context. Social Justice Research, 21, 490-508. doi:10.1007/s11211-008-0078-8

Bacon

D. R.

(2005). The effects of group projects on content-related learning. Journal of Management Education, 29, 248-267. doi:10.1177/1052562904263729

Bacon

D. R.

Stewart

K. A.

Silver

W. S.

(1999). Lessons from the best and worst student team experiences: How a teacher can make a difference. Journal of Management Education, 23, 467-488. doi:10.1177/105256299902300503

Barry

C. A.

Britten

Barber

Bradley

Stevenson

(1999). Using reflexivity to optimize teamwork in qualitative research. Qualitative Health Research, 9, 26-44. doi:10.1177/104973299129121677

Benbunan-Fich

Hiltz

S. R.

(2003). Mediators of the effectiveness of online courses. IEEE Transactions on Professional Communication, 46, 298-312. doi:10.1109/TPC.2003.819639

10.

Bonner

B. L.

Baumann

M. R.

(2012). Leveraging member expertise to improve knowledge transfer and demonstrability in groups. Journal of Personality and Social Psychology, 102, 337-350. doi:10.1037/a0025566

11.

Bonner

B. L.

Baumann

M. R.

Dalal

(2002). The effects of member expertise on group decision making and group performance. Organizational Behavior and Human Decision Processes, 88, 719-736. doi:10.1016/S0749-5978(02)00010-9

12.

Bonner

B. L.

Bolinger

A. R.

(2013). Separating the confident from the correct: Leveraging member knowledge in groups to improve decision making and performance. Organizational Behavior and Human Decision Processes, 122, 214-221. doi:10.1016/j.obhdp.2013.07.005

13.

Bonner

B. L.

Sillito

S. D.

(2011). Leveraging member knowledge in groups: Expertise, extroversion, and feedback. Group Dynamics: Theory, Research, and Practice, 15, 233-245. doi:10.1037/a0022735

14.

Bonner

B. L.

Sillito

S. D.

Baumann

M. R.

(2007). Collective estimation: Accuracy, expertise, and extroversion as sources of intra-group influence. Organizational Behavior and Human Decision Processes, 103, 121-133. doi:10.1016/j.obhdp.2006.05.001

15.

Borredon

Deffayet

Baker

A. C.

Kolb

(2011). Enhancing deep learning: Lessons from the introduction of learning teams in management education in France. Journal of Management Education, 35, 324-350. doi:10.1177/1052562910368652

16.

Bottger

Yetton

(1988). An integration of process and decision scheme explanations of group problem solving performance. Organizational Behavior and Human Decision Processes, 42, 234-249. doi:10.1016/0749-5978(88)90014-3

17.

Bowman

N. A.

(2010). Assessing learning and development among diverse college students. New Directions for Institutional Research, 145, 53-71. doi:10.1002/ir.322

18.

Cameron

K. S.

Whetten

D. A.

(1981). Perceptions of organizational effectiveness over organizational life cycles. Administrative Science Quarterly, 26, 525-544. doi:10.2307/2392338

19.

Campbell

D. J.

(1984). The effects of goal-contingent payment on the performance of a complex task. Personnel Psychology, 37, 23-40. doi:10.1111/j.1744-6570.1984.tb02245.x

20.

Campbell

D. J.

(1988). Task complexity: A review and analysis. Academy of Management Review, 13, 40-52. doi:10.5465/AMR.1988.4306775

21.

Carter

S. M.

West

M. A.

(1998). Reflexivity, effectiveness and mental health in BBC-TV production teams. Small Group Research, 29, 583-601. doi:10.1177/1046496498295003

22.

Chapman

K. J.

Meuter

M. L.

Toy

Wright

L. K.

(2009). Are student groups dysfunctional? Perspectives from both sides of the classroom. Journal of Marketing Education, 32, 39-49. doi:10.1177/0273475309335575

23.

Cole

M. S.

Walter

Bruch

(2008). The affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study. Journal of Applied Psychology, 93, 945-958. doi:10.1037/0021-9010.93.5.945

24.

Cooper

M. M.

Cox

Nammouz

Case

Stevens

(2008). An assessment of the effect of collaborative groups on students’ problem-solving strategies and abilities. Journal of Chemical Education, 85, 866-871. doi:10.1021/ed085p866

25.

Dane

Pratt

M. G.

(2007). Exploring intuition and its role in managerial decision making. Academy of Management Review, 32, 33-54. doi:10.5465/AMR.2007.23463682

26.

Daudelin

M. W.

(1996). Learning from experience through reflection. Organizational Dynamics, 24, 36-48. doi:10.1016/s0090-2616(96)90004-2

27.

De Dreu

C. K. W.

(2007). Cooperative outcome interdependence, task reflexivity, and team effectiveness: A motivated information processing perspective. Journal of Applied Psychology, 92, 628-638. doi:10.1037/0021-9010.92.3.628

28.

Delany

Watkin

(2009). A study of critical reflection in health profession education: “Learning where others are coming from.” Advances in Health Science Education, 14, 411-429. doi:10.1007/s10459-008-9128-0

29.

Dijksterhuis

van Knippenberg

(1998). The relation between perception and behavior, or how to win a game of Trivial Pursuit. Journal of Personality and Social Psychology, 74, 865-877. doi:10.1037/0022-3514.74.4.865

30.

Dovidio

J. F.

Gaertner

S. L.

Validzic

(1998). Intergroup bias: Status, differentiation, and common in-group identity. Journal of Personality and Social Psychology, 75, 109-120. doi:10.1037/0022-3514.75.1.109

31.

Dunning

Heath

Suls

(2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69-106. doi:10.1111/j.1529-1006.2004.00018.x

32.

Dunning

Johnson

Ehrlinger

Kruger

(2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12, 83-87. doi:10.1111/1467-8721.01235

33.

Edmondson

(1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44, 350-383. doi:10.2307/2666999

34.

Ellis

A. P. J.

Bell

B. S.

(2005). Capacity, collaboration, and commonality: A framework for understanding team learning. In Neider

L. L.

Schrieshiem

(Eds.), Understanding teams (pp. 1-26). Greenwich, CT: Information Age.

35.

Ellis

Davidi

(2005). After-event reviews: Drawing lessons from failed and successful events. Journal of Applied Psychology, 90, 857-871. doi:10.1037/0021-9010.90.5.857

36.

Erbert

L. A.

Mearns

G. M.

Dena

(2005). Perceptions of turning points and dialectical interpretations in organizational team development. Small Group Research, 36, 21-58. doi:10.1177/1046496404266774

37.

Facchin

Tschan

(2006, July). Team reflexivity and team effectiveness: The moderating effect of task type and decision latitude. Poster presented at Interdisciplinary Network for Group Research Conference, Pittsburgh, PA.

38.

Ferrin

D. L.

Dirks

K. T.

(2003). The use of rewards to increase and decrease trust: Mediating processes and differential effects. Organization Science, 14, 18-31. doi:10.1287/orsc.14.1.18.12809

39.

Foldy

E. G.

Buckley

T. R.

(2010). Re-creating street-level practice: The role of routines, work groups and team learning. Journal of Public Administration Research and Theory, 20, 23-52. doi:10.1093/jopart/mun034

40.

Frese

Zapf

(1994). Action as the core of work psychology: A German approach. In Triandis

H. C.

Dunnette

M. D.

Hough

L. M.

(Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 4, pp. 271-340). Palo Alto, CA: Consulting Psychologists Press.

41.

George

J. M.

(2011). Asymmetrical effects of rewards and punishments: The case of social loafing. Journal of Occupational and Organizational Psychology, 68, 327-338. doi:10.1111/j.2044-8325.1995.tb00591.x

42.

Gordon

Smith Hullfish

P. H.

(1961). Reflective thinking: The method of education. New York, NY: Dodd, Mead.

43.

Gurtner

Tschan

Semmer

N. K.

Nägele

(2007). Getting groups to develop good strategies: Effects of reflexivity interventions on team process, team performance, and shared mental models. Organizational Behavior and Human Decision Processes, 102, 127-142. doi:10.1016/j.obhdp.2006.05.002

44.

Hackman

J. R.

(1987). The design of work teams. In Lorsch

(Ed.), Handbook of organizational behavior (pp. 315-342). Englewood Cliffs, NJ: Prentice-Hall.

45.

Hackman

J. R.

Wageman

(2005). A theory of team coaching. Academy of Management Review, 30, 269-287. doi:10.5465/AMR.2005.16387885

46.

Hall

Watson

W. H.

(1970). The effects of a normative intervention on group decision-making performance. Human Relations, 23, 299-317. doi:10.1177/001872677002300404

47.

Hayes

A. F.

Krippendorff

(2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89. doi:10.1080/19312450709336664

48.

Hinsz

V. B.

Tindale

R. S.

Vollrath

D. A.

(1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121, 43-64. doi:10.1037/0033-2909.121.1.43

49.

Hollingshead

A. B.

(1998). Group and individual training: The impact of practice on performance. Small Group Research, 29, 254-280. doi:10.1177/1046496498292006

50.

Johnson

(1987). Joining together: Group theory and group skills. Englewood Cliffs, NJ: Prentice-Hall.

51.

Karau

S. J.

Williams

K. D.

(1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65, 681-706. doi:10.1037/0022-3514.65.4.681

52.

Kerr

N. L.

Bruun

(1983). Ringelmann revisited: Alternative explanations for the social loafing effect. Personality and Social Psychology Bulletin, 7, 224-231. doi:10.1177/014616728172007

53.

Kirkman

B. L.

Jones

R. G.

Shapiro

(2000). Why do employees resist teams? Examining the “resistance barrier” to work team effectiveness. International Journal of Conflict Management, 11, 74-92. doi:10.1108/eb022836

54.

Kraiger

Ford

Salas

(1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328. doi:10.1037/0021-9010.78.2.311

55.

Kray

L. J.

Galinsky

A. D.

Markman

K. D.

(2009). Counterfactual structure and learning from experience in negotiations. Journal of Experimental Social Psychology, 45, 979-982. doi:10.1016/j.jesp.2009.03.008

56.

Kruger

Dunning

(1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134. doi:10.1037/0022-3514.77.6.1121

57.

Kyndt

Raes

Lismont

Timmers

Cascallar

Dochy

(2013). A meta-analysis of the effects of face-to-face cooperative learning. Do recent studies falsify or verify earlier findings? Educational Research Review, 10, 133-149. doi:10.1016/j.edurev.2013.02.002

58.

Laughlin

P. R.

Carey

H. R.

Kerr

N. L.

(2008). Group-to-individual problem-solving transfer. Group Processes & Intergroup Relations, 11, 319-330. doi:10.1177/1368430208090645

59.

Lim

D. H.

Morris

M. L.

(2006). Influence of trainee characteristics, instructional satisfaction, and organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17, 85-115. doi:10.1002/hrdq.1162

60.

Littlepage

G. E.

Robison

Reddington

(1997). Effects of task experience and group experience on group performance, member ability, and recognition of expertise. Organizational Behavior and Human Decision Processes, 69, 133-147. doi:10.1006/obhd.1997.2677

61.

Littlepage

G. E.

Schmidt

G. W.

Whisler

E. W.

Frost

A. G.

(1995). An input-process-output analysis of influence and performance in problem-solving groups. Journal of Personality and Social Psychology, 69, 877-889. doi:10.1037/0022-3514.69.5.877

62.

Lombard

Snyder-Duch

Bracken

C. C.

(2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587-604. doi:10.1093/hcr/28.4.587

63.

Louis

M. R.

(1980). Surprise and sense making: What newcomers experience in entering unfamiliar organizational settings. Administrative Science Quarterly, 25, 226-251. doi:10.2307/2392453

64.

MacCurtain

Flood

P. C.

Ramamoorthy

West

M. A.

Dawson

(2010). Top management team knowledge sharing and new product performance: A study of the Irish software industry. Creativity and Innovation Management, 19, 219-232. doi:10.1111/j.1467-8691.2010.00564.x

65.

Marcic

(1995). Organizational behavior: Experiences and cases. Minneapolis, MN: West Publishing.

66.

Marks

M. A.

Mathieu

J. E.

Zaccaro

S. J.

(2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26, 356-376. doi:10.5465/AMR.2001.4845785

67.

Mathieu

J. E.

Heffner

T. S.

Goodwin

G. F.

Cannon-Bowers

J. A.

Salas

(2005). Scaling the quality of teammates’ mental models: Equifinality and normative comparisons. Journal of Organizational Behavior, 26, 37-56. doi:10.1002/job.296

68.

Mayer

M. E.

Sonodas

K. T.

Gudykunst

W. B.

(1997). The effect of time pressure and type of information on decision quality. Southern Communication Journal, 62, 280-292. doi:10.1080/10417949709373065

69.

Mayer

R. E.

(2001). Cognitive, metacognitive, and motivational aspects of problem solving. In Hartman

H. J.

(Ed.), Metacognition in learning and instruction (pp. 87-101). Norwell, MA: Kluwer Academic. doi:10.1007/978-94-017-2243-8_5

70.

Mayer

R. E.

(2009). Advances in specifying what is to be learned: Reflections on the themes in Chapters 6-8. In Ericsson

K. A.

(Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 203-214). Cambridge, UK: Cambridge University Press. doi:10.1017/cbo9780511609817.011

71.

Mezirow

(2003). Transformative learning as discourse. Journal of Transformative Education, 1, 58-63. doi:10.1177/1541344603252172

72.

Moreland

R. L.

McMinn

J. G.

(2010). Group reflexivity and performance. In Thye

S. R.

Lawler

(Eds.), Advances in group processes (Vol. 27, pp. 63-95). Bingley, UK: Emerald Press. doi:10.1108/S0882-6145(2010)0000027006

73.

Morgan

R. B.

Casper

W. J.

(2000). Examining the factor structure of participant reactions to training: A multidimensional approach. Human Resource Development Quarterly, 11, 301-317. doi:10.1002/1532-1096(200023)11:3<301::AID-HRDQ7>3.0.CO;2-P

74.

Okhuysen

G. A.

(2001). Structuring change: Familiarity and formal interventions in problem-solving groups. Academy of Management Journal, 44, 794-808. doi:10.2307/3069416

75.

O’Leary

M. B.

Mortensen

Woolley

A. W.

(2011). Multiple team membership: A theoretical model of its effects on productivity and learning for individuals and teams. Academy of Management Review, 36, 461-478. doi:10.5465/AMR.2011.61031807

76.

Olivera

Straus

S. G.

(2004). Group-to-individual transfer of learning: Cognitive and social factors. Small Group Research, 35, 440-465. doi:10.1177/1046496404263765

77.

Osborn

A. F.

(1957). Applied imagination. New York, NY: Scribner.

78.

Paulus

P. B.

Brown

(2003). Enhancing ideational creativity in groups: Lessons from research on brainstorming. In Paulus

P. B.

Nijstad

B. A.

(Eds.), Group creativity: Innovation through collaboration (pp. 110-137). New York, NY: Oxford University Press. doi:10.1093/acprof:oso/9780195147308.003.0006

79.

Peterson

S. E.

Miller

J. A.

(2004). Comparing the quality of students’ experiences during cooperative learning and large-group instruction. Journal of Educational Research, 97, 123-134. doi:10.3200/JOER.97.3.123-134

80.

Piaget

(1962). Plays, dreams, and imitation in the child. New York, NY: W. W. Norton.

81.

Quinn

R. W.

Bunderson

J. S.

(2013). Could we huddle on this project? Participant learning in newsroom conversations. Journal of Management. Advance online publication. doi:10.1177/0149206313484517

82.

Raelin

J. A.

(2002). “I don’t have time to think!” versus the art of reflective practice. Reflections: The SoL Journal on Knowledge, Learning, and Change, 4, 66-79. doi:10.1162/152417302320467571

83.

Rasker

P. C.

Post

W. M.

Schraagen

J. M. C.

(2000). Effects of two types of intra-team feedback on developing a shared mental model in Command & Control teams. Ergonomics, 43, 1167-1189. doi:10.1080/00140130050084932

84.

Reynolds

S. J.

(2006). A neurocognitive model of the ethical decision-making process: Implications for study and practice. Journal of Applied Psychology, 91, 737-748. doi:10.1037/0021-9010.91.4.737

85.

Rogoff

Goodman Turkanis

Bartlett

(2001). Learning together: Children and adults in a school community. New York, NY: Oxford University Press.

86.

Sawyer

R. K.

Berson

(2004). Study group discourse: How external representations affect collaborative conversation. Linguistics and Education, 15, 387-412. doi:10.1016/j.linged.2005.03.002

87.

Schippers

M. C.

Den Hartog

D. N.

Koopman

P. L.

Wienk

J. A.

(2003). Reflexivity and diversity in teams: The moderating effects of outcome interdependence and group longevity. Journal of Organizational Behavior, 24, 779-802. doi:10.1002/job.220

88.

Schultze

Mojzisch

Schulz-Hardt

(2012). Why groups perform better than individuals at quantitative judgment tasks: Group-to-individual transfer as an alternative to differential weighting. Organizational Behavior and Human Decision Processes, 118, 24-36. doi:10.1016/j.obhdp.2011.12.006

89.

Sitzmann

Ely

Brown

K. G.

Bauer

K. N.

(2010). Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning & Education, 9, 169-191. doi:10.5465/AMLE.2010.51428542

90.

Stroebe

Frey

B. S.

(2011). Self-interest and collective action: The economics and psychology of public goods. British Journal of Social Psychology, 21, 121-137. doi:10.1111/j.2044-8309.1982.tb00521.x

91.

Sutton

Dalley

(2008). Reflection in an intermediate care team. Physiotherapy, 94, 63-70. doi:10.1016/j.physio.2007.04.008

92.

Thompson

C. J.

Locander

W. B.

Pollio

H. R.

(1989). Putting consumer experience back into consumer research: The philosophy and method of existential-phenomenology. Journal of Consumer Research, 16, 133-146. doi:10.1086/209203

93.

Tjosvold

Tang

M. M. L.

West

M. A.

(2004). Reflexivity for team innovation in China: The contribution of goal interdependence. Group & Organization Management, 29, 540-559. doi:10.1177/1059601103254911

94.

Urbach

Fay

Goral

(2010). Extending the job design perspective on individual innovation: Exploring the effect of group reflexivity. Journal of Occupational and Organizational Psychology, 83, 1053-1064. doi:10.1348/096317909X479394

95.

van Ginkel

W. P.

Tindale

R. S.

Van Knippenberg

(2009). Team reflexivity, development of shared task representations, and the use of distributed information in group decision making. Group Dynamics: Theory, Research, and Practice, 13, 265-280. doi:10.1037/a0016045

96.

West

M. A.

(1996). Reflexivity and work group effectiveness: A conceptual integration. In West

M. A.

(Eds.), The handbook of work group psychology (pp. 555-579). Chichester, UK: John Wiley.

97.

Witt

P. L.

Wheeless

L. R.

(2001). An experimental study of teachers’ verbal and nonverbal immediacy and students’ affective and cognitive learning. Communication Education, 50, 327-342. doi:10.1080/03634520109379259