An Integrative Framework for Problem-Based Learning and Action Learning

Abstract

Experiential learning methods for leadership development have grown in popularity, but empirical research to support design, implementation, and ongoing evaluation has not kept pace with the demand for these programs. Problem-based learning (PBL) and action learning (AL) are two examples of these interventions used for developing agile practitioners while solving challenges found in the workplace. Addressing the complexity of these approaches, this article presents a framework that integrates PBL and AL design features to facilitate research into these similar interventions for management and leadership development. The PBL and AL literatures are compared to reveal analogous characteristics and to highlight gaps in research. The integrative framework is applied to make recommendations for human resource development practitioners conducting program evaluations.

Keywords

problem-based learning action learning framework leadership development evaluation

Designing and delivering leadership development programs are among the central functions of human resource development (HRD) practitioners (Ardichvili, och Dag, & Manderscheid, 2016; Waddill, Banks, & Marsh, 2010). Fulfilling these responsibilities means attention to both leadership development, the interpersonal processes among organization members, and leader development, the intrapersonal factors that affect individual effectiveness as a leader (Day, Fleenor, Atwater, Sturm, & McKee, 2014). As such, it is recommended that HRD practitioners design leadership development programs that can address the complex individual and social aspects of leader and leadership development (Day, 2000; Day et al., 2014). Among the key leadership development trends recently identified for HRD practitioners is the use of experience-based programs (Ardichvili et al., 2016). It is commonly said that companies are shifting their leadership development resources away from formal training events toward a “70-20-10 model” that situates the majority of learning activities in work-based, experiential interventions (O’Leonard & Loew, 2012). Experiential leadership development initiatives are cited as an essential part of leadership development (Allio, 2005; Amagoh, 2009; Dalakoura, 2010), and they seem to be particularly well suited for developing the complex skills required of leaders (Hezlett, 2016).

This article focuses on one type of experiential leadership development approach, action learning (AL) and problem-based learning (PBL), which has grown in popularity within the last few decades and is used in organizations and higher education programs around the world (Anderson & Coleman, 2015; Leonard & Lang, 2010; Marquardt & Banks, 2010; Waddill et al., 2010). According to Amagoh (2009), “Action learning is a vital component of most leadership development programs, and constitutes one of the core leadership development methods at best-practice organizations” (p. 996). Experiential pedagogical techniques like AL and PBL often are described as more effective than traditional classroom-based approaches for developing leaders and practitioners (Brownell & Jameson, 2004; Johnson & Spicer, 2006; Leonard & Lang, 2010; Tushman, Fenollosa, McGrath, O’Reilly, & Kleinbaum, 2007). Overuse of conventional didactic methods such as lecturing and teacher-centered instruction consistently tops critics’ lists of reasons why institutions have failed to develop the teamwork, leadership, and problem-solving skills necessary for managers to be successful in the 21st century (Brotheridge & Long, 2007; Feldman, 2005; Gosling & Mintzberg, 2006; Roglio & Light, 2009; Samuelson, 2006).

However, these approaches face the same criticism as other leadership development programs: Although it is assumed that they have a positive impact, their processes and outcomes have not been rigorously examined to provide evidence-based guidance for designing and continuously improving leadership development programs (Allio, 2005; Amagoh, 2009; Garavan, O’Brien, & Watson, 2015; Mehrabani & Mohamad, 2015). Despite their popularity, they lack sufficient models for evaluation (Cho & Egan, 2009; Edmonstone, 2015; Hezlett, 2016). Demonstrating the impact of experience-based leadership development programs requires a framework that facilitates design, measurement, and evaluation to answer questions about which techniques and conditions work to achieve specific learning outcomes, and why (Allio, 2005; Hezlett, 2016; Hung, 2011). As summarized by Waddill et al. (2010),

Although various research accounts have placed the business-wide return on investment from action learning as anywhere from 5 to 25 times its cost, there is little empirical research or evidence on why and how action learning works. A future challenge for action learning, therefore, is to quantify the results of action learning. This will earn action learning a place at the research table and further assert its credibility to the academic community. (p. 276)

Purpose

The purpose of this article is to propose an integrative framework that can be used to help fill this gap in our understanding of AL and PBL program design effectiveness and promote evaluation studies that can provide stronger evidence for these designs when used for leadership development. Although AL and PBL are implemented in a variety of ways, typically both refer to an approach to learning that uses an ill-structured, real-life problem or opportunity that initiates the learning process, a team working together to learn while solving the problem, learner-directed action toward problem resolution, reflection on learning experiences, and a learning process facilitator or coach (Barrows, 1980; Hmelo-Silver, 2004; Hung, Jonassen, & Liu, 2008; Marquardt, 1999; Marsick & O’Neil, 1999; Revans, 1982). Drawing from prior reviews of AL and PBL (Cho & Egan, 2009, 2010; Hmelo-Silver, 2004; Hung et al., 2008; Loyens, Kirschner, & Paas, 2012; Marsick & O’Neil, 1999; O’Neil & Marsick, 2007), Table 1 presents an outline of the structure and general approaches typically followed in AL programs delivered within organizational settings and PBL programs delivered within academic settings. Although AL and PBL originated in different fields and contexts, this study will show that they are nearly identical in their core design features. By bridging the research on AL and PBL, incorporating the extensive empirical research on PBL effectiveness in medical education and the qualitative studies on AL applications in business settings, an integrative framework can advance our understanding of how these interventions can be designed for leadership development while generating a model for research and evaluation.

Table 1.

Steps in the Action Learning and Problem-Based Learning Processes.

Generic steps in the learning process	Action learning	Problem-based learning
Group formation and complex problem introduction	Small groups are formed, ideally a diverse mix of five to seven participants, and provided a complex, real work problem with an unknown solution. With support from an organization sponsor and a coach, they set learning goals, discuss and frame the problem, and identify action plans and questions that need to be answered to solve the problem.	Small groups are formed, typically five to eight participants, and provided an ill-structured problem found in real life. Under the guidance of a facilitator, they set learning goals, define and bound the problem, brainstorm hypotheses, and discuss what steps to take and what they need to learn to solve the problem.
Student-directed learning	Participants engage in self-directed learning and take actions to solve the problem.	Participants engage in self-directed learning, collect resources for themselves and their group, and take actions to solve the problem.
Collaborative learning (with varying degrees of facilitator support)	Group members meet to reflect on their learning, ask questions, challenge assumptions, and work together to solve the problem.	Group members meet to share their learning, reflect, revisit the problem, and continue to work together to solve the problem.
Problem resolution and feedback	Participants reflect on their actions, solution, and feedback from the sponsor, organization members, or coach.	Participants receive feedback about their solutions, summarize their learning, and reflect on their actions and solution.

The integrative framework proposed in this article specifies the essential design characteristics that can be measured to examine relationships between individual- and team-level elements and leadership development outcomes. In the next section, the steps used to search the literature for relevant sources are described, followed by a review of the theory and research behind PBL to articulate the core set of defining features for readers who are less familiar with this approach. The fourth section briefly reviews the AL literature to reveal the conceptual overlap between PBL and AL designs, leading to the proposed framework that integrates recommendations from both streams of research. The article concludes by highlighting the multilevel nature of the variables and offering recommendations for program evaluation.

Method

This research to construct an integrative framework began with an investigation into AL program design, specifically articles from scholarly and peer-reviewed sources that examined how and why AL designs are effective. Because this research focused on HRD efforts in organizations, the initial search used the EBSCOhost Business Source Premier database, which generated 389 articles from 1965 to 2012 using the search term “action learning.” Practitioner books about AL also were reviewed, and citations from articles were examined to expand the search. In addition, Web of Science was used to find articles similar to the most relevant manuscripts. Finally, additional searches of EBSCOhost Business Source Premier, Academic Search Complete, and PsycINFO were conducted between 2012 and 2016 to incorporate recently published articles relevant to the framework as it was being constructed.

The search of the AL literature confirmed conclusions from earlier reviews, which found AL research lacking in critical investigations into the key features and underlying dynamics of AL (Cho & Egan, 2009, 2010). To find additional support for AL design characteristics, the literature search was expanded to review the PBL literature, which seemed to identify many of the same design features as AL. The EBSCOhost Business Source Premier search for “problem-based learning” articles from 1965 to 2012 identified 117 articles from peer-reviewed journals. However, much of the PBL literature resides outside of business and HRD periodicals, so additional articles were collected using the reference sections of articles, books, and reviews of the PBL literature and by using Web of Science searches for similar work. PBL has been popular in both higher education and K-12 contexts, but this review focused on articles that examined PBL designs for adults and higher education. Consistent with the AL literature review, additional searches for articles in EBSCOhost Business Source Premier, Academic Search Complete, and PsycINFO were conducted between 2012 and 2016 to incorporate recently published papers relevant to this study.

PBL Background and Design

PBL Objectives and Outcomes

The origins of PBL can be traced to the medical faculty at McMaster University, who introduced problem-solving instructional techniques in the late 1960s to improve the diagnostic skills of their medical students (Barrows, 1980; Barrows & Mitchell, 1975; Hung et al., 2008; Neufeld & Barrows, 1974). Medical education programs around the world began adopting PBL as they encountered similar challenges effectively preparing physicians for clinical practice, and in the 1990s, PBL spread to other disciplines in higher education as well as K-12 settings (Hung et al., 2008). Although PBL has been used extensively in medical education, there are few published studies of its use in business or management education (Brownell & Jameson, 2004; Hallinger & Lu, 2011; Ungaretti, Thompson, Miller, & Peterson, 2015; Walker & Leary, 2009).

The objective of the McMaster faculty was to create an innovative approach to medical education that emphasized the development of capabilities and characteristics that would help students become better physicians, rather than a singular focus on teaching factual knowledge of medicine (Neufeld & Barrows, 1974). Literature reviews of PBL tend to converge around the following learning outcomes: (a) an extensive yet flexible base of knowledge, (b) problem-solving skills, (c) self-directed learning (SDL) skills, (d) intrinsic motivation to learn, and (e) collaboration skills (Hmelo-Silver, 2004; Loyens et al., 2012; Loyens, Magda, & Rikers, 2008).

Because PBL emerged as a way to develop better practitioners, the specific objectives of PBL can vary within and between programs (Barrows, 1986). For example, although developing collaboration skills was an objective of McMaster’s medical program, PBL was initially described as a technique that could be presented in a variety of formats, to individuals or to small groups (Barrows & Mitchell, 1975; Neufeld & Barrows, 1974). Barrows (1986) addressed this issue, describing PBL as a variety of educational methods with the common feature being the use of problems for instruction. Studies suggest that it supports the development of SDL strategies and skills required for complex problem solving in professional practice (Barrows, 1980, 1986; Hmelo & Lin, 2000; Hung et al., 2008). Applied to management development, PBL has been described as a powerful approach to simultaneously teach transferable critical thinking skills and domain-specific content, allowing learners to achieve both the cognitive and behavioral development necessary to solve complex problems (Brownell & Jameson, 2004; Kek & Huijser, 2011; Ungaretti et al., 2015). Furthermore, it may be an effective technique for developing tacit knowledge, or knowledge that is difficult to explain to others, derived from experience, and associated with higher levels of job performance and career success (Armstrong & Mahmud, 2008; Raelin, 2007, 2008). PBL interventions can provide contextualized learning opportunities where participants can practice activities and methods to solve problems experienced in the workplace (Hung et al., 2008). With complex, ill-structured problems serving as the catalyst for learning, PBL provides opportunities for participants to apply their knowledge in new situations, see how they deal with ambiguity, and learn skills that will help them navigate through uncertainty (Hmelo & Lin, 2000; Hung, 2011).

A thorough review of the literature examining PBL’s effectiveness in achieving these learning objectives is beyond the scope of this article, but several highlights are worth noting. First, research suggests that PBL improves a variety of professional skills, including social, critical thinking, and problem-solving skills (Hmelo-Silver, 2004; Hung et al., 2008; Loyens et al., 2012; Schmidt, Rotgans, & Yew, 2011; Schmidt, van der Molen, te Winkel, & Wijnen, 2009). Second, several meta-analyses reveal mixed support for PBL (Gijbels, Dochy, den Bossche, & Segers, 2005; Schmidt et al., 2009; Strobel & van Barneveld, 2009; Walker & Leary, 2009), and reviews of empirical studies echo these conflicting findings (Hmelo-Silver, 2004; Hung et al., 2008; Loyens et al., 2012; Mamede, Schmidt, & Norman, 2006; Schmidt & Moust, 2000; Schmidt et al., 2011; Svinicki, 2007). Taken as a whole, this literature suggests that PBL can foster long-term retention of knowledge and practical application of knowledge. However, there are many challenges associated with PBL research, including inconsistencies in PBL design (e.g., problem types) and implementation (e.g., facilitation techniques), a lack of reporting about how PBL was designed or implemented, and widely varying measures of design and outcome variables (Hung, 2011). Although most of these studies are situated in medical education, those that have examined the effectiveness of PBL in business and management education provide encouraging results (Hallinger & Lu, 2011; Walker & Leary, 2009). The next section begins with a review of research on the basic design features of PBL before elaborating on the specific design and implementation characteristics.

Defining Features of the PBL Process

Prior research suggests that five features distinguish PBL from other experiential learning techniques: (a) starting the process with the problem, (b) requiring SDL throughout the process, (c) reflection about the problem solving and learning, (d) small group collaboration, and (e) facilitation to guide learning (Barrows, 1980; Hmelo-Silver, 2004; Hung et al., 2008; Schmidt et al., 2011). Following is a review of each feature and the theories used to explain why they are critical for achieving PBL objectives.

A primary differentiating feature of PBL (and AL) compared with other types of experiential learning designs is that PBL begins with the introduction of a problem for learners to solve prior to offering any instruction about the topic (Barrows, 1980; Hmelo-Silver, 2004; Hung et al., 2008; Schmidt et al., 2011). The content to be learned is organized around the problem as students define it and formulate ideas about possible solutions. Introducing clinical problems first in the instructional process was a significant innovation for medical educators because it challenged the “sequence myth” that basic science needed to be learned before clinical science (Neufeld & Barrows, 1974). The theoretical rationale is that the problem motivates learners and becomes the focus of student-directed learning activities about the topic (Barrows & Mitchell, 1975; Hmelo-Silver, 2004; Hung, 2009; Loyens et al., 2008; Schmidt, 1983; Schmidt et al., 2011). Consistent with constructivist theories suggesting that people learn new knowledge more effectively when they can connect it with what they already know, starting with the problem is believed to activate existing knowledge schemas to help learners process new information (Hmelo-Silver, Duncan, & Chinn, 2007; Norman & Schmidt, 1992; Schmidt, 1993; Schmidt & Moust, 2000; Schmidt et al., 2011). PBL assumes that participants bring a variety of experiences and baseline knowledge into the problem-solving process, so they can identify their unique learning needs and goals based on what they know and do not know to solve the problem.

This leads to a second distinguishing feature of PBL, the emphasis on SDL (Barrows, 1986; Hmelo-Silver, 2004; Loyens et al., 2008; Schmidt et al., 2011). Although experienced facilitators guide learning as needed, participants are given the authority and time to pursue their own learning goals. Boredom, fatigue, and retention of factual knowledge were among the concerns that medical educators were trying to address with PBL (Barrows & Mitchell, 1975; Hung et al., 2008). Some PBL research suggests that SDL is an important feature because it may activate intrinsic motivation compared with passive, didactic teaching methods, and the sense of ownership and responsibility for learning may raise participant engagement in the inquiry process, fostering deep learning (Hmelo & Lin, 2000; Loyens et al., 2008; Schmidt et al., 2009). Another reason why SDL is an important part of the process is because improving students’ SDL skills is typically one of the learning objectives of PBL (Barrows, 1986; Hmelo-Silver, 2004). Building SDL skills is particularly important for physicians and practitioners who need to continually update and maintain their skills beyond school and to be effective problem solvers (Barrows, 1986; Barrows & Mitchell, 1975; Hmelo & Lin, 2000; Miflin, Campbell, & Price, 2000). For organization leaders faced with continuously shifting industry demands and complex business challenges, the ability to take control of their own learning is a critical leadership competency (Nesbit, 2012). However, if participants are poor self-regulated learners, they may struggle with PBL and require additional support (Hmelo-Silver, 2004; Miflin et al., 2000).

Reflection is a required part of PBL because it is a mechanism by which participants construct their flexible knowledge base. Reflection facilitates sense making, and it helps learners connect their PBL actions (and reactions) to their individual learning goals and metacognitive skill development (Hmelo-Silver, 2004; Pease & Kuhn, 2011; Wirkala & Kuhn, 2011). As described by Brownell and Jameson (2004), affective learning (e.g., self-monitoring, understanding others, appreciating individual differences) results from what they call the interpretive learning process whereby learners explore their values and beliefs, make comparisons with others, and develop insights about themselves from interactions. As such, reflection is an important feature that must be present throughout PBL. Because reflection is itself a cognitive skill that individuals can perform to varying degrees, a variety of techniques are recommended to support both individual and group reflection, including using structured learning journals, diaries, and facilitation (Hmelo-Silver, 2004).

Small group collaboration is a key feature noted by most, but not all, PBL scholars (Barrows, 1986; Dolmans & Schmidt, 2006; Hmelo-Silver, 2004; Loyens et al., 2012; Norman & Schmidt, 1992; Schmidt, 1993; Schmidt et al., 2011). To the extent that improving collaboration and interpersonal skills is an objective of PBL, problem solving in teams clearly would be essential to provide students opportunities for practice. However, many authors argue that team learning is essential for PBL because group discussions motivate participants to share prior knowledge, question and challenge their colleagues’ understanding of the problem, and elaborate on the shared knowledge, which is believed to optimize learning and transfer (Dolmans & Schmidt, 2006; Innes, 2006; Schmidt, 1983, 1993; Schmidt et al., 2011; van Blankenstein, Dolmans, van der Vleuten, & Schmidt, 2011). Group members also can play a scaffolding role by supporting each other as they solve problems, allowing less experienced learners to practice new skills with coaching from their more experienced peers (Hmelo-Silver et al., 2007; Reiser, 2004). Learners who have different levels of prior knowledge and expertise can observe and interact to foster understanding and sense making. Hmelo-Silver’s (2004) review of the literature supports the role of group collaboration in promoting intrinsic motivation and knowledge construction, but she also noted that differences in group dynamics and facilitation may influence the degree of collaboration and thus lead to inconsistent findings about whether PBL can develop better collaborators. Some studies comparing individual- and team-based PBL designs do not support the notion that social interaction and collaborative learning are the primary drivers of engagement, thus underscoring the need to further examine the relative impact of small group collaboration on various outcomes (Pease & Kuhn, 2011; Wirkala & Kuhn, 2011).

Finally, experienced facilitators play an important role in the learning process because PBL is not unguided discovery learning (Hmelo-Silver et al., 2007; Hung, 2009; Schmidt, Loyens, van Gog, & Paas, 2007). Participants must have the freedom to reason and learn without a directive facilitator, but careful PBL design, coaching, and scaffolding to support individual learning are required to achieve the learning objectives (Hmelo & Lin, 2000; Schmidt et al., 2011). Facilitators are used in a variety of ways to support PBL, which will be discussed in the “Integrative Framework for PBL and AL” section of this article.

To summarize, PBL features a group of learners collaborating to solve a problem while engaging in reflection and SDL under the guidance of a facilitator. The design of PBL can change to adjust features that will meet the needs of participants and the objectives and constraints of the organization. However, such flexibility is a liability for researchers wishing to evaluate PBL or understand which features influence the desired results. Before elaborating on these issues, the next section provides a comparison of PBL and AL design features.

AL Background and Design

AL Objectives and Outcomes

Unlike PBL’s origins in medical education, AL was first applied in practical business contexts by Reg Revans, who in the 1940s and 1950s began using AL within organizations to solve business problems while developing employees (Anderson & Coleman, 2015; Marquardt, 1999; O’Neil & Marsick, 2007). Revans attempted to introduce AL to academic institutions in Europe, but it has not been widely adopted in management education (Johnson & Spicer, 2006; Pedler, Burgoyne, & Brook, 2005; Raelin, 1997a, 2009). In a review of the top 50 U.S. business schools, Navarro (2008) found that only a small number of these schools required multidisciplinary experiential exercises as a required component of their curriculum. Brook and Milner (2014) reported a recent increase in the use of AL in postgraduate business education programs within the United Kingdom, and a few studies have reported successful implementations in academic settings (Bannan-Ritland, 2001; Johnson & Spicer, 2006). Research points to a variety of institutional and logistical reasons why management educators have been slow to adopt these approaches, including administrative, design, and implementation challenges (Brook & Milner, 2014; Johnson & Spicer, 2006; Raelin, 2009). However, as noted in the introduction to this article, AL is widely used within organizations around the world (Anderson & Coleman, 2015; Leonard & Lang, 2010; Marquardt, 2011).

As with PBL, the objectives of AL include helping participants develop knowledge, problem-solving skills, collaboration skills, and skills necessary for continuous learning, or “learning how to learn” (Leonard & Lang, 2010; Marquardt, 1999; Marsick & O’Neil, 1999; Raelin & Coghlan, 2006). Because AL originated in organization development contexts and is most frequently implemented in workplaces, an additional objective noted by authors is that AL can be used to advance learning, change and knowledge sharing within the organizational system (Marquardt, 1999; Marquardt & Banks, 2010; Volz-Peacock, Carson, & Marquardt, 2016; Yeo & Nation, 2010). However, this objective is not always applicable in leadership development programs, especially when AL is delivered in academic contexts (Bannan-Ritland, 2001; Johnson & Spicer, 2006). The flexible application of AL to address a variety of learning objectives and the philosophy used to implement AL can result in different AL types, but the primary focus is typically individual development (Cho & Egan, 2010; Marsick & O’Neil, 1999; Raelin, 2008).

There are fewer quantitative studies of AL compared with PBL research, and most empirical articles about AL use descriptive, case study approaches (Anderson & Coleman, 2015; Cho & Egan, 2009; Johnson & Spicer, 2006). Reviews of the research suggest that AL improves a variety of leadership skills, including collaboration, conflict management, self-awareness, coaching, communication, and continual learning skills (Cho & Egan, 2009; Leonard & Lang, 2010; Leonard & Marquardt, 2009; Raudenbush & Marquardt, 2008; Volz-Peacock et al., 2016). Studies also suggest that AL improves motivation to learn and learning transfer because it uses real work problems (Leonard & Lang, 2010; Raelin, 2007; Raelin & Coghlan, 2006; Volz-Peacock et al., 2016). However, these results also share the same set of challenges found in PBL research, which is that not all AL designs and implementations are the same. Overall, this review found far fewer quantitative studies of AL design effectiveness compared with the research on PBL design in medical education.

Defining Features of the AL Process

Similar to PBL, a variety of experiential learning programs have been called AL, making a clearly defined single method elusive, but as described by Cho and Egan (2009), “The real value of AL that differentiates it from other action strategies is a pragmatic focus on learning for the sake of problem solving” (p. 435). The “classical principles” of AL often are referenced to distinguish it from other experiential approaches, but many authors have espoused variations in these principles and subcategories of AL that make it particularly difficult to define (Johnson, 1998; Marquardt, 1999; Marquardt & Banks, 2010; Marsick & O’Neil, 1999; O’Neil & Marsick, 2007; Pedler et al., 2005; Smith & O’Neil, 2003). But, at its core, AL is practiced when a small group of learners collaborate to support individual learning through problem solving that centers around an authentic work problem, under the guidance of an adviser or coach (Marquardt, 1999; Marsick & O’Neil, 1999; Pedler et al., 2005; Revans, 1982; Smith & O’Neil, 2003). Thus, the defining features of AL are identical to PBL.

A comparison between the five PBL design features described in the previous section and commonly prescribed components of AL can further highlight the conceptual overlap. First, AL typically begins with an ill-structured problem, rather than problems delivered around predetermined curriculum or formal instruction (Leonard & Lang, 2010; Marquardt, 1999; Yeo & Nation, 2010). Second, participants are expected to take self-directed action to learn and solve the problem by asking questions to clarify and define the problem, establishing learning goals, identifying their knowledge gaps and organizational constraints, and reflecting upon possible solutions (Leonard & Lang, 2010; Marquardt, 1999; Raelin & Coghlan, 2006). The AL process should be iterative, with problem analysis, self-study, reflection, and action toward problem solving occurring until a solution is implemented and the participants receive feedback about their actions and solutions (Johnson & Spicer, 2006; Yeo & Nation, 2010). Third, reflection is a key feature of the AL process, with individual and group reflection exercises used to help learners make sense of their experiences (Leonard & Lang, 2010; Raelin & Coghlan, 2006; Raelin & Raelin, 2006; Sofo, Yeo, & Villafañe, 2010; Yeo & Nation, 2010). Fourth, participants work in small groups to engage in problem solving and learning (Leonard & Lang, 2010; Marquardt, 1999; Marsick & O’Neil, 1999). The processes and activities used within groups can vary across AL interventions, as they do with PBL, depending on the design and facilitation provided. And finally, an experienced facilitator or coach guides the group members through the learning process, serving as a model for questioning and reflection (Marquardt, 1999; Volz-Peacock et al., 2016). As with PBL, the role of the facilitator depends on the underlying school of thought in which the AL design is grounded (Anderson & Coleman, 2015; Marsick & O’Neil, 1999; Waddill & Marquardt, 2003).

Although authors have differentiated AL from other experiential methods like simulations and outdoor adventures, or other action-oriented interventions like action research (Marquardt, 1999; Marsick & O’Neil, 1999; Raelin, 2007; Raelin & Coghlan, 2006), few comparisons are made between AL and PBL. Lohman (2002) sought to classify several types of training techniques that use problem solving as a core design feature. One distinction she selected was problem type, describing PBL as using more prototypical problems compared with AL. However, given the range of problem types used in both PBL and AL, this is a less meaningful distinction than the affordances of the problem, which from a theoretical perspective can be used to explain why problem characteristics may be associated with learning outcomes. In a critique of Lohman’s classification, Mayer (2002) described the line drawn between AL and PBL as fuzzy.

Yeo and Gold (2011) also suggested that AL is distinctly different from PBL because of the problem type, how learning occurs throughout the process, and the facilitator role in both guiding learning and enforcing group norms. However, problem type and facilitation can take a variety of forms in both PBL and AL, as noted earlier and discussed in the next section. The distinction between AL and PBL in how learning occurs, described by Yeo and Gold to be “a more holistic approach where learning is largely integrated throughout the entire project through both planned and spontaneous processes” is a claim that requires more evidence (p. 514). This integrative study suggests that uniting these two approaches under a common theoretical framework will advance research and provide insights into how AL can be implemented and evaluated more effectively.

Institutional and program design factors create barriers to adopting AL in academic contexts, suggesting a need for innovative ways to integrate it into management education to support leadership development (Johnson & Spicer, 2006; Raelin, 2007, 2009). Given the widespread use of PBL in higher education and the volume of empirical studies supporting it, much can be drawn from that literature. For example, PBL research on problem characteristics like problem clarity and authenticity can offer new directions for the research and design of problems that are most suitable for leadership development purposes. In addition, AL scholars have focused on questioning and reflection techniques, and group design characteristics like team diversity, which have received relatively less attention in the PBL literature. The next section demonstrates how the characteristics that have been investigated in PBL and AL research can be merged into an integrative framework to provide a stronger conceptual and theoretical grounding for leadership development research and evaluation.

Integrative Framework for PBL and AL

Although there is a growing body of research into the underlying processes of PBL and AL, relatively fewer studies have examined which design characteristics are essential to producing desired outcomes (Cho & Egan, 2009; Hmelo-Silver, 2004; Leonard & Marquardt, 2009; Scott, 2014; Sockalingam, Rotgans, & Schmidt, 2011; Svinicki, 2007). Systematic program evaluation research in this area would help educators and practitioners make evidence-based design and implementation decisions. The complexity of program design, the multiple conceptual definitions, and numerous combinations of unspecified, interacting variables have made it difficult to make progress in this area (Cho & Egan, 2009; Hung, 2011; Mamede et al., 2006; Scott, 2014; Sockalingam et al., 2011). Another challenge is that some design characteristics are individual-level variables, whereas others are variables more properly examined at the group level of analysis (Scott, 2014). Advancing our understanding of leadership development requires research that examines its multilevel and longitudinal nature, examining both the interpersonal and intrapersonal processes that unfold over time (Day et al., 2014). For proper evaluation, it is critical to represent variable levels in theoretical and statistical models, particularly when constructs are measured at one level and specified at another level (Garavan, McGuire, & O’Donnell, 2004; House, Rousseau, & Thomas-Hunt, 1995).

To address these issues, this study builds upon earlier work (Scott, 2014) to propose an integrative framework that is multilevel, based on variables identified in prior research, and includes variables requiring further examination, such as goal orientation and scaffolding mechanisms. Table 2 presents the design characteristics and defining terms, along with their proposed relationships (i.e., input or process) with learning outcomes. Characteristics are grouped into individual- or group-level concepts from a theoretical and statistical modeling perspective. Table 2 also includes institution-level variables identified as important in understanding the effectiveness of program implementation and should be included in statistical models examining PBL effects across organizations and institutions (Chaharbaghi & Cox, 1995; Cho & Egan, 2009; Loyens et al., 2012; Neufeld & Barrows, 1974; Schmidt & Moust, 2000). Although an in-depth consideration of third-level effects is beyond the scope of this article, it is important to recognize that organizational-level characteristics such as culture and the availability of learning resources in the organization also may affect the effectiveness of the intervention and learner outcomes.

Table 2.

Multilevel PBL and AL Design Characteristics.

Variables	Variable definitions	Path in model
Individual-level variables
Problem authenticity	Importance and meaningfulness Practical relevance—based on “real work” Provides context for knowledge transfer	Input
Problem familiarity	Problem understanding and knowledge Context and content familiarity	Input
Problem challenge	Degree of complexity Ill-structured problem Difficulty balanced by prior knowledge	Input
Problem clarity	Concise, comprehensible problem statement Useful problem keywords Identifiable learning objectives	Input
Experience	Amount of prior knowledge about the problem Work experience Experience with PBL	Input
Learning orientation	Learning vs. performance goal orientation Deep vs. surface learning Learning styles/preferences	Input or moderating
Perceived autonomy	Control over learning issues and processes	Input
Self-directed learning	Selection of learning issues and strategies Planning and time management Engagement in self-study and monitoring Individual commitment to learning	Input, mediating, and outcome
Reflection	Thinking about new/dissonant ideas Considering emotional reactions and behaviors Questioning beliefs and assumptions	Mediating
Motivation	Intrinsic motivation Willingness to engage in learning	Mediating and outcome
Team-level variables
Facilitator/tutor effectiveness	Cognitive and social congruence Expertise Scaffolding learning activities Stimulate elaboration, knowledge integration Stimulate interaction and accountability	Input or moderating
Scaffolding mechanisms	Structuring tools Problematizing tools Collaboration tools	Input or moderating
Team autonomy	Team control over learning, problem solving Climate supportive of team autonomy	Input or moderating
Diversity	Demographic diversity Functional diversity Goal orientation diversity	Input or moderating
Learning team collaboration	Sharing expertise, distributed problem solving Shared responsibility for learning and action Questioning and challenging ideas Climate of openness, trust, and encouragement	Mediating
Institution-level variables
Learning resources	Content knowledge materials Training for PBL novices Course and curriculum goal clarity	Input or moderating
Institution culture	Supportive learning environment Interdisciplinary collaboration	Input or moderating

Note. PBL = problem-based learning; AL = action learning.

Figure 1 represents the individual- and group-level design variables for the integrative framework and the proposed relationships between them, based on the few existing PBL studies that have used path analysis to examine these relationships (see Gijselaers & Schmidt, 1990; Schmidt & Moust, 2000; van Berkel & Schmidt, 2000). The outcomes presented in this model are the individual-level learning objectives typically included in PBL research. The group- and organizational-level outcomes sometimes associated with AL are not discussed here. Because the focus of this integrative framework is designing and evaluating programs to improve leader capabilities, it is beyond the scope of this article to address other potential benefits that may follow from implementing PBL within organizations, such as greater knowledge sharing among participants and organizational learning. To advance the use of PBL for leadership development, it is important to identify and examine those variables believed to have an impact on the effectiveness of this technique, which is an area of study that has received less attention. Each of the design characteristics is described in more detail next, starting with the individual-level variables in the framework.

Figure 1.

Integrative framework for PBL and AL design.

Individual-Level Characteristics: Input Variables

Problem quality

Because problems are the central feature of PBL and AL, identifying or constructing problems is a key step in design (Hung, 2009; O’Neil & Marsick, 2007). In AL, problem criteria are defined broadly and based on individual learning objectives with consideration given to strategic business issues when delivered in an organizational context. One difference between AL and PBL designs is that AL typically uses “real work” problems (Marquardt, 1999; Marsick & O’Neil, 1999; Pedler et al., 2005), whereas PBL stresses the use of real-life problems in the field (van Berkel & Schmidt, 2000). Research also suggests that AL problem criteria include problem complexity, challenge, and low familiarity (Johnson, 1998; Leonard & Lang, 2010; Marsick & O’Neil, 1999; Marquardt, 1999; Yeo & Nation, 2010). Problem quality in PBL has received more research attention as scholars began examining why PBL works. In earlier models, “problem quality,” as defined by problem statement clarity and suitability for group work and SDL, had direct and indirect effects on both student achievement and motivation (Schmidt & Moust, 2000; van Berkel & Schmidt, 2000). However, problem quality is multifaceted, and problem characteristics may have interacting or differential effects.

Some authors have defined problem characteristics as “high quality” to the extent that they trigger individual student engagement, collaborative discourse, and SDL toward desired outcomes (Mamede et al., 2006; Schmidt & Moust, 2000), but problem characteristics should be operationalized exclusive of their anticipated impact on mediating processes, such as promoting SDL. Subsequent studies have sought to extend research by more clearly defining problem characteristics (Hung, 2006, 2009; Sockalingam et al., 2011; Sockalingam & Schmidt, 2011). Drawing from PBL theory and existing evidence, four problem characteristics appear to create high-quality problems for participants: authenticity, familiarity, challenge, and clarity.

Problem authenticity

Authenticity is a ubiquitous problem characteristic in PBL and AL theory and research. To the extent that learners perceive the problem as authentic, meaningful, and related to their field of practice, they should be more cognitively and affectively engaged in solving the problem, thereby increasing the likelihood that they will learn and transfer new knowledge into practice (Hung, 2006; Innes, 2006; Leonard & Lang, 2010; Loyens et al., 2008; Marquardt, 1999; Revans, 1982; Schmidt et al., 2009; Vince, 2004; Vince & Martin, 1993; Wirkala & Kuhn, 2011). As mentioned previously, AL typically requires the problem to be an actual challenge from the organization in which the learner works, whereas in PBL the problem may not be an actual work problem depending on the program objectives. However, if problem authenticity can be established to emotionally engage participants and provide opportunities for action to solve a problem, regardless of whether learners are stakeholders within the organization, AL might become more easily implemented in management education programs.

Furthermore, it may be incorrect to assume that situating participants within their own organizations with actual work problems creates the authenticity needed to achieve learning goals. For effective leadership development, a new context may be necessary. Hung’s (2006) 3C3R model of problem design extends the conceptualization of authenticity by specifying two components for PBL: (a) contextual validity, or the practical relevance to the learner’s professional setting; and (b) the degree of problem contextualization, or the range and degree of context information provided about the problem. Both need to be considered in conjunction with the problem content and learner needs (Hung, 2006), thus the double-headed arrow between “Problem Characteristics” and “Learner Characteristics” shown in Figure 1. The 3C3R model can be applied to measure problem authenticity (e.g., the extent to which the problem is perceived to be a valid challenge faced by leaders) and compare the extent with which it affects learner motivation and development.

Problem familiarity

PBL and AL scholars have offered divergent proposals about problem familiarity. In AL design, low familiarity with the problem topic or setting is believed to be more motivating and trigger more questioning and reflection among students, thereby providing more meaningful learning opportunities (Marquardt, 1999). This proposition is supported by studies of experiential learning in leadership development based on activation theory, which suggests that motivation and arousal in cognitive processing increases when tasks are optimally unfamiliar (DeRue & Wellman, 2009). Conversely, PBL scholars suggest that unfamiliar problems prevent learners from relating to the challenge and having productive group discussions (Loyens et al., 2012; Schmidt et al., 2011). One experiment designed to compare the effects of familiar and unfamiliar problems found that student groups assigned to solve familiar problems found them to be more interesting and of higher quality than groups solving unfamiliar problems, but there were no significant differences between groups in the amount of knowledge acquired, group discussion, or time engaged in self-study (Soppe, Schmidt, & Bruysten, 2005). Conversely, Scott (2014) found that students with higher problem familiarity had significantly lower levels of perceived learning from the PBL experience, although there also were no significant associations between familiarity and student engagement in SDL and reflection.

PBL studies generally show that familiarity can have positive effects on learner interest and learning activities as well as direct effects on learning outcomes, which is consistent with theory suggesting that tasks need to be familiar to at least some group members so they may offer some scaffolding to support learning and performance (Hmelo-Silver et al., 2007; Loyens et al., 2012; Reiser, 2004; Schmidt et al., 2011; Sockalingam & Schmidt, 2011). However, given the mixed empirical results and theoretical perspectives, it is likely that the effects are more complicated and nonlinear. The interrelationship between problem familiarity, prior knowledge, and challenge is discussed next.

Problem challenge

The degree of challenge or “difficulty” level of the problem has been addressed in a variety of ways, but this characteristic is consistently included in PBL and AL descriptions alike (Barrows, 1986; Marquardt, 1999; Revans, 1982; Schmidt, 1993; Sockalingam & Schmidt, 2011; Yeo & Nation, 2010). In AL, problems can be selected either by the individual, based on a workplace issue she is facing, or by the organization for a group to tackle collaboratively. In either case, the problem must be a challenge with no predefined answer or clear process toward a solution (Johnson, 1998; Marquardt, 1999; Yeo & Nation, 2010). Similarly, PBL designs can present problems for either individuals or groups to solve, but problems must be complex, ill-structured or open-ended enough to engage learners in action to solve the problem and build a flexible knowledge base (Barrows, 1986; Hmelo-Silver, 2004; Hung et al., 2008; Loyens et al., 2012). However, several authors also note the importance of calibration between the breadth and depth of problem content and participants’ prior knowledge, so learners can integrate new and existing knowledge and engage in productive discussions without cognitive overload (DeRue & Wellman, 2009; Hung, 2006; Loyens et al., 2012).

Problem clarity

Several PBL studies suggest that problem statements that are clear, concise, and include useful key words can help participants identify learning goals, direct their learning, and have more successful SDL experiences (Hung, 2009; Sockalingam et al., 2011). Having clearly stated problems guides participants toward setting learning goals that are more consistent with the desired learning objectives, and avoid focusing on irrelevant issues that may contribute to ineffective group discussions and self-study activities (Hmelo-Silver, Chernobilsky, & Jordan, 2008; Sockalingam et al., 2011; Sockalingam & Schmidt, 2011). Because problem clarity may directly affect the level of learner engagement, designers should pay particular attention to presenting a thorough, informative statement of the problem and check to ensure participant understanding early in the process so as to avoid frustration and fruitless group discussions. The rigorous problem design process proposed by Hung (2006, 2009) offers several methods that PBL designers can use to ensure problem clarity.

Learner characteristics

Aside from SDL and prior knowledge/experience, other learner characteristics have received limited research attention in PBL (Hung et al., 2008; Loyens et al., 2012). Because SDL is an essential feature of the process, serving primarily as a mediating variable in theoretical models, it is addressed in a later section even though it also may be viewed as a learner characteristic (Loyens et al., 2008). Individual differences described under “Learning Orientation” section are included in the model even though they have not yet been widely examined in empirical studies of PBL or AL.

Experience

As discussed earlier, problem familiarity and challenge are related to the learner’s experience level, which includes the amount of prior knowledge about the problem content as well as experience with PBL. Experience plays an important role in PBL design because learners must have sufficient prior knowledge to apply during the problem-solving process, and it serves as the foundation for collaborative discourse and constructing flexible knowledge structures (Hmelo-Silver, 2004; Schmidt et al., 2011; Sofo et al., 2010). In addition, several authors have suggested and found evidence that experience has a direct, positive effect on AL outcomes, including achievement and motivation (Araz & Sungur, 2007; Raelin, 1997b; Schmidt & Moust, 2000). Theoretical and empirical PBL models suggest that the prior knowledge participants bring to PBL can have both direct and indirect effects on learning outcomes, but empirical research has been mixed and inconclusive (Gijselaers & Schmidt, 1990; Schmidt & Moust, 2000; van Berkel & Schmidt, 2000). Additional research is needed to understand these effects, particularly in leadership development contexts where participants often bring a high degree and variety of work experiences.

Learning orientation

Several authors suggest that self-efficacy, perceived control, and personality characteristics like openness may be particularly relevant for experiential learning (Araz & Sungur, 2007; DeGeest & Brown, 2011; DeRue & Wellman, 2009; Hezlett, 2016; Raelin, 1997b). Because PBL requires learners to engage in reflection, SDL, and collaborative discourse to draw meaning from their problem-solving experiences, its effectiveness may be more susceptible to these individual differences, particularly goal orientation. According to achievement goal theory, individuals differ in the extent to which they have a learning goal orientation (LGO) and a performance goal orientation (PGO) when presented with a challenge. Individuals demonstrating high LGO are comfortable engaging in challenging activities and self-improvement efforts because they believe their capabilities can be developed, whereas individuals demonstrating high PGO tend to avoid mistakes and prefer nonchallenging activities because they believe their capabilities are relatively fixed (Button, Mathieu, & Zajac, 1996; Dweck, 1986; Dweck & Leggett, 1988). Goal orientation influences how learners perceive tasks and process information, how they respond to feedback, and what actions they take to accomplish tasks (Button et al., 1996; Dierdorff & Ellington, 2012; Dweck & Leggett, 1988; Phan, 2009; Senko, Hulleman, & Harackiewicz, 2011). This research and initial empirical results suggest that participants with higher levels of LGO may also have higher engagement in the SDL required in PBL (Hezlett, 2016; Scott, 2014).

Individuals with more experience tend to have higher levels of LGO and lower levels of PGO, so goal orientation may interact with other individual-level characteristics (Button et al., 1996). Evidence also suggests that PGO negatively affects group collaboration, so PGO may interfere with collaborative discourse (Janssen, Poortvliet, Van de Vliert, & Van Yperen, 2007; Senko et al., 2011). However, studies find that both orientations can be associated with learning achievement, and with PBL’s focus on solving problems, additional research is needed to determine whether LGO and PGO are directly associated with PBL outcomes (Senko et al., 2011). Such effects may depend on the team context, discussed in a later section.

Individual-Level Characteristics: Process/Mediating Variables

Three individual-level design characteristics are process variables in PBL models: SDL, reflection, and intrinsic motivation. Because these variables are central features of PBL, as discussed earlier, the theoretical rationale will not be presented again in this section. However, because program designers can make significantly different design and implementation choices for facilitating SDL and reflection, this section highlights those areas where differences frequently occur and how they can be incorporated into this framework for evaluation purposes.

SDL

To achieve the objective of developing SDL, participants need to have the time and autonomy to engage in SDL activities with an optimal balance of scaffolding to support learning (Loyens et al., 2008; Wijnia, Loyens, & Derous, 2011). As a learning process, SDL includes selecting learning issues, planning how to solve the problem, deciding which learning resources and strategies to use, effectively managing time, engaging in self-study, and monitoring and evaluating progress (Loyens et al., 2012, Loyens et al., 2008). As a learner characteristic, SDL includes the ability and willingness to engage in these learning activities (Loyens et al., 2008; Miflin et al., 2000; van Berkel & Schmidt, 2000). SDL is similar to the “individual commitment to learning” that is a key characteristic in AL (Marquardt, 1999). But SDL goes beyond motivation and the ability to listen to include metacognitive skills and strategies that learners use to guide their learning, capabilities that designers should not assume their adult learners already have mastered through prior experience. In their review of the research, Loyens et al. (2008) found mixed support for the hypothesis that PBL develops SDL skills. Frameworks developed in medical education to address the difficulties that graduate students face when first engaging in PBL, such as the one described by Miflin and her colleagues (2000), can be adapted for leadership development programs to provide the necessary guidance for developing SDL skills. Nesbit’s (2012) framework for self-directed leadership development articulates the metaskills of self-reflective practice, managing emotional reactions to feedback, and self-regulation that HRD practitioners can use to prepare leaders for PBL.

Reflection

Reflection is another important PBL process that does not consistently receive explicit research attention. Occasionally, it is noted as an aspect of SDL (Loyens et al., 2008), but in other PBL studies, it is regarded as the key metacognitive skill and process that makes PBL effective (Hmelo & Lin, 2000; Hmelo-Silver, 2004; Hung, 2009; Kek & Huijser, 2011; Pease & Kuhn, 2011). AL scholars also emphasize the role of reflection in the learning process (Anderson & Coleman, 2015; Leonard & Lang, 2010; Yeo & Nation, 2010). Reflection is part of the meaning-making process, whereby learners seek to understand new or dissonant ideas and integrate them into their knowledge structures. Questioning is often part of the reflective thinking process, individually (metacognitively), with a facilitator, and among group members (Armstrong & Mahmud, 2008; Johnson, 1998; Kek & Huijser, 2011; Kolb & Kolb, 2009; Marquardt, 1999; Marquardt & Waddill, 2004; Phan, 2009; Yeo & Nation, 2010). To the extent that reflection is emphasized, scaffolded, and reinforced using assignments like learning journals, blogs, or diaries, it should have positive effects on SDL, collaborative discourse, and PBL outcomes.

Team-Level Characteristics

The team-level characteristics in PBL are the facilitator (including the instructional scaffolding provided for participants), team autonomy, diversity, and collaboration. The first three of these group-level variables are input variables, whereas collaboration is a process variable proposed to explain how team interactions affect PBL outcomes. For evaluative purposes, these variables are treated as group-level constructs because they create shared effects across members within a team. Measures that are similar to climate measures, which use aggregated individual-level perceptions to represent these group-level variables, may be particularly useful because they also could capture potentially interesting individual differences in perceived facilitator effectiveness and team dynamics (Scott, 2014). Furthermore, because the effects of individual-level variables may vary depending upon the group-learning context that is created by the team, modeling them as context variables can create openings for new hypotheses about cross-level interactions between individual characteristics and team context.

Facilitator and scaffolding effectiveness

Evaluating facilitator effectiveness as a group-level variable is important because facilitators usually modify their support to fit the needs of individual and team learning goals, which may result in different experiences between teams even though they may have the same facilitator (Hung, 2011; Scott, 2014). Research suggests that effective facilitation can occur even when coaches rotate between multiple teams or when participants play the role of facilitator with proper guidance (Hmelo-Silver, 2004). Facilitator effectiveness typically is gauged by their use of questioning techniques to develop metacognitive and collaboration skills, by how well they encourage team members to share knowledge, by the extent to which individuals and the team takes ownership of their learning, by the successful creation of a climate that encourages collaboration, and by the problem-solving scaffolding and learning strategies they model (Hmelo-Silver, 2004; Hmelo-Silver & Barrows, 2008; Leonard & Lang, 2010; Sofo et al., 2010; Yeo & Nation, 2010). These facilitator behaviors and scaffolding tactics are intended to directly affect PBL process variables (e.g., SDL, reflection, and group collaboration) and outcomes (individual learning and motivation) without reducing learner autonomy or providing too much instruction. Other important measures may include facilitator expertise (the extent to which participants perceive them to be experts), the amount of facilitation provided, and how effectively the facilitator reduces instructional scaffolding as learners develop their skills (Ates & Eryilmaz, 2010; Hmelo-Silver, 2004; Hmelo-Silver et al., 2007; Hung, 2011).

Team autonomy

Providing team autonomy means that teams have responsibility and authority to determine how members will collaborate and what actions they will take to achieve their shared learning objectives. Similar to the individual empowerment granted in PBL and AL, team autonomy is expected to create intrinsic motivation and a sense of ownership over the learning process (Wijnia et al., 2011). The rationale for promoting team autonomy in PBL follows the same premise as providing individual autonomy: It allows participants to practice SDL, and their ownership of the problem-solving process fosters learner engagement (Hmelo-Silver & Barrows, 2008; Liu & Fu, 2011; Rotgans & Schmidt, 2011). Participants need to have the freedom to take risks in solving the problem, stretching beyond their comfort zones, and exploring solutions that may be atypical in their usual work contexts. Organizations that rely on team-based work processes expect their leaders to develop the skills to successfully navigate team dynamics.

Yeo and Gold (2011) articulated the importance of supportive environments in providing the right environment for a collaborative and dynamic exchange of ideas. Facilitators need to strike a balance between allowing teams to set their own direction and norms versus intervening to ensure that the team sets itself up for effective collaboration, dialogue, and learning. Research suggests that conflict and disputes may be more likely in autonomous heterogeneous teams, and when members do not share the same work-related attitudes or beliefs, particularly when their tasks are unstructured (Molleman, 2005). As discussed in the next section, facilitators should be prepared to help teams address issues that arise in diverse, autonomous teams.

Team diversity

In both PBL and AL, the team diversity is viewed as important to its success, with more diversity in gender, role, age, experience, and perspective generally proposed to have a positive influence on group collaboration, decision making, and learning (Clarke, Thorpe, Anderson, & Gold, 2006; Marquardt, 1999; Marquardt & Waddill, 2004; Yeo & Nation, 2010). Team diversity may increase member access to the resources, skills, and knowledge to help them with problem solving and learning tasks (van Knippenberg & Schippers, 2007; Yeo & Nation, 2010). However, although there are many potential benefits of selecting a highly diverse team of learners, there also are negative effects to mitigate. For example, Vince and Martin (1993) suggested that power differences in AL teams can influence learning outcomes, and how such differences are managed by the team or supported by a facilitator will create opportunities for members to learn about the effects of power relations. As such, the effects of team diversity are likely to be moderated by facilitator effectiveness.

Recent research about team goal orientation and diversity also raises potential challenges for PBL design. Studies suggest that PGO in particular can have negative consequences for teams because it heightens competition and the use of guarded approaches to knowledge sharing among high PGO individuals (Janssen et al., 2007; Senko et al., 2011); thus, teams with high average PGO levels may be less likely to foster individual learning. Conversely, teams whose members share high LGO may benefit from an agreed focus on mastery. Dierdorff and Ellington’s (2012) longitudinal study of group composition effects during team training found evidence that teams with high average LGO levels may counteract the potentially negative influence of members who have high PGO, providing a “buffering effect” for team members. This raises additional questions about the moderating effects of team goal orientation and goal orientation diversity on PBL outcomes, and what designers and facilitators should do about it.

Pieterse, van Knippenberg, and van Ginkel (2011) characterized goal orientation diversity as deep-level diversity that influences mental representations of tasks and routines. A high degree of variance in the LGO and PGO within teams may result in less effective group collaboration and performance because individual differences in task representations disrupt team coordination and communication (Scott, 2014). Facilitators can help teams establish shared mental models of team learning to minimize negative effects of goal orientation diversity, so scaffolding mechanisms and reflection may be key moderators (Pieterse et al., 2011). To improve PBL design, goal orientation can be measured prior to making learning team assignments so that high levels of team diversity or cases of member dissimilarity can be monitored and their effects closely examined.

Learning team collaboration

The dialogue, practices, and learning environment within the team form this essential mediating process—team collaboration—that allows learners to achieve their objectives and practice SDL activities (Dolmans & Schmidt, 2006; Gijselaers & Schmidt, 1990; Hmelo-Silver & Barrows, 2008; Hmelo-Silver et al., 2008; Innes, 2006; Johnson, 1998; van den Hurk, Dolmans, Wolfhagen, & van der Vleuten, 2001). This mediating process involves “team commitment to learning,” which is a norm described in both PBL and AL research as members taking responsibility for learning, sharing expertise, and collaborating effectively in small groups (Hmelo-Silver & Barrows, 2008; Hung, 2011; Marquardt, 1999). Studying these concepts is difficult because they are complex and dynamic, involving not only the language and conversation patterns participants use but also the tools and resources they apply during the process (Hmelo-Silver et al., 2008). Ideally, PBL team members will support each other as they work toward a common goal while constructively questioning each other’s ideas and having an open dialogue about their experiences (Innes, 2006; Johnson, 1998). In practice, teams may rarely achieve this level of collaboration, instead opting for the “divide and conquer” method of problem solving that improves efficiency but sacrifices the benefits of conversation (Innes, 2006).

Evidence about the importance of social interaction and collaborative learning is mixed (Clarke et al., 2006; Nieminen, Sauri, & Lonka, 2006; Wirkala & Kuhn, 2011). Such studies reinforce the importance of carefully designing for and facilitating collaboration while measuring the extent to which it actually occurs. Because these team processes are shared and norms develop over time, measurement strategies must address the fact that this is a group-level variable that includes individual perceptions of the team environment.

Discussion and Conclusion

This article bridges the PBL and AL research to propose an integrative framework for leadership development design and evaluation. Although authors have noted that group- and individual-level factors influence learning outcomes (Cho & Egan, 2009; Loyens et al., 2012), this framework proposes a comprehensive approach for examining contextual effects not typically included in PBL or AL evaluation studies. To produce the evidence needed to build confidence in leadership development initiatives, practitioners can include measures of participant characteristics, problem characteristics, process variables, and outcomes as an integral part of implementation. By including measures for all parts of the integrative framework—before, during, and after implementation—program designs can be compared to improve our understanding of what works and why. Deciding what to measure, and how and when to measure it, is not only a strategic move toward evidence-based design but also a mechanism for achieving learning objectives.

One assumption underlying this suggested approach is that the design characteristics and overall effectiveness of constructivist learning approaches like PBL and AL can be evaluated successfully using positivist, quantitative methods. The constructivist learning foundations of PBL and AL might seem to be in stark contrast to the positivist lens used here to define the integrative framework. However, the PBL literature provides numerous quantitative studies using modeling and experimentation to examine design effectiveness (see Hung, 2011; van Berkel & Schmidt, 2000). In their meta-analysis of 270 PBL curriculum comparisons, Schmidt and colleagues (2009) pointed to the high-stakes nature of medical education and the medical field’s tradition of large-scale investigations into treatment effects as reasons why there have been numerous PBL comparison studies: “From this, it is a small step to apply this approach to educational treatments as well” (p. 231). Because of PBL’s popularity in education programs, researchers have access to large student populations and standard measures of outcomes (e.g., tests of knowledge and skills) that are less common in organizational settings.

By contrast, AL research has been dominated by qualitative program descriptions, and there are few peer-reviewed empirical studies that report findings from high-quality research (Cho & Egan, 2009). Of the 50 empirical studies identified by Cho and Egan (2009), 74% used a case study design, a trend that earlier literature reviews also had noted. AL research must become more inclusive of methodologies that will allow for experiments and questions to be answered about the impact of specific practices on leader and organizational performance (Allio, 2005; Cho & Egan, 2010; Garavan et al., 2015). Steps to improve evaluation methods while adding more precision by defining and measuring design characteristics with greater consistency and accuracy may allow HRD professionals to make more decisive statements about leadership development effectiveness that is called for in the literature (Garavan et al., 2015; Hannum & Craig, 2010). To advance our understanding of experience-driven leadership development, quantitative studies, careful operationalization of variables with reporting of instrument validity and reliability, and additional methods are needed so that effect sizes can be generated and compared (Cho & Egan, 2009; Hezlett, 2016; Hung, 2011)

Table 3 presents a sample measurement schedule that could be used to plan for and track measures. This table reveals the challenges faced by researchers and practitioners as they consider how to thoroughly assess a complex intervention like AL and PBL. It is overwhelming to see the number of variables and measurement frequency required to examine how and why these designs work. Fortunately, a variety of sources, such as multirater feedback tools, self-assessments, performance management systems, learning journals, facilitator observations, and program evaluation surveys, can be used to gather such data in organization contexts. And, measures can be used for multiple purposes. For example, participant characteristics and perceived problem affordances are inputs for design and also useful assessment results to discuss with participants as they experience PBL, make sense of their reactions, and develop their SDL skills. Ongoing measures of learner motivation, reflection, SDL skill, and team collaboration may indicate opportunities for facilitator intervention or a need for additional scaffolding mechanisms. Several studies report that learners initially feel anxious and frustrated with the PBL process, and that an environment fostering risk-taking and learning from mistakes is critical for effective implementation (Barrows & Mitchell, 1975; Hung et al., 2008; Johnson & Spicer, 2006; Miflin et al., 2000; Raelin & Raelin, 2006). Repeated measures of these variables can help facilitators monitor participant engagement, understand the causes of participant reactions, and introduce resources or instruction when needed. Entries from reflection journals, blogs, or discussion posts can be evaluated to measure levels of reflection and metacognitive skill development.

Table 3.

Sample Measurement Schedule for Evidence-Based PBL and AL Design and Implementation.

Measures	PBL design and implementation phases
Measures	Pre-PBL launch	Problem introduction	Problem representation	Solution generation/selection	Solution design	Solution implementation	Solution evaluation	Post-PBL assessment
AL/PBL motivation	x	x	x	x	x	x	x	x
Learning orientation	x							x
Experience/background	x
Leadership skills assessment	x							x
SDL skills assessment	x		x	x	x	x	x	x
Collaboration skills assessment	x			x		x		x
Problem characteristics (expert analysis)	x
Problem characteristics (participant analysis)		x						x
Awareness of scaffolding/resources		x
Perceived autonomy		x		x		x		x
Team collaboration		x	x	x	x	x	x	x
Reflection (depth/type)		x	x	x	x	x	x
Use/effectiveness of scaffolding and facilitation		x	x	x	x	x	x	x

Note. PBL = problem-based learning; AL = action learning; SDL = self-directed learning.

With the appropriate levels of the variables established in the framework, researchers can apply hierarchical linear modeling to examine individual- and group-level relationships (Scott, 2014). To effectively apply this technique, validated measures are needed to examine these design characteristics. Although authors have measured process and outcome variables like reflection and critical thinking using participant surveys (e.g., Phan, 2009), alternative approaches to evaluate learning artifacts or behaviors would provide stronger evidence and allow for triangulation of data collected from surveys. PBL research such as the study by Visschers-Pleijers, Dolmans, De Leng, Wolfhagen, and van Der Vleuten (2006) provides a model for how variables like collaborative learning behaviors can be coded to understand team processes.

The framework provided in this article is intended to encourage PBL and AL scholars to leverage the efforts across their respective fields to improve the design of leadership development programs. There are more similarities than differences between PBL and AL approaches. In their review of the AL literature, Cho and Egan (2009, 2010) identified the need for additional conceptual and theoretical development to advance research. Using this integrative framework, PBL and AL scholars can improve our understanding about how these initiatives work and encourage consistent measurement strategies that will produce evidence for ongoing design.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Kimberly S. Scott is Assistant Professor and Director of the Master’s Program in Learning & Organizational Change at Northwestern University. Her work focuses on improving workplace environments and practices that foster employee wellness, learning, and success. More recently, she has examined the transition of individuals into leadership roles, and how learning environments can be designed to accelerate leadership development.

References

Allio

R. J.

(2005). Leadership development: Teaching versus learning. Management Decision, 43, 1071-1077.

Amagoh

(2009). Leadership development and leadership effectiveness. Management Decision, 47, 989-999. doi:10.1108/00251740910966695

Anderson

Coleman

(2015). Action learning: Approaches, applications, and outcomes. In Kraiger

Passmore

dos Santos

N. R.

Malvezzi

(Eds.), The Wiley Blackwell handbook of the psychology of training, development, and performance improvement (pp. 261-277). Chichester, UK: John Wiley.

Araz

Sungur

(2007). The interplay between cognitive and motivational variables in a problem-based learning environment. Learning and Individual Differences, 17, 291-297. doi:10.1016/j.lindif.2007.04.003

Ardichvili

och Dag

K. N.

Manderscheid

(2016). Leadership development: Current and emerging models and practices. Advances in Developing Human Resources, 18, 275-285. doi:10.1177/1523422316645506

Armstrong

S. J.

Mahmud

(2008). Experiential learning and the acquisition of managerial tacit knowledge. Academy of Management Learning & Education, 7, 189-208.

Ates

Eryilmaz

(2010). Factors affecting performance of tutors during problem-based learning implementations. Procedia—Social and Behavioral Sciences, 2, 2325-2329.

Bannan-Ritland

(2001). Teaching instructional design: An action learning approach. Performance Improvement Quarterly, 14(2), 37-52. doi:10.1111/j.1937-8327.2001.tb00208.x

Barrows

H. S.

(1980). Problem-based learning: An approach to medical education. New York, NY: Springer.

10.

Barrows

H. S.

(1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481-486. doi:10.1111/j.1365-2923.1986.tb01386.x

11.

Barrows

H. S.

Mitchell

D. L.

(1975). An innovative course in undergraduate neuroscience. Experiment in problem-based learning with “problem boxes.” British Journal of Medical Education, 9, 223-230.

12.

Brook

Milner

(2014). Reflections on “creative” action learning in business education: Some issues in its theory and practice. Teaching in Higher Education, 19, 126-137. doi:10.1080/13562517.2013.827651

13.

Brotheridge

C. M.

Long

(2007). The “real-world” challenges of managers: Implications for management education. The Journal of Management Development, 26, 832-842.

14.

Brownell

Jameson

D. A.

(2004). Problem-based learning in graduate management education: An integrative model and interdisciplinary application. Journal of Management Education, 28, 558-577. doi:10.1177/1052562904266073

15.

Button

S. B.

Mathieu

J. E.

Zajac

D. M.

(1996). Goal orientation in organizational research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67, 26-48.

16.

Chaharbaghi

Cox

(1995). Problem-based learning: Potential and implementation issues. British Journal of Management, 6, 249-256.

17.

Cho

Egan

T. M.

(2009). Action learning research: A systematic review and conceptual framework. Human Resource Development Review, 8, 431-462. doi:10.1177/1534484309345656

18.

Cho

Egan

T. M.

(2010). The state of the art of action learning research. Advances in Developing Human Resources, 12, 163-180. doi:10.1177/1523422310367881

19.

Clarke

Thorpe

Anderson

Gold

(2006). It’s all action, it’s all learning: Action learning in SMEs. Journal of European Industrial Training, 30, 441-455. doi:10.1108/03090590610688825

20.

Dalakoura

(2010). Differentiating leader and leadership development: A collective framework for leadership development. The Journal of Management Development, 29, 432-441. doi:10.1108/02621711011039204

21.

Day

D. V.

(2000). Leadership development: A review in context. The Leadership Quarterly, 11, 581-613. doi:10.1016/S1048-9843(00)00061-8

22.

Day

D. V.

Fleenor

J. W.

Atwater

L. E.

Sturm

R. E.

McKee

R. A.

(2014). Advances in leader and leadership development: A review of 25 years of research and theory. The Leadership Quarterly, 25, 63-82. doi:10.1016/j.leaqua.2013.11.004

23.

DeGeest

Brown

K. G.

(2011). The role of goal orientation in leadership development. Human Resource Development Quarterly, 22, 157-175. doi:10.1002/hrdq.20072

24.

DeRue

D. S.

Wellman

(2009). Developing leaders via experience: The role of developmental challenge, learning orientation, and feedback availability. Journal of Applied Psychology, 94, 859-875.

25.

Dierdorff

E. C.

Ellington

J. K.

(2012). Members matter in team training: Multilevel and longitudinal relationships between goal orientation, self-regulation, and team outcomes. Personnel Psychology, 65, 661-703. doi:10.1111/j.1744-6570.2012.01255.x

26.

Dolmans

D. H. J. M.

Schmidt

H. G.

(2006). What do we know about cognitive and motivational effects of small group tutorials in problem-based learning? Advances in Health Sciences Education, 11, 321-336. doi:10.1007/s10459-006-9012-8

27.

Dweck

C. S.

(1986). Motivational processes affecting learning. American Psychologist, 41, 1040-1048. doi:10.1037/0003-066X.41.10.1040

28.

Dweck

C. S.

Leggett

E. L.

(1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273. doi:10.1037/0033-295X.95.2.256

29.

Edmonstone

(2015). The challenge of evaluating action learning. Action Learning: Research and Practice, 12, 131-145. doi:10.1080/14767333.2015.1041452

30.

Feldman

D. C.

(2005). The food’s no good and they don’t give us enough: Reflections on Mintzberg’s critique of MBA education. Academy of Management Learning & Education, 4, 217-220.

31.

Garavan

T. N.

McGuire

O’Donnell

(2004). Exploring human resource development: A levels of analysis approach. Human Resource Development Review, 3, 417-441.

32.

Garavan

T. N.

O’Brien

Watson

(2015). Leadership development and organizational success. In Kraiger

Passmore

DosSantos

N. R.

Malvezzi

(Eds.), The Wiley Blackwell handbook of the psychology of training, development, and performance improvement (pp. 354-397). Chichester, UK: John Wiley.

33.

Gijbels

Dochy

den Bossche

P. V.

Segers

(2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75, 27-61.

34.

Gijselaers

W. H.

Schmidt

H. G.

(1990). Towards a causal model of student learning within the context of a problem-based curriculum. In Nooman

Schmidt

H. G.

Ezzat

E. S.

(Eds.), Innovation in medical education: An evaluation of its present status (pp. 95-114). New York, NY: Springer.

35.

Gosling

Mintzberg

(2006). Management education as if both matter. Management Learning, 37, 419-428.

36.

Hallinger

(2011). Assessing the instructional effectiveness of problem-based management education in Thailand: A longitudinal evaluation. Management Learning, 42, 279-299. doi:10.1177/1350507610388596

37.

Hannum

K. M.

Craig

S. B.

(2010). Introduction to special issue on leadership development evaluation. The Leadership Quarterly, 21, 581-582. doi:10.1016/j.leaqua.2010.06.001

38.

Hezlett

S. A.

(2016). Enhancing experience-driven leadership development. Advances in Developing Human Resources, 18, 369-389. doi:10.1177/1523422316645887

39.

Hmelo

C. E.

Lin

(2000). Becoming self-directed learners: Strategy development in problem-based learning. In Evensen

D. H.

Hmelo

C. E.

(Eds.), Problem-based learning: A research perspective on learning interactions (pp. 227-250). Mahwah, NJ: Lawrence Erlbaum.

40.

Hmelo-Silver

C. E.

(2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16, 235-266.

41.

Hmelo-Silver

C. E.

Barrows

H. S.

(2008). Facilitating collaborative knowledge building. Cognition and Instruction, 26, 48-94.

42.

Hmelo-Silver

C. E.

Chernobilsky

Jordan

(2008). Understanding collaborative learning processes in new learning environments. Instructional Science, 36, 409-430. doi:10.1007/s11251-008-9063-8

43.

Hmelo-Silver

C. E.

Duncan

R. G.

Chinn

C. A.

(2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99-107.

44.

House

Rousseau Thomas-Hunt

(1995). The meso paradigm: A framework for the integration of micro and macro organizational behavior. Research in Organizational Behavior, 17, 71-115.

45.

Hung

(2006). The 3C3R model: A conceptual framework for designing problems in PBL. Interdisciplinary Journal of Problem-Based Learning, 1(1), 55-77.

46.

Hung

(2009). The 9-step problem design process for problem-based learning: Application of the 3C3R model. Educational Research Review, 4, 118-141. doi:10.1016/j.edurev.2008.12.001

47.

Hung

(2011). Theory to reality: A few issues in implementing problem-based learning. Educational Technology Research & Development, 59, 529-552. doi:10.1007/s11423-011-9198-1

48.

Hung

Jonassen

D. H.

Liu

(2008). Problem-based learning. In Spector

J. M.

Merrill

M. D.

van Merriënboer

Driscoll

(Eds.) Handbook of research on educational communications and technology (3rd ed.,Vol. 1, pp. 485-506). New York: Erlbaum.

49.

Innes

R. B.

(2006). What can learning science contribute to our understanding of the effectiveness of problem-based learning in groups? Journal of Management Education, 30, 751-764.

50.

Janssen

Poortvliet

P. M.

Van

Vliert

Van Yperen

N. W.

(2007). Achievement goals and interpersonal behavior: How mastery and performance goals shape information exchange. Personality and Social Psychology Bulletin, 33, 1435-1447.

51.

Johnson

(1998). The essential principles of action learning. Journal of Workplace Learning, 10, 296-300.

52.

Johnson

Spicer

D. P.

(2006). A case study of action learning in an MBA program. Education & Training, 48, 39-54.

53.

Kek

M. Y. C. A.

Huijser

(2011). The power of problem-based learning in developing critical thinking skills: Preparing students for tomorrow’s digital futures in today’s classrooms. Higher Education Research & Development, 30, 329-341. doi:10.1080/07294360.2010.501074

54.

Kolb

A. Y.

Kolb

D. A.

(2009). The learning way: Meta-cognitive aspects of experiential learning. Simulation & Gaming, 40, 297-327.

55.

Leonard

H. S.

Lang

(2010). Leadership development via action learning. Advances in Developing Human Resources, 12, 225-240. doi:10.1177/1523422310367800

56.

Leonard

H. S.

Marquardt

(August, 2009). The evidence for the effectiveness of action learning for leadership development. Academy of Management Annual Meeting, Chicago, IL.

57.

Liu

(2011). Motivating protégés’ personal learning in teams: A multilevel investigation of autonomy support and autonomy orientation. Journal of Applied Psychology, 96, 1195-1208. doi:10.1037/a0024716

58.

Lohman

M. C.

(2002). Cultivating problem-solving skills through problems-based approaches to professional development. Human Resource Development Quarterly, 13, 243-261.

59.

Loyens

S. M. M.

Kirschner

P. A.

Paas

(2012). Problem-based learning. In Harris

K. R.

Graham

Urdan

Bus

A. G.

Major

Swanson

H. L.

(Eds.), APA educational psychology handbook, Vol. 3: Application to teaching and learning (pp. 403-425). Washington, DC: American Psychological Association.

60.

Loyens

S. M. M.

Magda

Rikers

R. M. J. P

. (2008). Self-directed learning in problem-based learning and its relationships with self-regulated learning. Educational Psychology Review, 20, 411-427. doi:10.1007/s10648-008-9082-7

61.

Mamede

Schmidt

H. G.

Norman

G. R.

(2006). Innovations in problem-based learning: What can we learn from recent studies? Advances in Health Sciences Education, 11, 403-422. doi:10.1007/s10459-006-9018-2

62.

Marquardt

M. J.

(1999). Action learning in action: Transforming problems and people for world-class organizational learning. Mountain View, CA: Davies-Black.

63.

Marquardt

M. J.

(2011). Optimizing the power of action learning: Real-time strategies for developing leaders, building teams and transforming organizations. Yarmouth, MA: Nicholas Brealey America. Retrieved from http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10448431

64.

Marquardt

M. J.

Banks

(2010). Theory to practice: Action learning. Advances in Developing Human Resources, 12, 159-162. doi:10.1177/1523422310367878

65.

Marquardt

M. J.

Waddill

(2004). The power of learning in action learning: A conceptual analysis of how the five schools of adult learning theories are incorporated within the practice of action learning. Action Learning: Research and Practice, 1, 185-202.

66.

Marsick

V. J.

O’Neil

(1999). The many faces of action learning. Management Learning, 30, 159-176.

67.

Mayer

R. E.

(2002). Invited reaction: Cultivating problem-solving skills through problem-based approaches to professional development. Human Resource Development Quarterly, 13, 263-269.

68.

Mehrabani

S. E.

Mohamad

N. A.

(2015). New approach to leadership skills development (developing a model and measure). The Journal of Management Development, 34, 821-853.

69.

Miflin

B. M.

Campbell

C. B.

Price

D. A.

(2000). A conceptual framework to guide the development of self-directed, lifelong learning in problem-based medical curricula. Medical Education, 34, 299-306. doi:10.1046/j.1365-2923.2000.00564.x

70.

Molleman

(2005). Diversity in demographic characteristics, abilities and personality traits: Do faultlines affect team functioning? Group Decision and Negotiation, 14, 173-193. doi:10.1007/s10726-005-6490-7

71.

Navarro

(2008). The MBA core curricula of top-ranked U.S. business schools: A study in failure? Academy of Management Learning & Education, 7, 108-123.

72.

Nesbit

P. L.

(2012). The role of self-reflection, emotional management of feedback, and self-regulation processes in self-directed leadership development. Human Resource Development Review, 11, 203-226. doi:10.1177/1534484312439196

73.

Neufeld

V. R.

Barrows

H. S.

(1974). The “McMaster Philosophy”: An approach to medical education. Journal of Medical Education, 49, 1040-1050.

74.

Nieminen

Sauri

Lonka

(2006). On the relationship between group functioning and study success in problem-based learning. Medical Education, 40, 64-71. doi:10.1111/j.1365-2929.2005.02344.x

75.

Norman

G. R.

Schmidt

H. G.

(1992). The psychological basis of problem-based learning. Academic Medicine, 67, 557-565.

76.

O’Leonard

Loew

(2012). Leadership development factbook 2012: Benchmarks and trends in U.S. leadership development. Oakland, CA: Bersin.

77.

O’Neil

Marsick

V. J.

(2007). Understanding action learning. Saranac Lake, NY: American Management Association. Retrieved from http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10195623

78.

Pease

M. A.

Kuhn

(2011). Experimental analysis of the effective components of problem-based learning. Science Education, 95, 57-86. doi:10.1002/sce.20412

79.

Pedler

Burgoyne

Brook

(2005). What has action learning learned to become? Action Learning: Research and Practice, 2, 49-68.

80.

Phan

H. P.

(2009). Exploring students’ reflective thinking practice, deep processing strategies, effort, and achievement goal orientations. Educational Psychology, 29, 297-313. doi:10.1080/01443410902877988

81.

Pieterse

A. N.

van Knippenberg

van Ginkel

W. P.

(2011). Diversity in goal orientation, team reflexivity, and team performance. Organizational Behavior and Human Decision Processes, 114, 153-164. doi:10.1016/j.obhdp.2010.11.003

82.

Raelin

J. A.

(1997a). A model of work-based learning. Organization Science, 8, 563-578.

83.

Raelin

J. A.

(1997b). Individual and situational precursors of successful action learning. Journal of Management Education, 21, 368-394.

84.

Raelin

J. A.

(2007). Toward an epistemology of practice. Academy of Management Learning & Education, 6, 495-519.

85.

Raelin

J. A.

(2008). Work-based learning: Bridging knowledge and action in the workplace (New and Rev. ed.). San Francisco, CA: Jossey-Bass.

86.

Raelin

J. A.

(2009). The practice turn-away: Forty years of spoon-feeding in management education. Management Learning, 40, 401-410. doi:10.1177/1350507609335850

87.

Raelin

J. A.

Coghlan

(2006). Developing managers as learners and researchers: Using action learning and action research. Journal of Management Education, 30, 670-689.

88.

Raelin

J. A.

Raelin

J. D.

(2006). Developmental action learning: Toward collaborative change. Action Learning: Research and Practice, 3, 45-67.

89.

Raudenbush

Marquardt

(2008). Growing leaders at the U.S. Department of Agriculture: A case study of leadership development using action learning. Public Administration Quarterly, 32, 147-173.

90.

Reiser

B. J.

(2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13, 273-304.

91.

Revans

R. W.

(1982). The origin and growth of action learning. Brickley, UK: Chartwell-Bratt.

92.

Roglio

Light

(2009). Executive MBA programs: The development of the reflective executive. Academy of Management Learning & Education, 8, 156-173.

93.

Rotgans

J. I.

Schmidt

H. G.

(2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16, 465-479. doi:10.1007/s10459-011-9272-9

94.

Samuelson

(2006). The new rigor: Beyond the right answer. Academy of Management Learning & Education, 5, 356-365.

95.

Schmidt

H. G.

(1983). Problem-based learning: Rationale and description. Medical Education, 17, 11-16.

96.

Schmidt

H. G.

(1993). Foundations of problem-based learning: Some explanatory notes. Medical Education, 27, 422-432.

97.

Schmidt

H. G.

Loyens

S. M. M.

van Gog

Paas

(2007). Problem-based learning is compatible with human cognitive architecture: Commentary on Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 91-97.

98.

Schmidt

H. G.

Moust

J. H. C.

(2000). Factors affecting small-group learning: A review of the research. In Evensen

D. H.

Hmelo

C. E.

(Eds.), Problem-based learning: A research perspective on learning interactions (pp. 19-52). Mahwah, NJ: Lawrence Erlbaum.

99.

Schmidt

H. G.

Rotgans

J. I.

Yew

E. H.

(2011). The process of problem-based learning: What works and why. Medical Education, 45, 792-806. doi:10.1111/j.1365-2923.2011.04035.x

100.

Schmidt

H. G.

van der Molen

H. T.

te Winkel

W. W. R.

Wijnen

W. H. F. W

. (2009). Constructivist, problem-based learning does work: A meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist, 44, 227-249. doi:10.1080/00461520903213592

101.

Scott

(2014). A multilevel analysis of problem-based learning design characteristics. Interdisciplinary Journal of Problem-Based Learning, 8(2). doi:10.7771/1541-5015.1420

102.

Senko

Hulleman

C. S.

Harackiewicz

J. M.

(2011). Achievement goal theory at the crossroads: Old controversies, current challenges, and new directions. Educational Psychologist, 46, 26-47. doi:10.1080/00461520.2011.538646

103.

Smith

P. A. C.

O’Neil

(2003). A review of action learning literature 1994-2000: Part 1—Bibliography and comments. Journal of Workplace Learning, 15, 63-69.

104.

Sockalingam

Rotgans

J. I.

Schmidt

H. G.

(2011). The relationships between problem characteristics, achievement-related behaviors, and academic achievement in problem-based learning. Advances in Health Sciences Education, 16, 481-490. doi:10.1007/s10459-010-9270-3

105.

Sockalingam

Schmidt

(2011). Characteristics of problems for problem-based learning: The students’ perspective. Interdisciplinary Journal of Problem-Based Learning, 5(1), 6-33.

106.

Sofo

Yeo

R. K.

Villafañe

(2010). Optimizing the learning in action learning: Reflective questions, levels of learning, and coaching. Advances in Developing Human Resources, 12, 205-224.

107.

Soppe

Schmidt

H. G.

Bruysten

R. J. M. P

. (2005). Influence of problem familiarity on learning in a problem-based course. Instructional Science, 33, 271-281. doi:10.1007/s11251-004-7688-9

108.

Strobel

van Barneveld

(2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1), 44-58.

109.

Svinicki

M. D.

(2007). Moving beyond “it worked”: The ongoing evolution of research on problem-based learning in medical education. Educational Psychology Review, 19, 49-61. doi:10.1007/s10648-006-9040-1

110.

Tushman

M. L.

Fenollosa

McGrath

D. N.

O’Reilly

Kleinbaum

A. M.

(2007). Relevance and rigor: Executive education as a lever in shaping practice and research. Academy of Management Learning & Education, 6, 345-362.

111.

Ungaretti

Thompson

K. R.

Miller

Peterson

T. O.

(2015). Problem-based learning: Lessons from medical education and challenges for management education. Academy of Management Learning & Education, 14, 173-186. doi:10.5465/amle.2013.0245

112.

van Berkel

H. J. M.

Schmidt

H. G

. (2000). Motivation to commit oneself as a determinant of achievement in problem-based learning. Higher Education, 40, 231-242.

113.

van Blankenstein

F. M.

Dolmans

D. H. J. M.

van der Vleuten

C. P. M.

Schmidt

H. G

. (2011). Which cognitive processes support learning during small-group discussion? The role of providing explanations and listening to others. Instructional Science, 39, 189-204. doi:10.1007/s11251-009-9124-7

114.

van den Hurk

M. M.

Dolmans

D. H. J. M.

Wolfhagen

I. H. A. P.

van der Vleuten

C. P. M

. (2001). Testing a causal model for learning in a problem-based curriculum. Advances in Health Sciences Education, 6, 141-149.

115.

van Knippenberg

Schippers

M. C

. (2007). Work group diversity. Annual Review of Psychology, 58, 515-541. doi:10.1146/annurev.psych.58.110405.085546

116.

Vince

(2004). Action learning and organizational learning: Power, politics and emotion in organizations. Action Learning: Research and Practice, 1, 63-78.

117.

Vince

Martin

(1993). Inside action learning: An exploration of the psychology and politics of the action learning model. Management Education and Development, 24, 205-215.

118.

Visschers-Pleijers

A. J. S. F.

Dolmans

D. H. J. M.

De Leng

B. A.

Wolfhagen

I. H. A. P.

Van Der Vleuten

C. P. M.

(2006). Analysis of verbal interactions in tutorial groups: A process study. Medical Education, 40, 129-137.

119.

Volz-Peacock

Carson

Marquardt

(2016). Action learning and leadership development. Advances in Developing Human Resources, 18, 318-333. doi:10.1177/1523422316645884

120.

Waddill

D. D.

Banks

Marsh

(2010). The future of action learning. Advances in Developing Human Resources, 12, 260-279. doi:10.1177/1523422310367809

121.

Waddill

D. D.

Marquardt

(2003). Adult learning orientations and action learning. Human Resource Development Review, 2, 406-429.

122.

Walker

Leary

(2009). A problem based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 12-43.

123.

Wijnia

Loyens

S. M. M.

Derous

(2011). Investigating effects of problem-based versus lecture-based learning environments on student motivation. Contemporary Educational Psychology, 36, 101-113. doi:10.1016/j.cedpsych.2010.11.003

124.

Wirkala

Kuhn

(2011). Problem-based learning in K-12 education: Is it effective and how does it achieve its effects? American Educational Research Journal, 48, 1157-1186. doi:10.3102/0002831211419491

125.

Yeo

R. K.

Gold

(2011). The inseparability of action and learning: Unravelling Revans’ action learning theory for Human Resource Development (HRD). Human Resource Development International, 14, 511-526. doi:10.1080/13678868.2011.604956

126.

Yeo

R. K.

Nation

U. E.

(2010). Optimizing the action in action learning: Urgent problems, diversified group membership, and commitment to action. Advances in Developing Human Resources, 12, 181-204.