Abstract
It is now 25 years since “problem-based learning” (PBL) first entered the lexicon of school leadership preparation (Bridges, 1989). Initial efforts to implement PBL in Master and Doctoral degree programs at Stanford and Vanderbilt Universities, respectively, aimed at adapting innovative learning methods pioneered in medical education for the preparation of school leaders (Bridges, 1989, 1992; Bridges & Hallinger, 1993, 1995). The impetus for rethinking school leader preparation was grounded in a growing acceptance that the methods commonly employed in existing university-based programs were weak at best, and dysfunctional at worst (Bridges, 1977, 2012; Griffiths, 1988; Murphy, 1993). Daniel Griffiths (1998), for example, concluded, “Most programs for training school administrators range in quality from embarrassing to disastrous” (p. 6).
In the late 1980s and early 1990s, we proposed PBL as an alternative teaching–learning approach that sought to create an active, problem-focused, practice-oriented environment for management education (Bridges, 1989, 1992; Bridges & Hallinger, 1993, 1995). Analyses of school leadership preparation conducted since the 1990s have highlighted the subsequent use of PBL in the United States, Australia, Europe, and Asia (e.g., Bridges, 2012; Jackson & Kelley, 2002; McCarthy, 1999; Murphy, 1993, 2006; Orr & Orphanos, 2011; Walker, Bryant, & Lee, 2013; Young, Crow, Murphy, & Ogawa, 2009). Thus, at the outset of this study, we were surprised to find not even one “systematic review of research” on the use of PBL in educational leadership. This stands in marked contrast with fields such as medicine, engineering, and nursing where systematic reviews of research have guided the refinement of PBL program design, curriculum development, and teaching–learning processes (e.g., Albanese & Mitchell, 1993; Dolmans, De Grave, Wolfhagen, & Van Der Vleuten, 2005; Neville, 2009). Consequently, we believed it timely to conduct a systematic review of research on the use of PBL in educational leadership.
Our review of research addressed three guiding questions:
What is the composition of the research literature on the use of PBL in school leadership preparation and development in terms of volume, scope, sources, research methods, and geographic locales for implementation?
What have we learned about the process and outcomes of problem-based preparation and development of school leaders?
What are productive directions for future research and practice in using PBL in the preparation and development of school leaders?
This “study of the literature” employed a systematic approach to reviewing research (Evidence for Policy and Practice Information and Co-ordinating Centre, 2012; Gough, 2007; Hallinger, 2013, 2014). We examined the global literature on PBL in educational leadership preparation and development programs from its initial adoption in 1989 (Bridges, 1989) to the present. The review database consisted of 73 sources. We employed systematic methods for analyzing and synthesizing findings drawn largely from qualitative and case study research (e.g., Gough, 2007; Hallinger, 2013).
Conceptual Perspective
The definition of PBL employed in this review relied on Bridges and Hallinger’s (1993, 1995) adaptation of PBL for use in management education (see Albanese & Mitchell, 1993). Our model of problem-based management education proposed six defining characteristics.
The starting point for learning is a problem, not a theory.
Learning is stimulated by and organized around a high impact problem that students are apt to face in the workplace.
Curriculum units are organized around problems rather than subject disciplines.
Students assume major responsibility for managing their own learning in teams.
Most learning occurs in the context of student-directed small groups rather than teacher-directed lectures.
To the extent practical, students “demonstrate” simulated solutions to the problem (rather than just analyses) and receive formative feedback on their efforts (Bridges & Hallinger, 1995; Hallinger & Bridges, 2007).
The conceptual framework used to inform our analysis is presented in Figure 1. This framework conceptualizes PBL as a learning method embedded within a particular context. Key features of the “context” include the organization and program context (Griffiths, 1988; Young et al., 2009), professional norms (Bridges, 1977, 2012; Murphy, 2006), and sociocultural values (e.g., Hallinger & Kantamara, 2001; Walker, Bridges, & Chan, 1996). For example, university-based preparation (e.g., Bridges, 1992; Gupton, 1998) and field-based professional development (e.g., Ford, Martin, Muth, & Steinbrecher, 1997) programs represent different “contexts” for using PBL in the education of school leaders. Professional norms and sociocultural norms vary from society to society and shape the roles and practices expected of school leaders. These context differences could also potentially affect both teacher and learner attitudes, motivation, and engagement.
Conceptual framework for examining problem-based learning in school leadership preparation.
The personal characteristics of learners are also potentially relevant to users of PBL. For example, the effectiveness of PBL as a learning method could vary for learners of different organizational roles, gender, ages, levels of experience, personality types, and cultural orientations (see Hallinger & Kantamara, 2001; Mergendoller, Maxwell, & Bellisimo, 2006; Walker et al., 1996). These differences could present different challenges for program designers and shape learner responses to PBL.
PBL, as a learning method, can be implemented in different curricular and instructional configurations (Bridges, 1992, 1993; Bridges & Hallinger, 1995; Hallinger & Bridges, 2007). For example, educational leadership instructors have reported using PBL as a “teaching and learning method” in a single course or class unit (see Cordeiro & Campbell, 1995, 1996; Dimmock & Edwards, 1996; Griswold, 2006; Habschmidt, 1991; Hallinger & Bridges, 2007; Johnson, 2007). In other cases, preparation programs have conceptualized PBL as a pedagogical framework for a “track” within a curriculum (see Hallinger & Bridges, 2007; Hallinger & Lu, 2011a, 2011b) or as a pedagogical model running throughout the program (e.g., Bridges, 1992; Bridges & Hallinger, 1995; Ford et al., 1997; Gupton, 1998; Hallinger & Bridges, 2007; Klotz & Daniel, 1998).
Scholars studying PBL in other fields of professional education have not only described these alternative configurations, but also sought to examine their differential effects on learners (see Albanese & Mitchell, 1993; Dolmans et al., 2005; Gijbels, Dochy, Van den Bossche, & Segers, 2005; Hallinger & Lu, 2011a, 2011b; Neville, 2009). These studies emphasize that the “pedagogical power” of PBL is enhanced when students have the opportunity to “learn how to learn” in a PBL environment. Consequently, fields such as medical education have tended to opt for implementing PBL in integrated program designs rather than as a teaching learning method in an individual class (see Hallinger & Bridges, 2007).
Finally, the effects of problem-based management education can be conceptualized in terms of cognitive, skill, and affective impact on learners, as well as on the teachers, students, and schools that they serve (Murphy, 2006). Research on the effects of PBL aligns with the full range of goals that PBL proposes to address (Bridges, 1992; Bridges & Hallinger, 1993, 1995). These include capacities to lead collaboratively, achieve results through others, learn independently, apply management skills (e.g., time management, project management, goal-setting, etc.), engage in reflective practice, and transfer these attitudes and skills to the workplace. This conceptual framework provided a heuristic for organizing our analysis of the studies that comprised the PBL literature in educational leadership and management.
Method
This study was framed as an “exploratory” rather than an “explanatory” review of research (Hallinger, 2013). Exploratory reviews are appropriate when a domain of study does not yet possess a critical mass of studies capable of quantitative integration of research findings (Gough, 2007; Hallinger, 2013, 2014). Therefore, this review aims at describing the composition of the knowledge base, identifying thematic trends, synthesizing findings, and suggesting future directions for research and practice.
Sources for Review
The database for this review included all relevant sources published between the first paper identified in the literature on PBL in school leader preparation (Bridges, 1989) and the end of 2015 (Tang, 2015). The key criterion used to filter sources was a focus on “the use of problem-based learning in the preparation and/or development of school leaders.” We employed Google Scholar™ as our primary search tool due to its capacity to search simultaneously for books, conference papers, journal articles, and theses. This provides a far more comprehensive search than relying on proprietary databases, such as JSTOR™, EBSCO™, or Proquest™, which access large but constrained collections of journals. Moreover, newly enhanced features of Google Scholar™ enable the researcher to vary search terms and dates, and identify related articles very efficiently. Finally, when linked to a university’s subscription database, Google Scholar™ further enhances search efficiency by enabling the immediate download of selected articles (see Fehrmann & Thomas, 2011; Hallinger, 2013).
We carried out more than 20 search iterations using a variety of keywords: PBL, problem-based learning, school leader, educational leader, educational leadership, preparation, development, principal, leadership, and management. When a source appeared suitable, the PDF file was downloaded and stored for closer reading and validation of relevance. This first stage of the Google Scholar™ search yielded 77 sources consisting of journal articles, book chapters, conference papers, books, and dissertations. Next, we carried out a similarly structured search of the Proquest™ Dissertations Online database. This yielded 55 additional doctoral dissertations not identified by Google Scholar™. By the end of the search, we had identified 132 potential sources.
During the filtering stage, we closely reviewed these sources for relevance to our research foci. Here we referred to the six-element operational definition of PBL delineated in the prior section of the article. We offer two examples to illustrate how our definition of problem-based management education was applied. First, our search had identified more than 20 EdD dissertations from Saint Louis University that had incorporated the phrase “Problem-Based Project . . .” in the title or abstract. However, upon closer inspection, we found that only two of these met the criteria in our operational definition of PBL. Therefore, 25 of these theses were eliminated from the review database.
A second example concerned several sources that claimed to study the use of “problem-based computer simulations” (e.g., Hallinger & Kantamara, 2001; Hallinger & McCary, 1990; Ng, 2001; Showanasai, Lu, & Hallinger, 2013; Tang, 2015). Computer simulations represent a potentially different species of learning from PBL. However, we found that in these instances, the instructors had explicitly employed PBL as a pedagogical framework for learning via computer simulation (see also Hallinger, 2005; Showanasai et al., 2013) so these studies were retained in the review database.
Finally, we eliminated duplicate content covered in multiple sources produced by the same author (e.g., a journal article and a book chapter, or a dissertation and a journal article). In the end, we were left with a total of 73 relevant sources, consisting of 45 mixed sources and 28 doctoral dissertations.
Data Extraction
Data extraction entailed selecting, coding, and entering information from each source for storage. First, we extracted and coded descriptive information for entry into a MS Excel spreadsheet (e.g., Journal, Year, Locus, Type of Source, Type of Paper, Research Method, and Doctoral Degree). Second, information from each article was summarized and entered into a MS Word file using a common template to ensure comparability across sources. Next, the reviewers made additional judgments about the source content (e.g., Conceptual Focus, Thematic Focus, Analytical Level, and Research Quality), coded these judgments, and entered data into the spreadsheet.
The fact that our review database was heavily weighted toward unpublished sources was a potential cause for concern (Gough, 2007; Hallinger, 2013). Therefore, we decided to employ an analytical rubric to enhance our evaluation of the quality of sources. In an analytical rubric, the criteria that describe key elements of a performance or product—in this case a research report—are defined in terms of levels of quality. Rubrics for assessing the quality of research studies are widely available and have been used as tools in other systematic reviews of research (e.g., Evidence for Policy and Practice Information and Co-ordinating Centre, 2012; Gough, 2007; Hallinger, 2013, 2014).
We adapted a three-level analytical rubric used to assess research quality (Hallinger, 2014). The rubric focused on five criteria: research questions/goals, conceptual framework, research design, data collection, and data analysis (see Figure 2). Full marks for a study would yield a score of 15 points (i.e., 5 criteria × 3 points). Each study was assessed with the rubric and the scores entered into the Excel spreadsheet.
Analytical rubric used to assess quality of the studies.
Data Analysis
Data analysis proceeded in three stages. In the first stage, we used descriptive statistics to describe the structural composition of this literature. In the second stage, we analyzed each source to determine which relationship(s) implied in the conceptual framework (see Figure 1) represented the conceptual foci of the study (e.g., Context, Context-Classroom, Classroom, etc.).
In the third stage, we identified themes that described the substantive foci of the studies (e.g., PBL and technology, program design, PBL outcomes, classroom implementation, etc.). Themes were identified inductively and refined as we proceeded. Next, we worked toward synthesizing findings within each theme. For example, under the theme of “implementation,” there were numerous reports that documented obstacles faced by program designers and instructors. We listed “obstacles” as a subtheme, and then coded and tracked the frequency with which different obstacles (e.g., faculty lack of knowledge, resources, etc.) were mentioned. Although this approach to “quantifying qualitative findings” aided in a more systematic synthesis of the literature, we also employed our own critical eye to make sense of the findings as we proceeded.
Results
Composition of the Literature
Types of sources
As indicated above, this literature consisted of 73 sources, split between mixed sources (62%) and doctoral dissertations (38%). The mixed sources (see Table 1) were composed of 26 journal articles, 12 conference papers, 4 books, and 3 book chapters. Because we eliminated numerous “duplicate sources” from the database, the figures on book chapters and conference papers understate the actual number of sources in these categories. Twelve of the articles appeared in “mainline” educational leadership journals: Journal of Educational Administration (7), Journal of School Leadership (3), Educational Administration Quarterly (1), International Journal of Educational Management (1). The other articles were spread across an eclectic set of general education journals.
Composition of the PBL Literature in Educational Leadership.
In this analysis, the percentage was based on the number of empirical sources (i.e., 52) rather than total sources (i.e., 73).
The doctoral studies consisted of 26 EdD and 2 PhD dissertations (Copland, 2000; Engvig, 1997). An average of one doctoral dissertation was completed annually, distributed fairly evenly across the 25-year period. Three of the dissertations yielded findings that were subsequently reported in refereed journals (Copland, 2000; Hallinger & Kantamara, 2001; Showanasai et al., 2013).
The heavy concentration of EdD dissertations suggests that most of these doctoral studies were oriented toward understanding aspects of practice rather than theory development or testing. Indeed, 24 of the 26 EdD dissertations were organized around a dissertation model proposed by Bridges and Hallinger (1995) in a book chapter titled “Using PBL as a Focus for EdD Research.” This EdD dissertation model proposed the use of research and development (R&D) methodology (Gall, Borg, & Gall, 1996) to guide the design, field testing, and refinement of PBL “projects.” EdD dissertations using this model were completed at Vanderbilt University, Miami University of Ohio, Portland State University, and Saint Louis University. Notably, these R&D dissertations did not usually examine explicit research questions about the use of PBL as a learning method.
Types of studies and research methods
In the next step, we disaggregated the literature by Type of Studies. As indicated in Table 1, the 73 sources consisted of 54 empirical reports, 12 conceptual papers, 4 commentaries, 2 resource collections, and 1 research synthesis. Empirical reports were defined broadly as studies in which data were collected, analyzed, and reported, regardless of method. Conceptual papers discussed the underpinnings of PBL as a method of teaching and learning in educational leadership. Commentaries critically analyzed PBL use in educational leadership or features of the PBL method. Research reviews set out to examine a body of literature on PBL. Resource collections were compilations of PBL exercises. The high percentage of empirical reports (i.e., 71%) would normally suggest that the literature holds significant potential for learning about the use of PBL in school leader preparation programs. However, when these studies were disaggregated by research method (see Table 1), our initial optimism was placed on hold.
We classified 17 of the empirical studies as using “experiential description” (e.g., Cordeiro & Campbell, 1995; Estes, 1999; Ford et al., 1997). These were accounts of the “PBL implementation experience” authored by program designers and/or instructors. Although the authors often included narrative references to data (e.g., instructor observations, student feedback, and evaluations), there was seldom any description of methods of data collection and analysis. Thus, in our view, these case reports emphasized narrative description rather than empirical analysis.
The remaining empirical studies were composed of 5 qualitative, 6 mixed methods, and 24 R&D reports. 1 In contrast to the broader literature on PBL, the educational leadership literature contained no quantitative findings whatsoever outside of the six mixed methods studies. It should be further noted that few of the mixed methods studies employed strong quantitative research designs.
Consistent with the R&D methodology, the EdD dissertations tended to address “project goals” rather than formal research questions aimed at understanding the use of PBL in preparing school leaders. Thus, we found that only 11 of the 73 sources included in the review were structured as formal empirical investigations of the use of PBL in the preparation of school leaders (i.e., Barbour, 2006; Copland, 2000; Engvig, 1997; Griswold, 2006; Limerick, Clarke, & Daws, 1997; Mayer, Musser, & Remidez, 2001; Ng, 2001; Pajak, Tanner, Rees, & Holmes, 1995; Walker et al., 1996). These features of the literature placed constraints on our subsequent methods of review.
Research quality
After assessing each empirical report with the rubric, we incorporated the data in three ways. First, we analyzed the collective quality of the studies on the five criteria in our rubric (see Table 3). This produced an impression that the literature was generally of low to moderate quality. Indeed, only about half of the 52 studies reached the ‘moderate’ quality level on the rubric. Review of data presented in Table 2 further highlights the use of weak research designs and methods.
Frequency Distribution of Empirical Studies by Criteria on Research Quality Rubric.
Second, we broke the quality assessment out into three categories: experiential, qualitative/mixed methods, R&D studies. The results confirmed that the case reports based on experiential description were the weakest source of knowledge with an average quality index of 6.6 out of 15 points. R&D studies achieved an average of 10.6, with scores ranging from 8.75 to 12 points. The empirical research studies using qualitative and mixed methods attained a moderate quality index of 10.9, with scores ranging from 5 to 14 points.
Quality differences among the R&D reports depended largely on the extent to which the doctoral student had used summative evaluation to assess achievement of student learning objectives and supplementary instruments to shed light on the learning process of students. However, we reiterate that the R&D studies were focused primarily on the design and evaluation of PBL materials and only secondarily as research aimed at contributing to knowledge about PBL.
Finally, we referred to the quality index of each study on the Excel spreadsheet as we worked toward synthesis of findings. Here we sought to check findings obtained from lower quality studies (e.g., the experiential reports) against those drawn from higher rated studies (e.g., Copland, 2000; Engvig, 1997; Ng, 2001).
Synthesis of Substantive Findings
Conceptual features of the literature
Using the conceptual framework presented earlier (see Figure 1), we found that 53 sources (73%) analyzed PBL as a method of classroom learning (see Table 3). Ten papers analyzed connections between features of the context or characteristics of the learners and the use of PBL in school leadership preparation (e.g., Bridges, 2012; Estes, 1999; Ford et al., 1997; Gupton, 1998). Several of these focused on adaptations related to the cultural context (e.g., Tang, 2015; Walker et al., 1996). No scholars explicitly examined how characteristics of the learners influenced their attitudes toward learning via PBL.
Analysis of Conceptual Features of the PBL Literature in Educational Leadership.
Only three studies examined process–outcome relationships (e.g., Barbour, 2006; Copland, 2000). In contrast with the medical education literature, we found few studies of the impact of PBL on individual or collective capacities of students. Studies of distal outcomes such as learner behaviors in the workplace and effects on teachers or the school were absent from this literature. Moreover, very few of the empirical studies were informed by frameworks drawn from curriculum, cognitive, or instructional theory.
In sum, the studies comprising this literature have explored the use of PBL in school leadership preparation from a very limited set of conceptual perspectives. When we examined substantive themes within this literature, the focal issues that consumed the attention of scholars were the design and implementation of PBL, faculty and learners’ responses to using PBL, adaptations of PBL to different cultural contexts, and uses of learning technology in concert with PBL.
PBL design
Although the Bridges and Hallinger (1995) model of PBL predominated in this portion of the literature, authors also described attempts to experiment with hybrid models. This was especially evident in efforts to leverage PBL with learning technology (e.g., Hallinger, 2005; Mann, Reardon, Becker, Shakeshaft, & Bacon, 2011; Mayer et al., 2001), to adapt PBL for use in non-Western cultural contexts (e.g., Siamoo, 2013; Tang, 2015; Walker et al., 1996), and to modify PBL methods for use in professional development (e.g., Chompoowong, 1998; Cranston, 2002, 2008; Ford et al., 1997; Mascitti, Tramonti, & Feituri, 2011).
Over 85% of the PBL projects described in this literature were of the “problem-stimulated” rather than the “student-centered” variety (Bridges & Hallinger, 1995). As described by Bridges and Hallinger (1995),
The major differences between the two types of projects center on who identifies the learning objectives, the resources, and the guiding questions. In problem-stimulated projects, the instructor assumes primary responsibility for identifying the learning objectives, the resources, and the guiding questions. In a student-centered project, the student assumes primary responsibility for these three components. (p. 24)
The small number of “student-centered PBL projects” were designed and employed exclusively in Western cultural contexts such as the United States and Australia. Several authors referred to these student-centered PBL as forms of “authentic learning” or learning from “real problems” as opposed to “simulated problems’ (see Cordeiro & Campbell, 1995; Griswold, 2006). Here the distinction is based on whether the problem presented in the PBL scenario has been “found” by the student(s) or provided by the instructor, not on whether the problem itself is real or contrived (Cordeiro & Campbell, 1995; Griswold, 2006; Mayer et al., 2001; Pajak et al., 1995).
Indeed, as far as the reviewers could determine, all of the problem scenarios embedded in the problem-stimulated projects described by authors in this literature appeared to have been drawn from real situations. A “problem-stimulated” project on using data for decision making would, for example, provide students with a problem scenario about a school and a data set for analysis (e.g., Snidvongs, 2007). In an “authentic” PBL project, each student team would need to “find a problem” in a school and then identify, obtain, and analyze a relevant data set in order to inform their solution to the problem (e.g., Griswold, 2006; Mayer et al., 2001; Pajak et al., 1995). We note that this approach requires both the students and the instructor(s) to be comfortable with even higher levels of uncertainty than in the more common problem-stimulated PBL projects (see also Mann et al., 2011).
Another variant of the student-centered or authentic PBL project was piloted by professors and doctoral students at Portland State University and Saint Louis University. In this species of PBL, the student(s) identifies a “real problem” in a school and then develops a tool that can be used to inform and assist in solving the problem. Using a research and development process (see Bridges & Hallinger, 1996; Gall et al., 1996), the research-informed tool is field tested in a real setting, feedback obtained from practitioners, refined, and then disseminated for use by other practitioners. This model has been used in EdD dissertations where students have developed a school improvement handbook (Coakley, 2013; Draper, 2014), an instrument for facilities assessment (Hackenwerth, Randles, & Tucker, 2009), and a communications planning template (Wingert, Bradfield, & Marriott, 2013). These products of the PBL process were then disseminated as tools to support practitioners in successfully analyzing and developing solutions to common, high impact problems facing school leaders.
The PBL projects described in this literature covered a wide range of problem foci. These included organizational change (e.g., Chenoweth & Everhart, 1994; Hallinger & Kantamara, 2001; Tang, 2015), professional learning communities (Draper, 2014), instructional leadership (e.g., Dimmock & Edwards, 1996; Hallinger & McCary, 1990; Siamoo, 2013), technology implementation (e.g., Cordeiro & Campbell, 1995; Hall, 1997; Hallinger & Kantamara, 2001; Ng, 2001; Tang, 2015), ethics (e.g., Cranston, 2008; Johnson, 2007), equity in student achievement (e.g., Jennings, 1992; Shackleford, 1998), communication (e.g., Bridges, 1992; Cagle & Wiley, 2012; Wingert et al., 2013), school culture (e.g., Barbour, 2006; Coakley, 2013; Habschmidt, 1991; Walker et al., 1996), school improvement (Hallinger & McCary, 1990; Vermillion, 1999), school discipline (Bridges, 1992; Cagwin, 1997), teaching and learning (e.g., Dempsey, 2000), meeting management (Bridges, 1992), parent involvement (e.g., Chompoowong, 1998), facility management (e.g., Cordeiro & Campbell, 1995; Hackenwerth et al., 2009), data-based decision making (Griswold, 2006), teacher selection (Bridges, 1992), and education for second language learners (e.g., Bridges, 1992; Estes, 1999).
Two other innovative forms of PBL project design are worthy of mention: the use of “immersive online web environments” (Mann et al., 2011; Mascitti et al., 2011; Mayer et al., 2001) and computer simulations (e.g., Hallinger & Kantamara, 2001; Hallinger & McCary, 1990; Ng, 2001; Tang, 2015). These variants seek to leverage the pedagogical power of PBL with learning technologies and will be discussed at greater length below.
Implementation
Implementation studies took three main approaches. The first was represented by “case reports” of PBL implementation, typically written by instructors and program designers (e.g., Cordeiro & Campbell, 1995; Estes, 1999; Gupton, 1998; Pajak et al., 1995). The second consisted of EdD dissertation projects (e.g., Chompoowong, 1998; Dempsey, 2000; Habschmidt, 1991) in which the researchers discussed both project design and implementation. The third was the rather small set of empirical research studies (e.g., Barbour, 2006; Copland, 2000; Engvig, 1997; Mayer et al., 2001).
As implied earlier, implementation studies were conducted at both the program (e.g., Bridges, 1992; Copland, 2000; Cranston, 2008; Estes, 1999; Ford et al., 1997; Gupton, 1998; Johnson, 2007; Klotz & Daniel, 1998; Pajak et al., 1995) and classroom levels (e.g., Barbour, 2006; Cagwin, 1997; Dimmock & Edwards, 1996; Griswold, 2006; Habschmidt, 1991; Hackenwerth et al., 2009; Hallinger & Kantamara, 2001; Jennings, 1992; Limerick et al., 1997; Mann et al., 2011; Mayer et al., 2001; Tang, 2015). We reemphasize that our understanding of PBL implementation is informed almost entirely by findings from case reports of uneven quality and short duration.
Obstacles impeding successful implementation reprise those identified in the broader literature on PBL in higher education (e.g., Dolmans et al., 2005; Neville, 2009). Pajak et al. (1995) observed that, “even though individual faculty members understood intellectually what problem-based and student-centered meant to them, they did not anticipate how their differing expectations might be problematic to program implementation” (p. 15). These included,
Insufficient time to prepare the program/class before starting
Student and faculty anxiety concerning their new roles in a PBL classroom environment
Learning new “role behaviors” such as team-learning skills for students and facilitation, coaching, and feedback skills for instructors
Resources to support project implementation
Time flexibility in schedules to enable “block schedules” in which students could work in larger “chunks” of time
Adaptations of mastery-based assumptions used in PBL to norm-based grading systems
Need to learn new methods of assessment suitable for use in PBL
When interpreting these obstacles as well as the responses of students and faculty in the following subsections, the reader should bear in mind that the typical length of these studies was the duration of a single PBL project, a single course, or, in a few cases, a single year of program implementation. Thus, our knowledge base is comprised almost entirely of studies conducted during the “adoption” stage of implementation before faculty or students have achieved a reasonable level of skill in using the method (e.g., Engvig, 1997). Consequently, this literature offers few insights into the sustainability of PBL in educational leadership programs, perceptions of faculty and students after they have become competent users of PBL, or assessments of the impact of PBL on learners over time (e.g., Hallinger & Lu, 2011a, 2011b).
Student responses to PBL
The literature on PBL emphasizes the need for students to assume greater responsibility for directing their own learning. Bridges and Hallinger (1995) emphasized the need for program designers and instructors to give explicit attention to development of an array of new skills (e.g., project management, meeting management, goal setting, problem solving) and attitudes (e.g., risk taking, collaboration) that support independent and team-based learning. Otherwise, they asserted that a class or program is unlikely to achieve the projected benefits of using PBL (Bridges, 1992; Bridges & Hallinger, 1995; Hallinger & Bridges, 2007). This assertion was supported, for example, by Cranston’s (2008) study in which a participant observed how the instructors had enabled “preparedness of the participants to be risk takers, to share ideas . . . within a totally ‘safe’ environment, where each participant has to become the decision-maker as if it were in his or her own school” (p. 591).
When analyzing results concerned with student responses to PBL, we were struck by the consistency of findings across the case reports and empirical studies. In over 90% of the empirical reports, researchers reported a positive attitude towards using PBL in the preparation or development of school leaders, but with a consistent set of caveats. Based on interviews with faculty using PBL, Engvig (1997) found that “the students experienced some initial problems, especially in regard to group interaction. As they became more familiar with the method, they were very appreciative of problem-based learning” (1997, p. 175).
In the eyes of students, PBL appears to be successful in achieving the goal of narrowing the gap between the experience of learning to lead in the classroom and leading in the school (see Bridges, 1977).
Excitement and enthusiasm for grappling with a “real” problem encountered by leaders in the field; grappling with ethical and philosophical issues that were of interest and grounded in a real-world context. The complex and real-world nature of the problems enabled students to appreciate the application of both classic and contemporary ethics to every day leadership. (Johnson, 2007, p. 495)
Notably these findings, expressed in different words, were “repeated” across a wide range of cultural contexts including the United States (Cordeiro & Campbell, 1995, 1996; Ford et al., 1997), Malaysia (Ng, 2001), Thailand (Chompoowong, 1998; Hallinger & Kantamara, 2001; Hallinger & Lu, 2011a, 2011b; Hallinger, Lu, & Showanasai, 2010), China (Tang, 2015), Hong Kong (Showanasai et al., 2013; Walker et al., 1996), and Australia (Cranston, 2008; Dimmock & Edwards, 1996; Limerick et al., 1997). Motivation appeared to carry over into high levels of student engagement as observed by instructors and reported by students (e.g., Chenoweth & Everhart, 1994; Copland, 2000; Cordeiro & Campbell, 1995; Cranston, 2008; Dimmock & Edwards, 1996; Hallinger & Kantamara, 2001; Johnson, 2007; Limerick et al., 1997; Mann et al., 2011; Tang, 2015). Students often reported going above and beyond the stated objectives after they assumed greater control for their learning (e.g., Cordeiro & Campbell, 1995, 1996; Griswold, 2006; Habschmidt, 1991; Hallinger & Kantamara, 2001; Mann et al., 2011; Shackleford, 1998). Motivation appeared to be stimulated by the challenge of solving real problems under time constraints, needing to develop “products” that would be presented to relevant audiences, and the positive pressure that came from working collaboratively in a team.
Nonetheless, even students who felt largely positive about learning through PBL expressed a variety of concerns ranging from mild to quite strong. Lack of structure and inexperience with “learning how to learn” with limited faculty direction created anxiety (Estes, 1999; Griswold, 2006; Johnson, 2007; Limerick et al., 1997). Griswold (2006), for example, noted that “some students felt ‘the unknown’ of the course or ‘lack of clear understanding of the goals’ or ‘uncertainty of what had to be done’ was a weakness. Students often wanted more structure, more specific exercises, and deadlines” (p. 20).
The anxiety of dealing with ambiguous problems was exacerbated by two other frequently mentioned factors. The first was the need to complete the project “product” under time constraints (see Bridges, 1992; Bridges & Hallinger, 1995). A second was the need to develop the product collaboratively rather than individually (Barbour, 2006; Griswold, 2006). This challenged even those students who accepted the importance of learning to lead through others as a requirement for effective leadership (see Limerick et al., 1997).
Aside from these attitudinal responses to PBL drawn from qualitative and case studies, we were dismayed by the scarcity of empirical insights into the impact of PBL on student learning. Studies that inquired into learning outcomes of PBL consisted of several R&D projects and one PhD dissertation. Several R&D projects used a combination of product assessment and knowledge exams to assess learning outcomes (e.g., Chompoowong, 1998; Dempsey, 2000; Habschmidt, 1991; Hallinger & Kantamara, 2001; Jennings, 1992; Ng, 2001; Shackleford, 1998; Tang, 2015). These studies largely concluded that the use of PBL had enabled students to attain the stated learning objectives. Nonetheless, given their design as R&D studies, none employed strong research designs and methods capable of contributing to a solid “evidence base.”
The other study worthy of mention was conducted by Copland (2000). His PhD research examined the development of problem-solving skills attained by three succeeding cohorts of aspiring leaders studying in Stanford University’s principal preparation program in which PBL was featured in about 40% of the curriculum. Measurable differences in the problem-framing skills of students led Copland (2000) to conclude as follows:
[I]nstructional practices that activate prior knowledge and situate learning in contexts similar to those encountered in practice are associated with the development of students’ ability to understand and frame problems. Moreover, the incorporation of debriefing techniques that encourage students’ elaboration of knowledge and reflection on learning appear to help students solidify a way of thinking about problems. (p. 604)
Faculty responses to PBL
Findings from this largely qualitative literature affirm results reported in the broader literature on the range of responses of faculty members who “attempt” to use PBL (Engvig, 1997; Estes, 1999). Faculty require time and structured opportunities to learn new methods of curriculum design, classroom organization, structuring feedback, and assessment of learning (Bridges & Hallinger, 1995; Hallinger & Bridges, 2007). When faculty members receive proper training and follow-up support, their perceptions of the efficacy of PBL tend to be much more positive (e.g., Dimmock & Edwards, 1996; Engvig, 1997; Pajak et al., 1995).
Envig’s (1997) research followed up on the outcomes of training provided to a group of 40 educational leadership faculty members who had received training in the use of PBL at Stanford University in 1993. Envig (1997) found the following results:
All of the participants in the [Stanford PBL] seminar put PBL to use, even if all of them experienced one or more hindering factors. . . . All of them state that they intend to continue to use the method, and very few ever second-guessed their decision to use PBL. As they became more experienced in using the method, many professors expanded their use to additional courses, and some also used the method for comprehensive exams and EdD dissertations. At the same time, many of them from time to time modified one or several of the main elements of the method. (pp. 173-174)
Like students, faculty members also observed the emergence of stronger linkages between classroom learning and workplace practice (e.g., Engvig, 1997; Ford et al., 1997; Johnson, 2007). Cordeiro and Campbell (1996) asserted that,
PBL is one of the few instructional modes that allows social interactions within the context of an actual problem. If students are provided with multiple opportunities for practicing actual problems in real contexts, then there is greater likelihood that learning will be applied to far transfer situations. (p. 24)
Faculty members across different societies also tended to view the “observed outcomes” of PBL as well-suited to the contingencies of 21st-century leadership. For example, they found PBL uniquely aligned to the expectation for leaders to achieve results while working with and through others. In Australia, Limerick et al. (1997) observed,
We have no doubt that [in using PBL] significant valuable lessons are being learnt by students in relation to their sense of personal empowerment, the processual competencies they need to develop and the importance of networks in the development of interdependence. In our view, such concepts and skills are critical for effective leadership in a post-modern society. (pp. 269-270)
Faculty concerns over the use of PBL also emerged in this literature. As suggested above, the extent and seriousness of concerns was to some extent related to the context, time frame, resources, and support received by faculty during implementation. However, even faculty members who chose to use PBL and had received training often expressed concerns. A modal response was articulated by Dimmock and Edwards (1996) in Australia:
In attempting the transition from traditional lecture format to PBL . . . we have experienced considerable insecurity, anxiety and criticism. It has been necessary to learn a new trade. Skills that one has refined and honed over the years suddenly become necessary, but not sufficient. A new set of skills and instructional techniques are required. (p. 323)
The most common concerns expressed by faculty were the following:
How to cover course content due to block scheduling and rearrangement of the curriculum around problems rather than the disciplines
How to effectively engage the role as a “facilitator of learning,” providing students with “the right amount” of coaching and feedback
Adjustment to changing interpersonal dynamics between professors and students
How to manage the related processes of assessment, feedback, and grading
Adaptations of PBL to different cultural contexts
As noted above, PBL has been implemented in school leadership preparation programs in a wide range of national contexts. Consistent with our conceptual framework, scholars have explored how the cultural context influences the responses of teachers and learners to PBL as well as adaptations in societies with strong traditions of teacher-directed instruction (e.g., Chompoowong, 1998; Hallinger & Kantamara, 2001; Tang, 2015; Walker et al., 1996). Although these R&D studies have not generally employed strong research designs, the pattern of findings was quite consistent.
Despite the acknowledged anxieties and trepidations of instructors, students in non-Western contexts generally responded positively to the use of PBL in their leadership preparation programs (e.g., Chompoowong, 1998; Hallinger & Kantamara, 2001; Ng, 2001; Showanasai et al., 2013; Tang, 2015; Walker et al., 1996). Indeed, many of the same themes highlighted in the “Western” literature were reprised by these students, including the following:
Appreciating the opportunity to share knowledge and learn from and with their peers
Gaining strong insights into how they can apply knowledge learned in the classroom to leadership practice in schools
Developing and refining management skills, including project management, team leadership, meeting management, business communication, and using data for decision making
Obstacles and constraining factors related to the use of PBL in non-Western contexts have also been noted. In East Asia, instructors noted that the heavy reliance on cooperative group learning requires considerable attention to cultural norms of hierarchy and collectivism as students learn to lead and work through teams (Chompoowong, 1998; Hallinger & Kantamara, 2001; Tang, 2015; Walker et al., 1996). For example, the Asian norm of collectivism sometimes leads groups to avoid frank and open discussion of issues. Similarly, junior members of groups may tend to avoid speaking up, especially if their views conflict with those of senior members in the group. Nonetheless, when implementing PBL in a structured program, as opposed to a one-off experience, it has been observed that students do gain skills in working and learning effectively and harmoniously in teams (Hallinger & Lu, 2011a, 2011b).
Another important issue highlighted in these studies is the need to ground and/or adapt PBL problem scenarios to the local cultural cum educational context (Hallinger & Bridges, 2007; Hallinger & Lu, 2011a, 2011b; Tang, 2015). In line with Bransford’s (1993) observations about problem finding, the “reality” of the problem scenario must be one that learners can relate to and find compelling. Tang’s (2015) experience adapting a problem-based computer simulation, Making Change Happen™ The Network Inc., 1997, for use in Mainland China brought this point home. Although the focal problem—implementing new learning technology in a school system—hardly changed at all, almost everything about the “context for implementation” changed dramatically. This included the organizational structure, staff roles and responsibilities, cultural norms guiding the normative behavior of staff, and even the types of “activities” used by school leaders to foster the change. Her conclusions were particularly persuasive since she was able to compare the responses of Chinese principals who had used a translated but “nonadapted” version of the simulation with those of principals who had used the translated and culturally-adapted version.
PBL and learning technologies
PBL can be used as a pedagogical framework for various types of e-learning; conversely, learning technologies can be employed to increase the impact of PBL (Hallinger, 2005). In total, 10 sources explicitly examined how PBL could be blended with various forms of learning technology. These included computer simulation (Hallinger & Kantamara, 2001; Hallinger & McCary, 1990; Mann et al., 2011; Ng, 2001; Showanasai et al., 2013; Tang, 2015), delivery of PBL in online web-based environments (Cranston, 2002; Mann et al., 2011; Mascitti et al., 2011; Mayer et al., 2001; Tang, 2015), and exploration of video-enhanced presentation of PBL problems (Hallinger, 2005; Mann et al., 2011; Mayer et al., 2001).
Bransford (1993) offered a compelling case for how presentation of “the problem” in a PBL project can be enhanced through the use of video. He argued that conveying the problem via video rather than text provides learners with a richer more complex representation of the “problem context.” This challenges the learner to “find and interpret the problem” cues embedded in the video case. This type of “pattern recognition” distinguishes novice problem solvers from experts (Leithwood & Stager, 1989) and can be enhanced when PBL is blended with video presentation of the problem scenario (Bransford, 1993).
Both Mann et al. (2011) and Mayer et al. (2001) used PBL as a pedagogical framework for their development of “immersive online environments.” Mann and colleagues describe the learning environment as a “simulated or virtual school.” Learners
make decisions in a Web-enabled framework that facilitates the provision of individualized feedback tailored to the specific choices made by the learner, and that supports the collection of multiple metrics that relate to the performance of the learner and the learning framework itself. (Mann et al., 2011, p. 272)
Although data presented in the paper were largely anecdotal, they make a broad claim.
The outcomes from the first cohort of students to engage with this immersive, interactive, web-enabled computer simulation have provided ample evidence of the viability of this modality as a trigger for learning. Problem-based learning has played a part in developing educational leadership for some time (Bridges & Hallinger, 1991; Schechter, 2011), but the video simulation developed for Project ALL has set the direction for the next generation of problem-based learning. (Mann et al., 2010, p. 284)
Other scholars employing problem-based computer simulations have reported similarly positive experiences (Hallinger & Kantamara, 2001; Hallinger & McCary, 1990; Ng, 2001; Showanasai et al., 2013; Tang, 2015). Mixed methods data indicate high levels of learner engagement, a strong perception of learning that is connected to practice, and enhanced capacity for developing higher order thinking among learners through the interactive capacities offered by simulations (e.g., Hallinger et al., 2010; Hallinger & Kantamara, 2001; Hallinger & Lu, 2011a, 2011b; Mayer et al., 2001; Ng, 2001; Tang, 2015).
Ng’s (2001) research offers a particularly interesting offshoot within this line of research and practice. He redesigned and embedded an artificial intelligence module into an existing problem-based computer simulation and compared the results of the face-to-face and “self-tutoring” versions. The results suggested rich potential for the design of simulations that leaders can use to learn independently 24/7 at their convenience. More recent innovations that have made these problem-based simulations available online further increase that potential (e.g., Mann et al., 2011; Showanasai et al., 2013; Tang, 2015).
Although we remain disappointed by the absence of stronger quantitative and mixed methods evaluations of these technological adaptations of PBL in educational leadership (e.g., Hallinger et al., 2010; Hallinger & Lu, 2011a, 2011b; Showanasai et al., 2013; Tang, 2015), we concur that this line of curriculum and instructional design is worthy of additional empirical exploration. In this regard, we also wish to affirm the assertion made by scholars (Mann et al., 2011; Showanasai et al., 2013) that problem-based computer simulations offer a unique opportunity for conducting higher quality empirical research on PBL. The computer simulations, whether delivered online or face-to-face on individual computers, have the capacity both to “track” learner responses to dynamically changing problem contexts and to assess the “effectiveness” of the management strategies that they design and execute in the simulated school. Showanasai et al. (2013) offer examples of how data collected via an online simulation can be employed in different programs of research relevant to PBL.
Discussion
The low priority assigned to research on school leader preparation and development can be illustrated by reference to the first two editions of the Handbook of Research on Educational Administration (Boyan, 1988; Murphy & Louis, 1999). These compendiums of the field contained just a single chapter devoted to this subject (McCarthy, 1999). Although the descriptive literature on school leader preparation has proliferated in recent years (e.g., Jackson & Kelley, 2002; Lumby, Crow, & Pashiardis, 2009; McCarthy, 1999; Murphy, 2006; Orr & Orphanos, 2011; Walker et al., 2013; Young et al., 2009), empirical research remains notable by its scarcity (e.g., Heck, 2003; Leithwood, Riedlinger, Bauer, & Jantzi, 2003; Orr & Orphanos, 2011). Moreover, we were unable to locate any systematic reviews of research in this literature whatsoever. In this closing section of the article, we will discuss limitations of the review, our interpretation of the findings, and implications.
Limitations
Gough (2007) noted that a review of research is essentially a “study of the literature.” Therefore, the outcomes of a review of research depend in large part on the nature of available sources. The breadth, depth, and quality of the 73 scholarly papers analyzed in this review demonstrated the characteristics of an “immature literature” (Hallinger, 2014). Consequently, our review was aimed at “exploration” of the literature rather than “explanation” of findings. Thus, the main value of this review lies in summarizing where we stand today and charting future directions.
Another limitation of this review flows from our decision to include studies of both preservice preparation and in-service professional development. Theories of adult development indicate that the resources and needs of learners change over the course of an administrator’s career. Therefore, it is possible that PBL could be better suited or operate differently in preservice preparation and in-service professional development programs. Although neither our own experience nor the studies reviewed in this article support that proposition, this remains an open question.
Summary and Interpretation of the Findings
This review suggests a gradual but sustained global evolution in the use of PBL in school leadership preparation programs since its introduction at Stanford University in 1989. Despite a heavy concentration (77%) of sources focusing on the use of PBL in the United States, we also found substantial evidence of sustained use in Asia and Australia. We conclude that PBL has moved from an “innovation in school leader preparation” pioneered in the United States to an accepted, if not modal, approach to the education of school leaders in many parts of the world.
The review has documented several positive developments related to the use of PBL in school leader preparation and development.
Extensive development and field testing of an established model of PBL (Bridges & Hallinger, 1995) in both preparation and professional development programs
Affirmation that implementation requirements (e.g., faculty training, change management, curriculum models, assessment methods) identified in the broader PBL literature also apply in educational leadership programs
A reasonably broad base of implementation experience in a variety of cultural (e.g., United States, Australia, Hong Kong, Singapore, Malaysia, Thailand) and institutional (e.g., universities, school districts) contexts where program designers have adapted PBL to meet the needs of different learners
Accumulation of more than 50 PBL projects covering a wide range of “problem foci” relevant to the preparation and development of school leaders internationally
Multiple examples of potentially powerful hybrid models in which PBL has been combined with innovative learning technologies
Nonetheless, our analysis of this literature has also identified serious shortcomings. We noted, for example, a propensity for scholars to report on the first year of program or classroom implementation. Twenty-eight years has afforded ample time for longer term research and evaluation studies. Did the PBL classes and programs die out? Did instructors lose interest? If so, why? Or have faculty simply failed to conduct research on longer term efforts?
We observed that only 11 of the 73 sources included in the review were explicitly framed as original empirical research studies, and only three of these examined the outcomes of PBL. Moreover, there was a remarkable absence of quantitative and longitudinal studies. When compared with other professional fields where experimental, quasi-experimental, and longitudinal evaluation studies of PBL are common, research on PBL in educational leadership looks particularly weak. These features of the knowledge base could account for the relatively low level of penetration of research on PBL in core educational leadership journals (i.e., only 11 articles over a 28-year period). In sum, the scarcity of published empirical studies of PBL raises questions concerning the field’s commitment to generating an evidence base on the preparation of school leaders.
Implications
We suggest that the positive findings reported in this literature provide only the initial evidence needed to justify a broader based effort to implement PBL in school leadership preparation. When compared with other professional fields (e.g., medicine, engineering, nursing), the growth of PBL in educational leadership appears stunted. We expect that the scope of future developments will rest to a great degree upon the quality and scope of research that scholars conduct on the efficacy of PBL in strengthening the capacity of school leaders.
Fortunately, those who have pioneered the use of PBL in educational leadership have generated a useful base of experience as well as an abundance of PBL materials. Indeed, efforts should be made to develop a “global library” of PBL materials in order to provide easier access for program designers and instructors.
High-priority lines of research on the use of PBL in school leadership preparation should address gaps between the foci implied by our conceptual framework and the composition of the extant literature. First, we see a need for studies that apply cognitive, curriculum, instructional, and adult development theories toward understanding the learning processes and outcomes of PBL. Copland’s (2000) study stands out as a positive example in this regard.
As implied above, we also need longitudinal studies aimed at understanding the mature implementation of PBL. This research would not only document features of program design but also examine the impact on learner knowledge, skills, and attitudes. In a 6-year study of PBL implementation in an MBA program, Hallinger et al. (2010) demonstrated how institutional data can be employed to evaluate program effects on learner attitudes. With suitable planning, this type of research is replicable and can be complemented by useful qualitative data.
We also see a need for more research on the use of PBL in non-Western cultures (e.g., Ng, 2001; Tang, 2015). Initial skepticism over the receptivity of Asian instructors and students to PBL and other learner-directed method has not been borne out. Nonetheless, we encourage scholars to continue to study faculty and student responses to PBL outside of Western contexts and to complement past efforts with cross-cultural comparative research.
Space limitations preclude our describing suitable research designs for investigating these issues. However, scholars who take up these challenges will find ample examples in the broader literature on PBL (see Dolmans et al., 2005; Gijbels et al., 2005; Neville, 2009). Indeed, we reassert that the growth of PBL into a globally prominent model of medical education resulted in no small part from the abundance of empirical studies of learning and impact. Indeed, it is possible to track the growth of PBL in medical education via a series of systematic reviews of research and meta-analyses that appeared over the past 30 years (e.g., Albanese & Mitchell, 1993; Dolmans et al., 2005; Gijbels et al., 2005; Neville, 2009; Vernon & Blake, 1993; Walker & Leary, 2009).
What accounts for this contrast between the state of the art in the preparation of doctors and school leaders? Although answering this question goes beyond the scope of this article, we offer some initial thoughts. First, based on our reading of the literature in medical education, we find a high standard of research quality as evidenced by the scope of studies whose results could be integrated in meta-analytic reviews. Second, much of the medical education literature consists of funded research. Finally, medical education appears to operate with a more firmly rooted tradition of using empirical evaluations to assess the viability of innovations.
In contrast, our reading of the literature on PBL in the preparation of school leaders found that empirical research is scarce and funded research virtually nonexistent. Indeed, we recall that little, if any, of the funding provided by the Danforth Foundation to enhance innovation in school leadership preparation in the 1980s and 1990s was used for research and evaluation. Consequently, as Murphy (2006) observed, amid cyclical waves of critique and reform, our field has failed to produce solid evidence on “what works” when it comes to the preparation of school leaders.
With these points in mind, we suggest that achieving progress in this domain implies several requirements. First, funded empirical research on the scale of the Wallace Foundation’s effort to study “successful school leadership” is needed if we wish to stimulate development of a sound knowledge base. Second, as in the Wallace program, we need scholars capable of modeling the use of strong research designs that can be replicated by others. Third, our top journals and research handbooks should hold scholars who write about administrator preparation to a higher standard.
These initial suggestions represent an insufficient response to an important question. Nonetheless, as scholars who have toiled for several decades on the development and implementation of PBL, we see great potential value in the empirical evaluation of our collective efforts. Thus, we look forward to the day when the preparation and development of school leaders is deemed worthy of the same level of support and evaluation afforded to the education of doctors.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.