Abstract
This is a conceptual review of the literature variously referred to as faculty development, educational development, instructional development, and academic development in higher education. Previous empirical reviews covering more than 30 years of published literature could draw only tentative and weak conclusions about the effectiveness of educational development practices. The authors used different questions that queried the nature of educational development practice and the thinking underlying practice. Their conceptual review yielded a framework with six foci of practice (skill, method, reflection, disciplinary, institutional, and action research or inquiry) that was drawn from an analysis of the design elements of the educational development practices in the research they reviewed and from an analysis of the conceptual, theoretical, and empirical literature cited by those articles. This six-cluster framework provides a new way of thinking about the design of practice and a more meaningful basis for investigating the effectiveness of educational development practice.
Some long-standing assumptions in higher education are that academics are not adequately prepared for their teaching role, have unsophisticated conceptions of teaching and learning, and have little knowledge of effective teaching practices, both in general and in their own specific discipline (Evers & Hall, 2009). These assumptions have led universities to pursue a number of ways to develop teaching practice. At the same time, a literature variously referred to as faculty development, educational development, instructional development, and academic development (Gosling, 2009) has emerged in support of research and practice, and this literature is the focus of our review. We use the term educational development to describe actions, planned and undertaken by faculty members themselves or by others working with faculty, aimed at enhancing teaching. The sources of this literature are diverse and include journals that are (a) discipline specific (e.g., European Journal of Engineering Education), (b) about higher education in general (e.g., Research in Higher Education), (c) dedicated specifically to teaching in higher education (e.g., Teaching in Higher Education), and (d) focused on the broader academic role (including teaching) in higher education (e.g., International Journal for Academic Development).
Most would agree that educational development in higher education remains a developing field—one that includes practitioners (educational development staff), faculty, researchers, and those who assume all three roles simultaneously to focus on the improvement of teaching and learning in higher education. One of the ways a developing field is able to further define itself and to enhance practice and further research is through the periodic review, analysis, and synthesis of the published literature. Three systematic empirical reviews of the educational development literature have been conducted, all focused on the question of effectiveness of educational development practices (Levinson-Rose & Menges, 1981; Steinert et al., 2006; Stes, Min-Leliveld, Gijbels, & Van Petegem, 2010). Despite the review of more than 30 years of literature represented by these reviews, the good work reflected in the well-designed and conducted review processes, and the useful attention to levels of learning and assessment outcomes, the answer to the question posed—What are the features of educational development that make it effective?—has been elusive. Authors of these reviews were able to draw only tentative and weak conclusions about effectiveness. All three reviews call for not only more rigorous research designs but also more qualitative research, a better theoretical and conceptual grounding of educational development practice, and a more detailed description of practice, so that each new study can build more explicitly on previous ones. The paucity of findings and the strikingly similar recommendations of all three reviews led us to ponder whether they were asking the right question if their intention was to inform practice and further research.
Kennedy (2007) distinguished empirical reviews, often based on cause–effect questions, from conceptual reviews, which she suggested “share an interest in gaining new insights into an issue” (p. 139). Essentially, the two types of reviews ask different questions. A conceptual review would not ask, for example, What educational development practices have the greatest impact? but would instead ask, How are educational development practices designed? and What is the thinking underlying the design of practice? Kennedy argued that the term systematic review is usually used to refer to empirical reviews but that both empirical reviews and conceptual reviews may adhere to the criteria for a systematic review: defined question(s) to drive the review, ensuring that an attempt has been made to identify all literature relevant to the questions posed, and employment of specific inclusion and exclusion criteria in defining the body of literature to be reviewed.
This article describes a systematically conducted conceptual review. The questions that drove the review were How are educational development practices designed? and What is the thinking underpinning the design of educational development practice? We had two specific purposes for conducting this review. Our first purpose was to deepen the understanding of current practice by describing and characterizing educational development practice in more detail than others have done. We began by documenting the core characteristics of each educational development initiative described in the articles we reviewed. By core characteristics, we mean the intention or goal of an initiative, the processes and activities planned to realize the intention or goal, and the evidence collected to demonstrate success in achieving the intention or goal. A framework with six clusters of practice emerged from this process: skill, method, reflection, disciplinary, institutional, and action research or inquiry. Our second and related purpose was to explore the thinking underpinning the design of practice. Evidence of this was drawn from an analysis of the core characteristics of the educational development practice and from an analysis of the conceptual, theoretical, and empirical literature cited by the articles in our review.
We begin by discussing how this conceptual review builds on and adds to previous empirical reviews. We then present the design and procedures of our review, followed by a detailed description and analysis of the six-cluster framework. Our conclusion considers the implications of this study for educational development practice, further research, and the continuing definition of the field of educational development.
Previous Empirical Reviews
Previous empirical reviews grouped studies by format (e.g., workshop, one-to-one consultations), by level of learning outcome (e.g., self-report of attitude, observed behavior), and by individual variables (e.g., duration of the activity) in relationship to reported measures of impact. We are aware of three systematic and empirical reviews of the literature; 1 they are briefly described here to provide a sense of the work we are building on.
The earliest systematic review (Levinson-Rose & Menges, 1981) was based on 71 reports published between the mid-1960s and 1980 and had the expressed purpose of drawing conclusions for both “researchers and those who design and implement instructional improvement programs” (p. 416). Articles were grouped by the format of the initiative (e.g., workshops, one-to-one consultations) and discussed according to three levels of assessment of impact: teacher skill based on observation, student attitude from self-report, and student learning from tests or observer reports. The researchers evaluated the research design of each study and applied a rating (high, fair, or low) to indicate how much confidence should be placed in its findings; 78% of the articles reviewed supported the general success of the initiative described, although the level of support was reduced to 62% when the confidence ratings were considered. The authors noted that although workshops and seminars were the most common instructional development intervention (short, “one-shot” workshops were most numerous), they were also the least evaluated and, in their opinion, the least likely to “produce lasting changes in teaching behaviour or lasting impact on students” (Levinson-Rose & Menges, 1981, p. 419).
Steinert et al. (2006) conducted another empirical review 25 years later based on 53 identified articles published during the period 1980 and 2002 and specifically focused on educational development in the medical sciences. Like Levinson-Rose and Menges (1981), these researchers also hoped to draw conclusions about “features of faculty development that make it effective” and “the impact on knowledge, attitudes and skills of teachers” (Steinert et al., 2006, p. 499). Articles were coded by format of the initiative (e.g., workshop, seminar), level of learning or assessment outcome adapted from Kirkpatrick’s (1994) four levels of assessment outcomes and similar to the levels used by Levinson-Rose and Menges (1981), and research design type. Like the previous review, they developed a 5-point confidence rating scale based on quality of the research design; the mean confidence rating across the 53 articles they reviewed was 3.14 out of 5. The discussion of their findings is organized in two ways. First, they provide an overview of the articles in their review, providing frequency counts for several aspects including format, duration, and level of outcome assessed. Second, they discuss specific articles organized first by format (e.g., workshop, seminar) and within that the levels of learning or assessment outcome. The authors admitted that neither their review nor the work of others to date had the evidence to “tease apart features of faculty development that make it effective” (Steinert et al., 2006, p. 509), but they did provide some “preliminary conclusions” (p. 509). These include the value of feedback, the importance of peers, and the use of multiple instructional methods.
The most recent empirical review (Stes et al., 2010) was based on 36 identified articles; the review was not limited in terms of time (earliest article was 1977 and latest 2007) or source of publication, and the authors intentionally looked beyond peer-reviewed journals. Articles were coded by level of learning or assessment outcome and by format of the initiative (workshop, one-to-one consultation), consistent with Steinert et al.’s (2006) review, and by research design type (quantitative, qualitative, and mixed-methods). They also investigated the relationship of several individual variables (duration, format, target group) to impact levels. Impact level was assigned as follows: 0 = no indication of impact, + = indication of impact; = partial indication of impact; ? = unclear if there was any impact. They found weak evidence that interventions of longer duration were reported to have more positive learning outcomes at the level of the faculty participant and that course-length interventions showed more positive outcomes at the level of the student.
Little changed in the conclusions or recommendations made by authors from the first review (Levinson-Rose & Menges, 1981) compared to the most recent one (Stes et al., 2010). All three reviews were critical of the quality of the studies in terms of their theoretical grounding and clarity of goals. Steinert et al. (2006) reported that 57% of the articles in their sample of articles from educational development in the medical sciences cited a theoretical or conceptual framework, drawn primarily from adult learning, instructional design, or reflective practice. Still, they called for more explicit attention to this as well as to acknowledging the importance of context.
The authors of all three reviews noted the difficulties in drawing any meaningful findings because of the limited number of studies meeting their inclusion criteria and rigor in research design. All three called for the use of qualitative research designs that draw on more diverse forms of evidence (e.g., narratives, critical incident interviews, journals, portfolios). However, most articles matching these descriptions would likely not have met the selection criteria of these reviews and so would have been excluded. We return to this point later.
The particular features used to group educational development initiatives in all three of the previous reviews were, we argue, problematic and may provide at least a partial explanation for their lack of findings. All three reviews grouped educational development initiatives by format (e.g., workshop, one-to-one consultations). As best we can determine, authors of the reviews simply used the label given in the article, not defining it further. The term workshop, based on our reading of the literature, can refer to activities that vary widely in terms of purpose, content, duration, and quality. The format might be thought of as the outer shell, not reflecting the variation in actual practice contained inside. The feature of duration is also poorly defined. Two of the three reviews collected information about the duration of the initiative, but only Stes et al. (2010) used this information in a subsequent analysis of impact. We know from the articles we reviewed that information about duration, when provided, is variously described as number of contact hours, numbers of workshops, or the length of time over which the initiative spanned (number of days, months, or years).
The clear difficulty in comparing these diverse (and often inexact) measures may be what prompted Stes et al. (2010) to use a dichotomous classification for duration—one-time events versus longer initiatives. Grouping educational development initiatives using vague descriptors is, in our view, only part of the problem. The more important issue is the consideration of individual features separate from the overall design of an initiative. In the case of duration, it begs the questions, longer than what or longer to achieve what? Similarly, grouping initiatives by kind and level of assessment without consideration of the broader design ignores the critical link among conceptual or theoretical grounding, core characteristics of the design, and learning. Is this a reasonable outcome given the design of the initiative? is a more meaningful question to ask. The authors of the most recent review recognized this weakness:
Future research not focusing on one specific educational feature of instructional development (e.g., format, duration), but considering the core characteristics of instructional development initiatives (e.g., the theoretical foundation, goals and content) in their internal connection would be even more worthwhile. (Stes et al., 2010, p. 47)
This is where our conceptual review picks up. A conceptual review, with the stated interest of “gaining new insights into an issue” (Kennedy, 2007, p. 139), seemed to us an appropriate way to proceed. We began by identifying the literature that describes and analyzes initiatives that are planned and undertaken by faculty members themselves or by others working with them and that are aimed at enhancing teaching.
The Review Process
Review Team
Seven different individuals worked on the review: the two authors and five graduate students at Simon Fraser University in Canada. 2 At least three of these individuals were working on the team at any one time. Both authors were actively involved throughout the review process.
Inclusion and Exclusion Criteria
We used inclusion and exclusion criteria that moved from broad to more specific. We began by searching databases with these general keywords: instructional development, faculty development, professional development, educational development, academic development, teaching development, higher education, and post-secondary education. We then used the following three more specific criteria.
Focus of activities, initiatives, programs
We included all types of formal and informal teaching development initiatives, including those initiated by both centralized teaching support centers or other formal administrative groups (e.g., workshops, seminars, courses) and those initiated by individual academics or groups of academics (e.g., book groups, peer mentoring, learning communities, classroom research studies). To be included, initiatives had to be fully articulated in terms of the core characteristics of the design, including how effectiveness was determined.
Assessment of impact and evaluation of effectiveness
We began by including only reports of initiatives that went beyond satisfaction ratings, in keeping with the critiques of the literature articulated by two of the previous review teams. We adjusted this, however, to more fully capture the literature and included a few articles that provided a detailed account of procedures or activities that had been put in place to determine effectiveness but did not report any findings.
Institutional context
Our focus in this review is the university context—the context in which we work as teachers, researchers, and educational developers. One motivator to conduct this review was to situate our own educational development practice in the broader literature. Specifically, we included initiatives that were designed for two types of universities: medical or doctoral, and comprehensive, where faculty generally have joint roles as both teacher and researcher. These are labels commonly used in Canada. Medical or doctoral universities offer a broad range of Ph.D. programs and research; all institutions in this category have medical schools. Comprehensive universities have a significant degree of research activity and a wide range of programs at the undergraduate and graduate levels, including professional degrees. 3
Search Process
We followed multiple steps in identifying articles for review.
We searched three databases (ERIC, Google Scholar, PsycINFO) for literature published in peer review journals between 1995 and 2008 using the keywords listed above.
We identified conference papers, proceedings, and reports using the same keywords and the same databases, but only for 1997 to 2004. We wanted to include this literature because we knew it would be a good source, but we did not have the resources to consider the same span of years as for the peer-reviewed journals. Consequently, we selected a span of time that would provide a reasonable sampling of this literature. Based on our examination of our sampling of this literature, we were satisfied that it did not represent a literature with substantially different characteristics than that found in the peer-reviewed journals we reviewed.
We read the abstracts of the 3,048 peer-reviewed journal articles (in 104 different journals), conference papers, proceedings, and reports identified in the first two steps. Each abstract was read by a team member to determine if the article described an educational development initiative.
We conducted a manual search (reading article abstracts) of 11 journals known to our team as sources of articles concerning educational development. These 11 journals were the Journal of Higher Education, Studies in Higher Education, Teaching in Higher Education, Higher Education, Higher Education Quarterly, Adult Education Quarterly, New Directions for Teaching and Learning, the International Journal for Academic Development, Higher Education Research and Development, Active Learning in Higher Education, and Innovative Higher Education.
The database search and manual search left us with 428 articles (excluding duplicates) to read fully. These were read in the context of our three specific inclusion and exclusion criteria described above. Every article citation was entered into a reference management system (RefWorks), along with the names of the two readers; article copies were made for the readers and retained in our files for cross-checking.
The review and screening proceeded as follows:
At least two readers read each article in full and applied our specific criteria to decide to include or exclude that article. A third reader was used when there was disagreement, and that article was also discussed at our weekly meeting.
One reader for each article noted this specific information in an Excel table: first author, year of publication, reader name, description of the initiative, focus or goal of the initiative, duration of the initiative, theoretical or conceptual underpinnings or rationale for the design, evidence of impact or effectiveness, and philosophies, beliefs, and values about what constitutes effective teaching and/or learning in higher education.
We documented in RefWorks and in the Excel table the reasons for excluding any article.
Our final database of articles totaled 137. Of the 137 articles included in our review, 64 (47%) were published in the 11 journals listed above (some of these articles were discovered in our manual search and some in our electronic searches) and, of these, 26 (19%) were published in the International Journal for Academic Development. An obvious omission is the lack of articles published in languages other than English, although we did include more articles written by authors residing in countries other than the United States (20 other countries) than did previous reviews.
Emergence of a Conceptual Framework
We noticed that the initiatives we were reading about seemed to cluster based on the purpose or focus of the initiative, the core characteristics of the design, and the literature cited. In one of our weekly meetings, we began to sketch out these emerging clusters on a large piece of paper. We then moved to develop a coding sheet that was revised and refined as we progressed with the review process. For each article, coding sheets were completed by two readers and compared. If there was not consensus, a third reader read and coded the article and it was discussed at a full team meeting. The coding scheme and revisions were documented in a codebook. Once the clusters and the associated elements appeared stable (i.e., this iterative cluster-defining process was not changing our cluster definitions, and no new clusters were being identified), all articles were reread and recoded by the two authors. Essentially, we followed an emergent coding process as recommended by Huberman and Miles (2002). We were able to agree on a good fit with the focus of a particular cluster for every article; in other words, the educational development initiative described in each article adhered to all of the elements in one particular cluster. The last version of our coding framework included the following six clusters (see Table 1).
The skill focus cluster includes 14 articles that focus on the acquisition or enhancement of observable teaching skills and techniques (voice projection, presentation skills, discussion facilitation skills, etc.).
The method focus cluster includes 33 articles that focus on mastery of a particular teaching method (e.g., problem-based learning).
The reflection focus cluster includes 30 articles that focus on change in individual teacher conceptions of teaching and learning.
The institutional focus cluster includes 37 articles that focus on coordinated institutional plans to support teaching improvement.
The disciplinary focus cluster includes 4 articles that focus on disciplinary understanding to develop pedagogical knowledge.
The action research or inquiry focus cluster includes 19 articles that focus on individuals or groups of faculty investigating teaching and learning questions of interest to them.
Educational development clusters and associated elements of thinking and design
The next section describes these six clusters of practice in detail. It is important to note that it is not our intention to value one cluster over another; we discuss the rationale for this further in the concluding section. We attend, in the following discussion, to the questions motivating this review—How are educational development practices designed? and What is the thinking underpinning the design of educational development practice?—by describing the core characteristics of the initiatives in each cluster (intention or goal of an initiative, the processes and activities planned to realize the intention or goal, and the evidence collected to demonstrate success in achieving the intention or goal) and their internal consistency. To further address the conceptual consistency of the initiatives in each cluster and the thinking underpinning design, we discuss the literature most cited in the articles in each cluster and, to the extent possible given the information provided in each article, how this literature was drawn on in the design of each initiative. This citation information resulted from a separate analysis in which we identified all of the (first) authors who were cited at least three times in three separate articles in each cluster (the complete analysis is in Tables 2–7 in Appendix B in the online journal).
Cluster 1: Skill Focus
The 14 articles that were coded as having a skill focus describe initiatives that seek to improve teaching through the enhancement of observable teaching skills and techniques, primarily as demonstrated in a classroom setting. Initiatives took the form of pretest-intervention-posttest (sometimes administered more than once over time) or intervention-posttest. Specific skills or techniques to be learned or improved were identified by instructors themselves and/or by students (e.g., through course evaluations or interviews) and/or by educational developers. Targeted skills and techniques, as described in these articles, included questioning techniques, interactive techniques, presentation skills, consultation skills, clinical teaching skills, and lesson planning. Interventions took the form of one-to-one consultations or targeted workshops led by educational developers. Assessment of the effectiveness of the intervention was accomplished through student course ratings, structured student interviews, self-evaluation of performance using video classroom observations, self-report of teaching practice, self-perceptions of performance, and classroom or clinical observations by others.
Many of the articles in this cluster cited the work of authors who have contributed to the well-developed literature on faculty consultation using pre–post student course ratings (e.g., H. W. Marsh, P. A. Cohen) as the basis or rationale for the design of the initiative. 4 This is a largely atheoretical literature commonly employing quasiexperimental designs to determine change in pre–post measures. Another source regularly drawn on in the design of these initiatives was the literature discussing evidence-based teaching techniques (e.g., W. McKeachie, A. Saroyan) and the assessment of clinical teaching skills (e.g., M. Hewson). In total, there were 19 authors cited as the first author three or more times across the 14 skill focus articles we reviewed (see Table 2 in Appendix B online).
Example from the skill focus cluster
Nasmith and Steinert (2001) investigated the effectiveness of a workshop that allowed faculty to explore interactive techniques they can incorporate into their lectures. 5 The workshop design drew on the medical education clinical teaching strategies literature (e.g., Skeff, Stratos, Berman, & Bergen, 1992) and that dealing with evidence-based teaching techniques (e.g., McKeachie, 1994). The experimental group and comparison group were compared on an immediate postworkshop questionnaire, a six-month postworkshop questionnaire, and independently rated videotapes of selected faculty from both groups six months after the workshop. All measures showed a change in favor of the increased use of interactive techniques among the experimental group.
All of the initiatives included in this cluster were designed as interventions either to address a documented need of a particular instructor (individual consultation) or to teach a group of faculty how to incorporate a particular technique into their teaching practice. The emphasis was on an observable outcome. Quasi-experimental designs were used incorporating pre–post or posttest-only measures and sometimes incorporating longitudinal measures and control or comparison groups.
Cluster 2: Method Focus
The focus of the 33 articles in this cluster is a teaching method that is based on a particular view of learning and encompasses a set of teaching and learning strategies that, taken together, support the desired learning (e.g., problem-based learning; PBL). Thus, the purpose of these initiatives is not only learning to use the teaching method but also coming to understand the particular view of learning that underpins it. This is what primarily differentiates this cluster from the skill focus cluster, where the concentration is on individual, observable skills and techniques. The methods evident in these articles were originally developed based on an instructional need (e.g., PBL in medical education, case-based learning in business education) or, in a few cases, from a particular theoretical stance (e.g., constructivist course design, self-directed learning).
Of the 33 articles in this cluster, 14 described initiatives addressing online learning course design (8), PBL (3), or case-based learning (3). Other teaching methods included guided independent learning, self-directed learning, collaborative or cooperative learning, sustainable teaching and learning strategies, and course design for diversity and social justice. In many cases, the design of the training modeled the method being taught—for example, faculty learning to teach with a case-based method were taught using cases as their students would be. Assessments of effectiveness were developed to gather evidence of mastery of the method during the training as well as use of it in subsequent teaching practice. Self-report measures were most common (questionnaires, written reflections, interviews, focus groups, implementation logs), often administered multiple times after the completion of the training. Classroom observations with feedback were also common, followed by student evaluation of course materials developed or teaching practices specific to the training.
The authors cited most frequently in the method focus cluster fell loosely into four groups; the first two groups are most tightly linked to how an initiative was designed or the rationale for choosing a particular method. The first group contained articles that described a particular method of teaching, for example, technology integration (e.g., L. Cuban) and PBL in medical education (e.g., D. Irby). A second group was literature that provided a theoretical rationale for the selected teaching method: a social practice perspective (J. Brown & P. Duguid), constructivist design (D. Jonassen), adult learning principles (M. Knowles), types of faculty knowledge (L. Shulman), and reflective practice (D. Schön). The third group comprised citations to broad discussion articles about the practice of educational development or future directions (e.g., C. Bland). Finally, the fourth group of citations addressed the assessment of faculty learning (e.g., J. Biggs). In total, 15 authors were cited at least three times in the 33 articles we reviewed in the method focus cluster (see Table 3 in Appendix B online).
Example from the method focus cluster
Eisen and Barlett (2006), both faculty members, one in biology and the other in anthropology, and both with responsibilities for educational development in their academic units, provided a description of the Piedmont Project on environmental sustainability. Principles underlying the design of the initiative were drawn from Dewey’s notions of inquiry learning as well as from research in environmental education. Emphasis was on (a) interdisciplinary cooperation, (b) shifting from teacher as expert to teacher as facilitator of learning, (c) combining research skills with ethical reflection, and (d) personal responsibility toward environmental sustainability. Faculty participants identified a new course to develop or an existing course to revise and attended a two-day course development workshop led by faculty participants from the previous year. There was an end-of-summer field trip to local sustainability-relevant sites and a follow-up dinner a year later to discuss their experiences with their new or revised courses and the impact this project had had on their professional perspectives and teaching methods. Effectiveness was assessed through three surveys conducted a few days after the summer workshop, one year later, and four years later. Course materials and written reflection statements were collected and analyzed. Faculty reports and course materials evidenced change in teaching approaches (especially including field trips and real-world problem inquiry).
All of the initiatives in this cluster were designed with the intention of teaching faculty to use a particular method of teaching and to come to an understanding of the type of learning supported by that method. Generally, as part of the design of the initiative, faculty experienced the method as they learned to use it. Measures of effectiveness focused on the use of the method both in the training sessions and later in individual teaching practice. Thus, self-report measures of teaching practice were most evident, but there was also ample use of outside observers and targeted student feedback.
Cluster 3: Institution Focus
Each of the 37 initiatives coded as having an institutional focus outlined a coordinated plan for academic development at the institutional level, part or all of which had to do with teaching development. These initiatives commonly placed an emphasis on strategic planning and human resource development. Initiatives were “top-down,” in response to an institutional or national agenda (e.g., technology integration) and sometimes involved multiple universities. The assumption of the institution was that the initiative (e.g., technology integration) was generally beneficial to all instructors in all academic contexts. The emphasis of the development efforts was quite varied and included technology integration, developing a network of teaching scholars across the university who worked with colleagues to improve teaching, widespread adoption and resource management to support a particular approach (e.g., PBL), systematic evaluation of teaching across the institution, and specific curriculum addition or refocus (e.g., professionalism, technology literacy, action research).
Although the emphasis of the initiatives in this cluster may resemble some in the method focus cluster (e.g., PBL, technology integration), articles in this cluster have a stated goal to have an impact across the entire institution or academic unit (i.e., faculty or school). Thus, the focus of implementation is ensuring the successful diffusion of the initiative by building the structures and processes to accomplish this. Simpler to more elaborate structures and processes were used, including combinations of the following: departmental and cross-departmental development teams, curriculum redesign groups, mentoring processes, highlighting and modeling of existing practice consistent with the development initiative, project grants, and focused workshops. Assessment of effectiveness sought to show evidence of successful diffusion rather than assessing an individual instructor’s teaching. Thus, common broad indicators of successful diffusion were activity reports from various levels of the institution (faculty, department, program) about numbers of faculty participating, degree of uptake, and challenges met and overcome. More focused assessment considered the products of the diffusion process, such as redesigned courses, instructor portfolios, and targeted student feedback.
The cited literature in the institutional focus cluster moved beyond a focus on the individual faculty member to a broader organizational perspective on teaching and on educational development. The emphasis on strategic development and human resource development evident in many of the initiatives in this cluster was a reflection of the guidelines provided in the organizational change literature addressing institutional-level rethinking of teaching and learning (e.g., D. Laurillard; P. Ramsden; P. Trowler) and of educational development in general (e.g., G. Webb; G. Gibbs) and as specific to the health sciences (e.g., C. Bland; Y. Steinert). As with the method focus cluster, literature particular to the initiative being diffused across the institution was cited, for example, PBL in medical education (e.g., H. Barrows) and the design of technology integration (e.g., D. Jonassen). In total, 19 authors were cited as first author three or more times across the 37 institutional focus articles we reviewed (see Table 4 in Appendix B online).
Example from the institution focus cluster
Stigmar (2008) described an initiative (Educational Action Program) with the stated intention of making the “transition from conveying to creating knowledge” (p. 107). A continuous and structured process of dialogue in all academic units of the university was put in place to implement the stated intention of the Educational Action Program in a way that was meaningful to each unit. This initiative, like those described in other articles in this characterization, was a top-down response to a specific institutional mandate. Strategic planning led to work at three different levels: institutional (linking the program to the development plan for the whole university), educational development unit (staff of this unit work as change agents), and departmental (required to adapt the program to its activities by creating program action plans). The primary source of literature drawn on addressed organizational change through the scholarship of teaching and learning (e.g., Lueddeke, 2003). Evaluation of the initiative was conducted at the department level, using department reports of progress toward institutional goals, interviews with department representatives, and analysis of other relevant documents. Seminars and other activities to gather input on the process were used as safeguards against faculty perceiving this as only a top-down initiative. Departments were also encouraged to adapt the program to fit their contexts. The authors credit these activities and the general way the initiative was implemented for success in realizing the institutional goals across the university.
All of the initiatives in the institutional focus cluster address cross-institutional change around some aspect of teaching and learning. Thus, the design of these initiatives emphasizes the development of structures and processes at various levels of the institution to support the diffusion of the initiative with the assumption that it will become embedded in everyday practice. The assessment of successful diffusion focuses on the adequacy of these structures and processes to support the initiative and provide evidence of uptake.
Cluster 4: Reflection Focus
The focus of the 29 articles coded with a reflection focus is to engage faculty members in a process of reflection (individually and/or collaboratively) with the purpose of changing or clarifying their conceptions of teaching and learning and linking this to change in their teaching practice (McAlpine & Weston, 2000). The intention here is markedly different from the previous three clusters that could be broadly described as placing value on the outcome: acquisition or enhancement of specific skills and techniques in the skills focus cluster, demonstration and use of a particular method in the method focus cluster, and diffusion of an approach or perspective across the institution in the institution focus cluster. In this cluster, the reflection focus cluster, the value is placed on the process of reflection as learning.
Some articles in this cluster explicitly argued that a transformation in thinking or in conceptions of teaching and learning is a necessary pre- or corequisite step to changes in teaching practice. Focusing only on teaching skills or techniques, or even methods, as in the first two clusters, is seen as insufficient. In fact, many of the reflection-focused articles reported programs that were designed in reaction to workshops on teaching skills, techniques, and methods. The authors of one of the articles in this cluster, Hubball, Collins, and Pratt (2005), provided a clear example of this thinking in their explanation of the intention of the program they have developed at a university in Canada “to promote reflection defined as thoughtful consideration and questioning of what we do, what works and what doesn’t work, and what premises and rationales underlie our teaching and that of others” (p. 60).
Faculty who participated in these initiatives engaged in reflection in a number of ways. It could be prompted by reflecting on one’s conceptions and practices in relationship to a personal goal or ideal (reflect and discuss), to a colleague’s practice (observe, discuss, and reflect), to representations in the literature (read, discuss, and reflect), and/or to newly developed knowledge (i.e., read or learn, reflect, discuss). For example, an individual faculty member who desired to improve her interactions with students asked a colleague to review and discuss her written reflections and those of her students after each class session (Stewart & McCormack, 1997). Pairs of physicians involved in clinical coteaching followed a process of observing each other’s teaching, debriefing this, and planning the next session together (Orlander, Gupta, Fincke, Manning, & Hershman, 2000). A several-month-long course focused faculty on comparing their own practice to theory-based practices in the literature, ending with a provocative interview challenging them to integrate theory and their teaching practices (Halliday & Soden, 1998).
Although some of the tools used to determine success of these initiatives paralleled those evident in the first three clusters (questionnaires and surveys, classroom observations), others involved the analysis of the same activities used to promote reflection: personal journals, instructor and student reflections, focus groups, teaching philosophies, critical classroom incident assessments, teaching portfolios, projects.
The most cited literature in this cluster was decidedly more theoretical—not surprising given that reflection is a theoretical construct and is not directly observable in the same ways as the intended outcomes of initiatives in the skill and method focus clusters. Authors who wrote specifically about the process of reflection (e.g., D. Schön; J. Mezirow) and those who designed initiatives using reflection (e.g., P. Cranton; C. Kreber) were cited frequently. In addition, authors who wrote about conceptions of teaching and learning held by faculty and the relationship to approaches to teaching and student learning were cited consistently (e.g., P. Ramsden; A. Ho; K. Trigwell & M. Prosser). The design of initiatives in this cluster often worked to “ground” the reflection process in the individual faculty member’s teaching practice, citing literature that addressed the importance of disciplinary identification (e.g., R. Barnett), scholarly discussion (e.g., S. Rowland), adult learning principles (e.g., S. Brookfield), and embeddedness in the academic workplace (e.g., D. Boud). In total, 38 authors were cited as first author three or more times across the 30 reflection focus articles we reviewed (see Table 5 in Appendix B online).
Example from the reflection focus cluster
Sandretto, Kane, and Heath (2002) drew on the work of Argyris and Schön (1974) to design a program with the assumption that once instructors were aware of discrepancies between what they said they did (espoused theories) and what they actually did (theories in action), they would seek to address them. They also drew on the adult education literature (e.g., Brookfield, 1990) in their design. The authors began by identifying two groups: experienced excellent teachers in the sciences and early-career academics. The program consisted of 10 weekly 2-hour sessions. New faculty engaged in several activities: discussions (with experienced faculty), readings, reflective journals, writing of personal histories relevant to teaching, repertory grid interviews, written reflections based on viewing videotaped classroom observations of experienced teachers with accompanying explanations of thinking and practice, watching and reflecting on their own teaching, and ultimately articulating a teaching philosophy. An analysis of the resulting artifacts showed an articulation of espoused theories and theories in action and a movement to bring these together.
All of the initiatives in this cluster value reflection as learning, and, therefore, the activities faculty engage in are meant to support the process of reflection on their conceptions or understandings of teaching and learning and their teaching practice. Some initiatives go beyond this and specifically link or integrate what is learned through reflection to changes in teaching practice (e.g., teaching projects). For the first time, in this cluster we see a significant number of initiatives that are wholly designed by faculty to improve their own teaching, whereas in several others educational developers act as facilitators of an essentially faculty-driven process. In this cluster, documenting the reflection process and its link to conceptions of teaching and learning and to teaching practice was often accomplished through analyzing the same tools that supported reflection (e.g., journals, personal histories, teaching philosophies).
Cluster 5: Discipline Focus
Like many of the articles included in the reflection focus cluster, the four articles in the discipline focus cluster also noted the insufficiency of teaching skills or techniques and methods, but for a different reason. The thinking here is that skills or techniques and methods, if they are to be meaningful, must be understood through the lens of an instructor’s discipline (Davidson, 2004; Healey & Jenkins, 2003; Neumann, 2001; Neumann, Parry, & Becher, 2002). The stated assumption is that the structure of knowledge within a discipline should in good part determine course designs and teaching methods, and therefore teaching improvement requires in-depth knowledge of the discipline. Also assumed is that academics identify best with their own disciplinary culture, knowledge, and practices and that efforts for teaching improvement are best approached from this perspective. It is important to note that articles that simply drew their participants from one discipline for convenience or because of who initiated the program were not included in this cluster; the element of disciplinary understanding linked to teaching and student learning had to be explicitly stated. Characteristic of the stance taken in these four articles was the sentiment expressed by Mathias (2005):
A more fundamental issue is the extent to which the responsibility for the development of new university teachers has been gradually removed from the traditional academic disciplinary communities of practice and placed in the hands of education specialists, and whether this has undermined the ownership and commitment academic departments should have for the development of the teaching function within the context of disciplinary cultures and practices. (p. 97)
The designs of initiatives in this cluster were largely grounded in discussion about teaching experiences and/or educational readings. Designs supported developing teaching projects, collegial mentoring, and peer collaboration. Assessment of impact was accomplished through an analysis of teaching projects and self-reports of thinking and change in teaching practices. Given that there were only four articles in this cluster, we were not able to identify common citations meeting our criteria of three common citations in at least three separate articles in the cluster.
Example from the discipline focus cluster
Rowland and Barton (1994) and Rowland (1999) designed an initiative centered on scholarly critique, suggesting that critique is fundamental to academic discussion and critical to fostering understanding because it allows different disciplinary values or underlying theories to emerge (citing the work of Carl Rogers and Habermas and their work regarding the democracy of true discourse). Participants engaged in critique of readings, of each other’s ideas and perspectives, and of their reflections on their own teaching. Between weekly sessions lasting over the course of two years, participants read texts related to teaching and learning from the field of education as well as from other fields and were encouraged to “make observations and interpretations of their own teaching, try out different strategies in light of these and develop their own ongoing [teaching] projects” (Rowland & Barton, 1994, p. 369). The authors argued that the educational development program was built to “raise fundamental questions concerning pedagogy—the relationships between teacher, student and subject matter—within the particular social [disciplinary] context, rather than merely focus on the ‘how to do it’ of teaching skills” (Rowland & Barton, 1994, p. 369).
All four articles in this cluster took as a starting point that the structure of knowledge varies between disciplines and that faculty identify strongly with their discipline, both as a teacher and as a researcher. Thus, initiatives were designed to use scholarly processes to prompt thinking about teaching and learning from the perspective of the discipline. Activities challenged faculty to uncover their understandings about knowledge development within the discipline and to link this to their teaching practice.
Cluster 6: Action Research or Inquiry Focus
There were 19 articles in this cluster, all with the underlying rationale that teaching improvement is fostered by individual or group inquiry or research into topics or questions of interest related to teaching and learning. Like the two previous clusters where the value is placed on process as learning—in the reflection focus cluster, the reflective process, and in the discipline focus cluster, the process of scholarly discussion and critique—value is placed in this cluster on the process of inquiry as learning (Cox, 2001; Kreber, 2005; Kreber & Cranton, 2000; Richlin & Cox, 2004; Trigwell & Shale, 2004). Initiatives ranged from those investigating individual teaching practice to group inquiry motivated by the desire to explore questions of mutual interest. For example, in one initiative, junior faculty investigated questions specific to their own teaching but with the support of a cohort composed of other junior faculty and more senior faculty. Project reports focused on what was learned from the inquiry and resulting changes in teaching practice, including a systematic assessment of learning (Middendorf, 2004).
In another initiative, a group of faculty followed an action research process to investigate two questions concerning integrating technology and incorporating an interdisciplinary approach. The final activity was to construct an action plan for moving forward (Garcia & Roblin, 2008). Assessment of effectiveness was varied depending on what aspect was of interest. The meaningfulness of the inquiry process itself was assessed through questionnaires, focus groups, action plans, redesigned course materials, documentation of change in teaching practice, and portfolios. Some initiatives also sought to determine broader impact by looking at numbers of resulting conference presentations and publications, evidence of cross-unit collaboration, and alignment with strategic goals.
Authors who have written about the theoretical underpinnings of learning in a community, specifically in communities of practice (e.g., E. Wenger; J. Lave), and workplace learning (e.g., D. Boud) were cited regularly. Group inquiry initiatives were often designed around the elements described by E. Wenger as essential to a community of practice and around M. Cox’s guidelines in establishing learning communities in higher education. Individual investigations, on the other hand, tended to adhere to a more formal research structure, often citing the literature on the scholarship of teaching and learning (e.g., Boyer). As suggested in one article, “This approach shifts the attention . . . from the enhancement of teaching using traditional methods such as workshops, formal courses, or in-faculty curriculum and assessment advice, to encouraging and supporting participant research on learning and teaching, and disciplinary-based research” (Reid & Petocz, 2003, p. 105). Several initiatives targeted new faculty and cited literature specific to this group (e.g., R. Boice; A. Austin). In total, seven first authors were cited three or more times across the 19 action research or inquiry focus articles we reviewed (see Table 6 in Appendix B online).
Example from action research or inquiry focus cluster
Faculty and students (Koch et al., 2002) from a variety of disciplines (psychology, chemistry, political science, education, English, economics) collaborated in an initiative in which junior faculty investigated a self-selected facet of their teaching. They developed individual learning projects with goals, activities, and evaluation techniques. They worked with a faculty mentor and student associate of their choice over a one-year period to design, implement, and assess their project. Cross-disciplinary relationships developed from the regularly scheduled progress meetings. Evaluation was accomplished through student feedback, faculty observations, student performance, and self-reflection by student helpers and faculty mentors.
All of the articles in this cluster supported a process of inquiry that took the form of investigating one’s own teaching practice or collaborative forms of inquiry where groups of faculty explored questions of mutual interest. However, even when the investigation was about an individual’s teaching practice, they rarely worked alone and had support from other colleagues or educational developers. The investigations were initiated by faculty members or educational developers or both in collaboration—but faculty always chose the topic or question to be investigated. In keeping with an action research cycle (McNiff & Whitehead, 2006), the process of inquiry, the findings of the inquiry, and subsequent changes in teaching were variously documented as evidence of success of the different initiatives.
Looking Across the Six Clusters
Outcome Versus Process
The six clusters in our framework can be broadly characterized as reflecting an emphasis on either outcome or process. Initiatives designed to support a specific outcome are located in the skill, method, and institutional focus clusters of our framework, whereas initiatives designed to support a process are located in the reflection, disciplinary, and action research or inquiry focus clusters.
In initiatives emphasizing an outcome, the outcome is identified and anticipated ahead of time. In the skills focus cluster, where the intended outcome is to acquire or enhance specific observable skills or techniques, initiatives often determine a baseline against which to compare later posttesting. In the method focus cluster, where the intended outcome is that faculty come to understand learning as supported by a particular method (e.g., case-based learning) and use it in their teaching, assessment methods seek evidence of mastery of the method within the training sessions and evidence of use of the method in subsequent teaching practice. In the institutional focus cluster, the intended outcome is the successful diffusion and uptake of a particular teaching and learning innovation (e.g., technology integration) or orientation (e.g., student diversity). In this cluster, broad assessment measures consider the adequacy of structures and processes put into place to support effective diffusion and document evidence of uptake. In short, the core characteristics of initiatives in these three clusters have coherence in respect to the intended learning, intervention, and assessment of the predetermined learning.
Initiatives emphasizing a process, in contrast, do not specify a particular outcome but rather highlight a process of learning that may result in different outcomes for different faculty or multiple outcomes for an individual faculty member. The assumption in these initiatives is that engaging in the process (reflection, scholarly discussion and critique, action research or inquiry) will lead to changed thinking about teaching and, over time, more effective teaching. The focus is almost always on individual meaning making. A stated goal in many of these initiatives is to develop and support a questioning orientation to teaching and learning. Several initiatives formally measure changes in conceptions of teaching and learning through inventories, interviews, and/or the analysis of projects. Otherwise, effectiveness is most often determined by analyzing the same activities in which faculty engage to support the reflection process (faculty and student journals, teaching projects, teaching portfolios, peer observation, teaching philosophies). In short, the core characteristics of these three types of initiatives are consistently aimed at supporting a particular process of learning.
The differences between educational development initiatives highlighting outcomes and those highlighting process challenge us to think more deeply about these differences. Our thinking has led us to explore the various assumptions designers of educational development make about the institutional, intellectual, and contextual positioning of their work.
Positioning of Educational Development
We use three sets of contrasts to organize our discussion of positioning: institutional location (centralized or decentralized), intellectual location (focused on content or ongoing professional development), and contextual location (teaching as individual practice or as socially situated practice). We recognize that any attempt to categorize has its limitations, and this surely is the case here. These contrasts overlap to some extent. We think, however, that they provide a useful guide for considering educational development practice and for considering the various clusters of educational development practice composing the six-cluster framework we describe.
Institutional Positioning
The institutional positioning of educational development is often referred to as being either a centralized or a decentralized model, or a hybrid of the two. Centralized in this instance refers to initiatives led by educational development staff from a centralized teaching support center who deliver workshops, consultations, and other programs in the name of that center; faculty members leave their own departments to attend. Decentralized, by contrast, refers to models where educational development staff take on more facilitative roles, working with faculty in academic units (e.g., facilitating teaching circles) or as members of project teams with faculty. It is not surprising that these terms are most evident in institutional documents that describe the functions of centralized teaching support centers and the roles of staff assigned to them (Clegg, 2009).
The initiatives in the skill and method focus clusters of our framework could be almost exclusively identified with a centralized model where educational developers work with individual faculty or groups of faculty and are responsible for planning and carrying out the initiative. In the institutional focus cluster, the broad goals of these initiatives lead to both centralized and decentralized ways of working for educational development staff. In the reflection, discipline, and action research or inquiry focus clusters, we see a hybrid of centralized and decentralized with educational developers assuming roles that involve varying amounts of facilitation ranging from being the primary architects in planning and implementing an initiative (workshops on reflective practice) to acting as facilitator and resource person for an essentially faculty-led process (faculty learning community).
Intellectual Positioning
The intellectual positioning of an educational development initiative refers to the intended learning: an emphasis on acquiring either particular content or the tools of ongoing professional development (Webster-Wright, 2009). If we consider the learning intention of initiatives in the skill focus and method focus clusters, then certainly the emphasis is on a specific content. Initiatives in the institutional focus cluster are more mixed. In some initiatives in this cluster, considerable effort was made to adapt the innovation to existing teaching practices and departmental cultures, and in others, there was little or no customizing of an essentially top-down initiative. The learning intention of the remaining three clusters, reflection, discipline, and action research or inquiry focus, is to support a process (reflection, scholarly discussion and critique, action research and inquiry) that can lead to ongoing professional development. Many of the initiatives in the reflection and action research or inquiry focus clusters were initiated, planned, and implemented by faculty members for the purpose of their own professional development without the support of educational development staff.
Contextual Positioning
We think contextual positioning is the most interesting to think about and perhaps provides the most substantive contrast. Here the distinction is between activities focusing on improving or enhancing an instructor’s individual teaching practice versus activities that engage faculty in teaching enhancement as a socially situated practice (Boud, 1999; D’Eon, Overgaard, & Rutledge, 2000; Gregory & Jones, 2009; McAlpine, Weston, Timmermans, Berthiaume, & Fairbank-Roch, 2006). The initiatives in both the skill and method focus clusters are embedded in the context of individual teaching practice with assessment measures constructed to collect evidence of newly learned knowledge and skills within one’s teaching practice. In the reflection, disciplinary, and action research or inquiry focus clusters, some initiatives support the learning process only at the level of the individual instructor, whereas others either embed the focus on individual practice within a support network of colleagues or focus on a collaborative group process where individual meaning making of new knowledge is a social process with faculty colleagues.
Along with these contrasts has come debate over which types are best or most effective or what the balance should be. Knight, Tait, and Yorke (2006) characterized programs and initiatives as ranging from “event-delivery methods” to situated approaches that take place within the context in which faculty work. They maintained that “there is still a place for event-based educational professional development, but it [should] complement, rather than displace, situated social learning” (p. 320). Webster-Wright (2009) leveled a stronger critique at what she considered an overreliance on formal, didactic approaches in professional development, arguing that these types of programs and initiatives were often “decontextualized . . . and separated from engagement with authentic work experiences” (p. 703). We are not convinced that these black-and-white views of educational development are as meaningful as they could be in deepening our thinking about educational development. And, in fact, even Webster-Wright (2009), in her critique of the professional development literature, concluded that
the implication of this separation is that learning at work is different from learning through attending a professional development workshop. Although the activities may differ, if the professional learns from either or both experiences, then this separation is artificial; a convention reinforced by prevailing discourse. (p. 713)
We have paid close attention to the design of the initiatives in our review, particularly to the internal alignment and coherence of core characteristics of initiatives to support learning. We acknowledge, however, that even a seemingly coherent design, although critically important, does not necessarily result in learning that is subsequently reflected in teaching and student learning. We understand the development process as a dynamic interplay among individual, disciplinary, and organizational elements and mediated by reflection on action. This is consistent with the model of professional growth proposed by Clarke and Hollingsworth (2002) that focuses on K–12 teachers. Holding such a conceptualization of educational development suggests that there are different points of entry and multiple pathways to faculty learning. McAlpine, Amundsen, Clement, and Light (2009) suggested that ultimately change is an interplay between thinking and practice:
Changes in teaching practice (theories in use) may lead to changes in thinking about teaching and learning (espoused theories) or changes in thinking about teaching and learning (espoused theories) may lead to changes in teaching practice (theories in use). (p. 272)
One critical observation we make when looking across the initiatives in the six clusters of our framework is how infrequently these researchers explicitly recognize the broader context in which faculty teach and, therefore, in which educational development happens. Only a few initiatives were explicit about the academic and social context in which faculty work and in which new knowledge must be embedded, practiced, and refined (Boud & Walker, 1998; Trowler & Cooper, 2002). This may be an illustration of Webster-Wright’s (2009) contention that much of research and practice in professional development considers “the professional and learning context as separate though related” (p. 712).
We tend to align with Boud (1999), who recognized that most “development takes place where faculty spend most of their time, departments, professional settings and research sites” (p. 3) and contended that these informal learning experiences may have a more profound influence on the understanding of teaching and learning and on teaching practice than organized activities labeled as development. At the same time, Boud (1999) recognized the benefit of well-designed outside learning to challenge taken-for-granted assumptions and provide innovative approaches, but he argued that “formalized approaches are also usefully conceptualized as being located in sites of academic practice” (p. 3).
We interpret Boud’s (1999) words as essentially meaning that there is a place and purpose for different paths to improvement of teaching and learning but that all must take account of the situated and social nature of teaching. At this point in time, we know more about how to design educational development initiatives to improve individual teaching practice but less about how this learning is actualized and embedded in the academic workplace (Åkerlind, 2007, 2008; Dall’Alba & Sandberg, 2006; Fitzmaurice, 2010; Menges & Austin, 2001; Zellers, Howard, & Barcic, 2008).
Conclusion
We believe that the narrow questions about impact and effectiveness of educational development asked by previous empirical reviews of the educational development literature may not be the best questions to ask to more deeply understand practice and to build a sound foundation for further practice and research. Kennedy (2007) reminded us that empirical reviews are necessarily narrow, and this can make them less “informative, for such reviews are likely to eliminate studies that introduce new ideas, use new methodologies, or use unique methodologies” (p. 146). We believe that this is the case with previous empirical reviews. We argue that the primary attention in previous empirical reviews to how the impact of educational development initiatives is assessed has led to the exclusion of many articles that present educational initiatives that focus on the learning process rather than on specific learning outcomes. It is noteworthy that although the extensive overlap of articles in our review with those of the reviews by Steinert et al. (2006) and by Stes et al. (2010) provides some evidence that we are all exploring substantially the same body of work, only one (Hubball et al., 2005) of these common articles fell in a cluster of our framework that focused on process as learning.
We submit that the framework we articulate in this conceptual review is a fruitful way to build a better understanding of the variation and complexity of educational development practice and the thinking underpinning this practice. The framework has the potential to serve as a tool for those involved in the practice of educational development to engage in the analysis of their practice. The conceptually consistent core characteristics of initiatives in each cluster and at least some evidence of common citations to the literature in each cluster suggest that five of our clusters have integrity as descriptors of educational development practice and underlying thinking. The exception may be the discipline focus cluster, which contained only four articles and not enough common citations to meet our criteria of at least three citations in three separate articles. We think that this cluster is conceptually distinct from the others, but we realize that additional examples are needed to distinguish it more clearly, particularly from the reflection focus cluster. In short, our better articulation of practice provides a model of intentional design and reflects the amount and kind of detail that is necessary if, as a field, we hope to build on one another’s research and practice in a meaningful and systematic way.
In identifying the core characteristics of each initiative, we provided information about the intention or goal of an initiative, the processes and activities planned to realize the intention or goal, and the evidence collected to demonstrate success in achieving the intention or goal. How the effectiveness of a particular initiative was determined was but one core characteristic considered as part of the whole design; it was not considered separately from the design itself, as in previous reviews. We think this is a critical point to make. Developing a better understanding of effective practice must be done in consideration of the whole of the design. The clusters of the framework we have articulated are not meant to be compared to one another in terms of effectiveness; indeed, this may be meaningfully impossible. We have, however, provided at least a possible inroad in getting closer to the possibility of comparing “apples with apples” (Desimone, 2009) in the quest to answer an elusive question: What are the features of educational development that make it effective? New research approaches drawn from the expanding methodologies of metasynthesis and metastudy could provide the means for this further examination (Paterson, Thorne, Canam, & Jillings, 2001).
We reflect on the work we have done in this study and ask ourselves if the literature we identified is evidence that educational development has become a more well-defined field of inquiry since Levinson-Rose and Menges (1981), authors of the first empirical review, provided the following critique:
A well-defined field of inquiry should draw upon coherent theory, subscribe to high standards of research, and build upon previous research in a systematic way. By these criteria, research on improving college teaching is not a well-defined field. For most studies, the basis in theory is strained and for some it is nonexistent. (p. 418)
Our review identified literature between 1995 and 2008, a reasonable amount of time to reflect any changes from the body of literature reviewed by Levinson-Rose and Menges. We did find, unlike Levinson-Rose and Menges, some consistency in the conceptual, theoretical, and empirical literature cited by articles within each of the clusters of our framework. We also found 16 authors who were cited as first authors three or more times in at least two different clusters in our model (see Table 7 in Appendix B online for details). This suggests some consistency and integrity in the field as a whole. Some of these multiple most-cited authors referenced classic theoretical works (e.g., J. B. Biggs; M. S. Knowles; D. A. Kolb; J. Lave; P. Ramsden; D. A. Schön), providing further evidence that this field is becoming more grounded in an established theory base.
The separation of the literature of educational development in the medical sciences from that in other academic contexts that was noted by Steinert et al. (2006) was reconfirmed in this review. There is little evidence of citations across these two bodies of literature or the relatively more developed professional development literature focusing on K–12 teachers. These three contexts for educational development remain in quite separate silos, with very little sharing of experience. This discourages the cross-pollination required to build on relevant research.
We also did not find much evidence of practice systematically building on other published reports of practice, a finding consistent with Levinson-Rose and Menges’s (1981) critique, although there was more evidence of this in some clusters of our framework than others (skill, action research or inquiry). This may be partially attributed to Stes et al.’s (2010) contention that practice is not described well or in enough detail in the literature to allow others to build on it. We tried in our review to redress this by excluding articles without sufficient detail of the practice and providing a model for what to include so that others will be able to build on it.
Our review has a number of limitations. Using only published literature means that we have undoubtedly missed many diverse and interesting examples of educational development practice that have not been formally documented. We reviewed only articles describing initiatives at medical or doctoral and comprehensive universities, our primary context of interest, and so surely missed interesting examples of educational development practices in colleges and polytechnics. Finally, we could examine educational development initiatives only one by one as published articles, so we could not know the whole of educational development practice or thinking as it actually happens in any one place.
Our conceptual review has provided a broadened view of educational development practice and the thinking underpinning it. The six-cluster framework provides a new way of thinking about the design of practice and a more meaningful basis on which to investigate the effectiveness of educational development practice. The review also provides some guidance for the continued development of this field.
Footnotes
Appendix A
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
