Leveraging the Delphi Technique to Enrich Knowledge and Engage Educational Policy Problems

Abstract

The Delphi technique is a research method that involves experts and stakeholders in structured deliberation about topics through multiple, iterative rounds of data collection such as surveys or in-person meetings. Although it has been utilized extensively in other fields for researching problems, when there is disagreement among key stakeholders or as issues are just emerging, it has been less common in educational research. In this article, the authors outline the Delphi technique, review the scope of its recent applications in educational research, and offer recommendations to advance efforts to support changes in policies and practices through Delphi approaches to research.

Keywords

educational policy policymakers educational reform qualitative research survey research

There are many challenges facing K-12 and higher education institutions today; educators, educational researchers, policymakers, and other stakeholders are often divided on enduring problems and are unsure about how to handle newer ones (Birnbaum & Shushok, 2001; Cuban, 1990; Dee, 2006; Ellett & Teddlie, 2003; Hendrickson, Lane, Harris, & Dorman, 2013; Labaree, 2004; Ramaley, 2006; Tyack & Cuban, 1995; Zemsky, 2009). There are the perennial questions about how to effectively evaluate teaching and learning, ensure access to education, and distribute the burden of cost for education that remain mostly unresolved despite many years of debate and extensive research. Other challenges have emerged more recently through the introduction of new technologies and their applications in education. For example, although there appear to be benefits, serious concerns are being raised about massive open online courses (MOOCs) and their implications for the quality of teaching, learning, and student outcomes amid a scramble to adopt these potentially disruptive technologies (Marshall, 2014; Sadhasivam, 2014). A commonality, though, among these issues—old and new—is that the topics at their core are not always fully understood; sometimes, there is substantial disagreement and conflict among stakeholders about the nature of problems and potential solutions. A persistent problem that sometimes stands in the way of resolving these challenges is an often-criticized gap between research and practice, which will be described in the first section of the article.

Tackling major challenges facing institutions, educators, and students is an important priority for educational research (Herrington & Gadsden, 2013). And, there is increasing interest among scholars in applying research to support changes in policies and practices and overcoming the traditional research–practice gap (Levin, 2004; Plank, 2011). One promising tool for addressing challenges faced by the education sector is the Delphi technique, a research method that involves various experts and stakeholders in a structured deliberation about important topics through the use of multiple, iterative rounds of data collection such as surveys or in-person meetings (Adler & Ziglio, 1996; Clayton, 1997; Linstone & Turoff, 1975; Salkind, 2010); researchers provide summaries between rounds to inform participants about the different perspectives that are emerging and to build on or refine the responses with each round. Delphi studies can be leveraged to serve goals for using educational research to facilitate change, but have been underutilized in our field. The knowledge that is cultivated and documented through research using the Delphi technique can be invaluable for achieving goals for helping educational researchers and practitioners to better understand problems that have been difficult to study, evaluate competing data and perspectives, inform policy design and implementation, and identify priorities for research and policy that are attentive to a constantly changing environment (Adler & Ziglio, 1996).

Over the years, different permutations have emerged that are well suited for a wide range of research questions and applications including policy development and analysis, research agenda creation, and action research (Franklin & Hart, 2007; Kezar & Maxey, 2014). Although other fields such as medicine, nursing, public policy, business, public health, and social work have effectively used Delphi approaches to help resolve complex problems, educational researchers have lagged behind in their adoption of the method (Salkind, 2010). We propose that the Delphi technique is not only underutilized in educational research, but that reliance on the Conventional Delphi approach (explained below) when the technique is utilized means that the potential benefits of these newer permutations have not been realized. We articulate the value of the Delphi technique to address a common critique of educational research—that it does too little to shape policy and practice. We feel that educational research and practice can benefit from the expanded use of this method.

To accomplish these goals, we begin by describing how some of the main problems associated with a persistent gap between research and practice create challenges for using research to effectively respond to challenges facing education. Next, we propose the Delphi technique as one tool to help overcome these challenges; we attend to the methodological characteristics and purposes of Conventional, Policy, and Change-Oriented Delphi studies, as well as some other common modifications. Then, we describe the very limited application of the Delphi technique in educational research. We conclude the article by offering a few brief examples of ways the method could be used in education. Our hope is to stimulate readers’ thinking about the opportunities and benefits that can be realized for leveraging Delphi approaches to produce new knowledge, seek resolution for perennial problems, and help decision makers better understand emerging challenges to improve the effectiveness of policy design and implementation.

Problems of a Persistent Gap Between Research and Practice

One of the persistent challenges facing researchers, practitioners, and policymakers in responding to issues—both long-term problems and emerging ones—is a long-standing and often critiqued gap between research and practice in K-12 and higher education (Burkhardt & Schoenfeld, 2003; Cohen & Garet, 1991; Hirsch, 2000; Kaestle, 1993; Kezar, 2000a, 2000b; Lagemann, 1999; Robinson, 1998; Sroufe, 1997). There are a number of explanations for why this gap exists and persists—including concerns among educators about prestige and respect for the field, a preference for the development of theoretical knowledge, the influence of tenure and other incentive schemes, the perpetuation of these norms through socialization and gatekeeping behaviors among top academic journals, and demands placed on practitioners—which are detailed below. Although we do not detail all explanations for the research to practice gap, we highlight some of the most often cited explanations. Researchers and practitioners are characterized as living in different worlds (Hirsch, 2000); and many of these issues outlined below are suggestive of different cultures, challenges, concerns, and incentive schemes that sometimes place their priorities at odds with one another (Cross, 2000; Floden, 2011; Kezar, 2000a; Plank, 2011). We argue methods can either exacerbate or contribute new approaches to overcoming challenges created by these distinctive cultures.

The level of prestige ascribed to educational research as a discipline in the university is one factor that has had a role in creating and perpetuating the gap (Burkhardt & Schoenfeld, 2003; Colbeck, 2000; Labaree, 2003, 2004). Although educational research is often thought of as a more applied field, the perception of low prestige of the field as compared with other disciplines has increased pressure to focus efforts on building a stronger foundation of pure or theoretical, rather than applied knowledge (Hargreaves, 1999; Labaree, 1998, 2004). As a result, researchers have sought to contribute to the development of a more autonomous field of abstract, theoretical knowledge viewed by practitioners and policymakers as being far removed from conditions and circumstances experienced in practice in schools and an esoteric vocabulary intended to demonstrate their status as experts (Brown, 1992; Cohen & Barnes, 1999; Colbeck, 2000; Hirsch, 2000; Labaree, 1998). However, as Burkhardt and Schoenfeld (2003) pointed out, “theory qua theory will only take us so far (and not far enough)” (p. 13). Journal editors and reviewers reinforce these behaviors; as gatekeepers for publication, they command adherence to these norms.

In service of these arrangements, reward structures and high-stakes incentives such as tenure lead researchers to pursue work that will gain them maximum recognition and be received favorably by their peers or selected for publication in prestigious research journals, but that may never be of use to the field (Conrad, 1989; Fox, 1985; Keller, 1985). The result is a lack of useful and accessible information to solve pressing challenges facing our education systems (Sroufe, 1997). All of this naturally creates distance between researchers and practitioners or policymakers and fuels criticism that educational research is too often inattentive to solving problems encountered in the provision of education—at all levels—in an increasingly complex environment (Kezar, 2000b; Levin, 2004).

Researchers are also accused of attending repeatedly to the same topics, rather than creatively engaging new and emerging challenges, passing over opportunities to push the boundaries of knowledge and understanding of important issues that could help to improve policy and practice (Cohen & Barnes, 1999; Cross, 2000; Goldhaber & Brewer, 2008; Keller, 1985; Kezar, 2000a; Tierney & Rhoads, 1994). For example, in higher education, faculty focus on issues of retention based on theoretical concerns but ignore current technology issues; research in K-12 often focuses more generally on teacher development based on cognitive theories, overlooking new challenges related to the common core standards. Young scholars, in particular, are discouraged from taking risks, questioning the status quo, utilizing innovative methodologies, and producing work intended for use by practitioners if they desire to achieve tenure (Colbeck, 2000; Fox, 1985). Unfortunately, these patterns of early-career faculty life often become habits that persist long after one receives tenure.

Practitioners and policymakers are also blamed for contributing to the gap. They face increasingly complex and fast-paced conditions for decision making, unique local challenges, and accountability demands for immediate results (Cohen & Garet, 1991; Hill, 1996). As a result, they fail to find time to read or make use of research—or only do so when it serves to justify their existing ideas and plans (or conversely, to attack programs). Instead, they rely more on personal experience, practical knowledge, and anecdotal evidence in responding to the challenges they face.

Studies could more frequently contribute to reducing disagreements about the nature of problems and potential solutions by generating knowledge about emerging problems in a timely manner or facilitating research that brings groups with divergent views together, for example. Instead, these tensions end up exacerbating the division among researchers and practitioners and their conflicting positions (Burkhardt & Schoenfeld, 2003; Cohen & Garet, 1991). As a result, both groups miss out on opportunities to work together to solve problems and learn from one another. Working collaboratively, researchers, practitioners, and policymakers—not merely as consultants or consumers of one another’s work, but as mutual learners—could facilitate the development of a robust research agenda that helps to resolve long-standing problems (Cohen & Garet, 1991; Kaestle, 1993). This would have the effect of helping to position the field to respond with greater haste to new challenges that emerge as a result of changes in the environment such as technological disruption and social, economic, and political change (Kezar, 2000b).

There are indications that educational researchers are open to this sort of change and increasing their collaboration with practitioners to overcome challenges (Herrington & Gadsden, 2013; Levin, 2004; Plank, 2011). For example, scholars are increasingly using research techniques such as action inquiry, participatory action research, and cooperative inquiry that help with creating changes in practice and developing new policies more directly, narrowing the gap between research and practice (Brydon-Miller, Kral, Maguire, Noffke, & Sabhlok, 2011; Cancian & Armstead, 1990; Levin, 2004; Plank, 2011; Reason, 1994). These sorts of participatory methods are just a few examples of approaches that focus on problems that exist in the real world, draw on the knowledge among stakeholders who are connected to those problems, and involve stakeholders in designing studies. They also include stakeholders in data collection, interpretation, and help to facilitate the integration of research findings by raising consciousness or engaging participants in problem definition, policy design, and political action (Brydon-Miller et al., 2011; Cancian & Armstead, 1990; Hirsch, 2000; Lincoln & Guba, 2000; McNiff, 2013).

In addition to these and other strategies to close the gap, educational researchers should continue to seek out additional innovative, methodological approaches for meeting the demands of a more collaboratively designed and applied research agenda. The Delphi technique is one tool that can be particularly helpful in advancing these efforts but is currently underutilized in educational research. Although the focus in this article is on one particular methodological approach to help narrow the research–practice gap, a number of publications suggest some other useful approaches. For an example, see Burkhardt and Schoenfeld (2003), which explores some of the barriers to change and potential solutions for overcoming them; Kezar and Eckel’s (2000) edited volume also includes several perspectives on narrowing the gap.

The Delphi technique is useful for studying a wide range of research questions; it has been utilized for cultivating data and information about a topic by documenting expert or stakeholder views, recording or exploiting collective knowledge that is shared among groups of professionals, and facilitating new ways of thinking about problems and potential solutions (Franklin & Hart, 2007). The Delphi technique has helped to cultivate knowledge to support a more complete understanding of topics and problems that are not easily examined using precise analytical techniques or where there is limited understanding or consensus (Adler & Ziglio, 1996; Linstone & Turoff, 1975). Scholars have also found the Delphi technique helpful for researching topics wherein rapid changes in the environment or increasing complexity necessitate a reexamination of previously held knowledge or historical perspectives (Franklin & Hart, 2007). And, because Delphi studies can examine topics as these changes are occurring, they can be an important tool for determining or refocusing a research agenda that serves emerging priorities for scholarship and policy development.

The Delphi Technique

The Delphi technique does not refer to a single process or approach; although it was originally developed and used for forecasting future conditions and building consensus, it has been adapted and modified to serve a much broader range of potential applications and purposes (Judd, 1972; Keeney, Hasson, & McKenna, 2001); the varied approaches also accommodate a wide range of epistemological stances. In their seminal work on the Delphi technique, Linstone and Turoff (1975) defined it as “a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem” (p. 3). The main characteristics of the Delphi technique are the continuous engagement of participants who are knowledgeable about an issue (i.e., experts or relevant constituencies), multiple and iterative rounds of data collection, and providing participants with structured summary feedback between rounds (Adler & Ziglio, 1996; Linstone & Turoff, 1975; Salkind, 2010). This facilitates a more deliberative process; participants are introduced to different perspectives and refine their own views as the group moves toward consensus or developing a better understanding of their differences. Typically, some degree of anonymity is maintained among the participants to allow for the free sharing of ideas without pressure to conform, but more participatory approaches are also bringing groups together to solve problems collaboratively.

In the sections that follow, we describe some of the different approaches that have emerged over time. We begin with the original Conventional Delphi and its key process points. Then, we review the Policy, Modified, and Change-Oriented Delphi approaches, explaining how each differs from the Conventional approach. The main characteristics of each version are also summarized in Tables 1 and 2.

Table 1.

Typical Characteristics of Conventional and Policy Delphi Studies.

	Conventional Delphi	Policy Delphi
Main purpose	Future forecasting; developing consensus on technical issues or problems.	Draw out opposing views about problems; expose range of potential policy alternatives.
Typical process	Participants are purposefully identified and invited to participate.
	Multiple (usually 3-5), iterative rounds of surveys are administered; the first survey is typically used to determine a baseline range of views and subsequent surveys help to refine views. Conventional Delphi studies may involve more rounds than Policy Delphi studies because they are often intended to develop consensus.
	Researchers provide structured feedback to participants between rounds to help to distill ideas, often toward building consensus.
Type of participants	Typically a homogeneous expert group with specialized knowledge about a topic.	Experts can be engaged, but the process benefits from a more heterogeneous group to support the identification of multiple perspectives; participants can alternately be practitioners or individuals who are affected by a policy or have familiarity with it.
Sample size	Varies depending on the scope of the issue or problem and the resources available; often between 30 and 60 participants are involved.	Varies depending on the scope of the issue or problem and the resources available; may be much larger because of the expanded scope.
Intended outcome	Consensus regarding the most likely future outcome for informing strategic planning or decision making.	Understanding of the range of views about an issue or problem and the potential solutions; participants help to refine knowledge of alternatives but often are not engaged in action or decision making.

Table 2.

Characteristics of a Change-Oriented Delphi Study.

	Change-oriented Delphi
Main purpose	To solve a complex social or natural problem collectively
Process	In-person meetings; resource materials; involves stakeholders in design of survey or meetings; likely involves much longer process
Type of participants	More intentionally involves individuals who can help address the problem or issue and implement solutions, often includes them at different time periods.
Sample size	May include larger samples to represent more diverse sets of stakeholders.
Intended outcome	A series of solutions that are actually implemented in practice.

Origins and the Conventional Delphi

The Delphi technique was created in the early 1950s by RAND as a tool for consensus building and future forecasting (Dalkey & Helmer, 1963; Linstone & Turoff, 1975). The first known application surveyed military defense experts about how to respond to a hypothetical nuclear attack (Salkind, 2010). This and other Conventional Delphi studies typically focus on building consensus about problems that are more technical in nature.

Samples

Identifying the correct participants is an important first step in a Delphi study (Linstone & Turoff, 1975; Pill, 1971). Determining the types and numbers of participants depends on the purpose and scope of the study, so there is a great deal of variation among studies (Delbecq, Van de Ven, & Gustafson, 1975). The participants in a Conventional Delphi study are typically more homogeneous; they are selected because they are part of a clearly defined group of experts. Although the matter of who has the requisite “expertise” to participate varies from one study to another, it is an important consideration for ensuring the validity of the outcomes of a Delphi study. So, a great deal of attention is given to sample selection in the conceptual and research literature on the method; there is general agreement that participants should possess sufficient knowledge about the topic being studied to contribute relevant inputs (Adler & Ziglio, 1996; Pill, 1971; Reeves & Jauch, 1978). Sample sizes range from only a few participants to several hundred; a typical sample includes between 30 and 60 individuals.

Survey design, data collection, and summaries

Survey protocols are the most common method used for collecting responses; questions may be quantitative (e.g., scales or Likert-type) or more qualitative (e.g., open-ended response; Linstone & Turoff, 1975; Salkind, 2010). Sometimes, a few participants are invited to help with designing the surveys or checking them to ensure comprehension. Typically three to five iterative rounds of surveys are administered (Clayton, 1997). The first survey is often more qualitative and may have a broader focus to help determine the areas of focus that will be examined in greater depth in subsequent rounds (Rowe, 1994). Between rounds, researchers analyze the responses and provide structured feedback to participants so they can better understand the range of perspectives that are emerging and consider alternate views or better articulate their own. Researchers utilizing the Delphi technique are encouraged to use care in developing data collection protocols and interpreting data for summary feedback between rounds and general research findings (Reeves & Jauch, 1978).

Emergence and Development of the Policy Delphi Approach

In the late 1960s, Delphi researchers broadened the representation of their samples to understand complex problems more completely (Franklin & Hart, 2007; Strauss & Zeigler, 1975). They would include not only recognized experts but also other stakeholders who have knowledge about or who are affected by a particular problem; studies often also include groups that would be responsible for generating and implementing potential solutions. Because these studies typically involve a more diverse set of participants, they are able to more completely identify the range of competing views about a topic and result in greater reliability and complexity among a study’s findings (Hussler, Muller, & Ronde, 2011). These changes from the Conventional Delphi approach expanded how the method could be used to study complex problems, incorporating a broader set of perspectives. The overall process largely remained the same (e.g., survey design, data collection, and summaries), but sample sizes were often expanded to accommodate the inclusion of diverse groups, and surveys were typically designed differently to draw out opposing views, rather than to build consensus (see Table 1).

These changes signaled a new type of application of the technique. Policy Delphi studies sought to inform and improve decision making by cultivating a more complete understanding of opposing or conflicting views that could influence policy design and implementation (Franklin & Hart, 2007; Linstone & Turoff, 1975). This new approach allowed researchers to support decision making by exploring questions that are more difficult to study using other methods. For example, Millar, Thorstensen, Tomkins, Mepham, and Kaiser (2006) used a Policy Delphi approach to research the various ethical dimensions of biotechnology assessments about genetically modified seafood; Udoh, Mantell, Sandfort, and Eighmy (2009) drew on the knowledge of researchers, activists, and HIV/AIDS advocates to better understand the intersections of poverty, migration, prostitution, and HIV transmission in Nigeria. Policy Delphi studies such as these can contribute to an expanded base of knowledge in a field by acknowledging the diverse range of perspectives and factors that shape extremely complex problems and their resolution.

Modified and Change-Oriented Delphi Approaches

Over time, researchers have made additional modifications to the Delphi technique (Hasson, Keeney, & McKenna, 2000). These variations allow researchers to be responsive to the unique requirements and goals of their research (Adler & Ziglio, 1996; Hussler et al., 2011). Modified Delphi studies amend one or more of the characteristic process points or methods described above. For example, sometimes individual or group interviews are used in one or more rounds instead of surveys, the number of rounds can be changed, or participants are convened for an in-person meeting. Modifications can be made while maintaining the benefits and goals of the Delphi technique. In addition to process modifications, new media and technology platforms have often been utilized to expedite data collection, keep participants engaged, and facilitate data analysis or the distribution of summaries between rounds (Franklin & Hart, 2007).

Recently, scholars have also begun to identify new ways the Delphi technique can go beyond studying problems to working with stakeholders to address them (Beamish & Bryer, 1999; Coady et al., 2007; Glass, Scott, & Price, 2013). This emerging application builds on the participatory nature of Delphi studies and aligns the processes and goals more closely to the core assumptions of the participatory paradigm. Researchers engaging in what has been labeled the Change-Oriented Delphi approach would select a diverse sample that includes additional stakeholders who are well positioned to work collaboratively with researchers with the goal of designing and implementing solutions and changes. Such studies differ from earlier iterations by more actively involving participants in deciding the research agenda, designing survey instruments, and implementing solutions that emerge from the studies.

Challenges and Limitations

Like any methodology or approach, the Delphi technique has its own set of challenges and limitations (Clayton, 1997; Powell, 2003). It has at times been critiqued as being too unscientific (Sackman, 1975) or too dependent on opinions or intuitive judgments (Helmer, 1977). More specific criticisms of the method include a low level of reliability of judgments among experts, the sensitivity of results to ambiguity in the survey that is used for data collection in each round, the potential for researcher bias to influence survey design and interpretation of participant feedback, and the difficulty in assessing the degree of expertise incorporated (Clayton, 1997; Franklin & Hart, 2007; Makridakis & Wheelright, 1978; Powell, 2003; Rowe and Wright, 1999). Over time, though, many of these earlier criticisms (e.g., being unscientific) have been reconsidered. Some challenges remain such as the difficulty of assessing expertise (Salkind, 2010), although others such as survey design and questions are shared with many methods.

Another challenge is the amount of time required of researchers and participants in Delphi-based research (Hsu & Sandford, 2007; Salkind, 2010). There are typically several rounds of iterative surveys and summaries. Researchers often devote large amounts of time to facilitating this process. And, participants might be limited in the amount of time they can devote to reviewing summaries and completing multiple rounds of surveys (Clayton, 1997). The overall timeline for conducting a Delphi study and realizing the intended benefits also presents limitations; when Delphi-based research is presented in research articles, the outcomes for policy and practice are often not reported—in part because they are not fully realized as quickly as research findings that serve as the basis for scholarly articles. This means that the full benefits of conducting Delphi-based research for improving practice are underreported in the literature.

Despite these challenges and limitations, the potential benefits of the Delphi technique, including the ability to draw on perspectives that are not typically accessible through the research experience and literature reviews alone, are greater than the risks (Clayton, 1997; Delbecq et al., 1975; Franklin & Hart, 2007). Through purposeful planning, these challenges and limitations can be easily mitigated (Landeta, 2006).

Common Applications in Educational Research

The Delphi technique has been used in educational research for several decades, although its application has been less common than in other fields and more narrow in scope (e.g., topics of studies, heavy reliance on Conventional Delphi approaches, more homogeneous samples). Enthusiasm for Delphi-based studies grew in education in the 1970s as the method developed (Judd, 1972; Linstone & Turoff, 1975). However, early studies were typically only intended to provide forecasts to support administrative planning (Clayton, 1997); the questions asked examined the types of changes that would occur or the likelihood and possible impact of events. Although the Delphi technique can contribute to knowledge to resolve unsatisfactory conditions for which few other means or methods have contributed to solutions, in educational research, studies are still largely confined to a narrow set of approaches and topics, which we describe below.

To show how the Delphi technique is being used in our field and suggest ways that it could be expanded to enhance the utility of educational research, we conducted a review of journal articles about Delphi-based research in education published between 2003 and 2012. Articles were identified through a search for manuscripts in the Education Resources Information Center (ERIC), the largest digital library of education literature. Although some articles may not be captured in ERIC, it is the most comprehensive database in education; the trends we observed in ERIC would likely be reflected in other databases, as well. Articles were individually evaluated to determine relevance, level of education (i.e., K-12 or higher education), alignment to Conventional, Policy, or Modified Delphi approaches, and the primary topic or purpose of each study.

The number of Delphi-based educational research journal articles published annually has increased over the past decade, from just nine articles in 2003 to 35 in 2012 (see Figure 1). By comparison, searches for recent Delphi-based research publications in other applied fields such as nursing or social work yield hundreds of articles. In addition, although the Delphi technique has been the main method used in a growing number of educational research articles over the last decade, the majority are still largely limited to the Conventional approach and a fairly narrow set of topics.

Figure 1.

Number of Delphi-based educational research articles in ERIC, 2003-2012.

Authors do not always explicitly identify whether theirs is a Conventional, Policy, or Modified Delphi study; however, the articles we reviewed typically involved mostly homogeneous samples and were intended to forecast future conditions or develop consensus. They were largely focused on identifying appropriate curricula, skills, and competencies for students and educators, evaluating measures of success or obstacles, and so on. They did not usually draw on the perspectives of more diverse samples to cultivate a more complete representation of conflicting views.

Most of the Delphi-based studies in educational research that were reviewed (78.1%) had one of seven topics as their primary focus (see Figure 2 and Appendix for additional details). Studies that were intended to determine the appropriate curriculum or competencies for students in specific disciplines or sub-disciplines comprised more than one quarter of the studies (27.6%); this was the largest single category and generally was the largest represented annually, as well. One example is an international study by Rossouw, Hacker, and de Vries (2011), which drew on experts’ perspectives to identify key concepts that should be taught to students in engineering and technology courses. The remaining foci were research agenda development (10%), educational technology (9.7%), evaluation of students’ performance or needs (9.2%), future forecasting (8.2%), identifying key competencies or knowledge for faculty members (7.1%), and program or service evaluation (6.1%), which mostly emerged in the latter half of the decade we reviewed. These limited applications and a tendency to involve more homogeneous expert groups, rather than the perspectives of diverse stakeholders, suggest there is untapped potential for utilizing the Delphi technique in educational research.

Figure 2.

Number of Delphi-based studies by topic, 2003-2012.

There was also an imbalance in the numbers of studies focused on topics in higher and K-12 education; Figure 3 illustrates how nearly 2 times as many Delphi-based articles about higher education were published. This disparity is partially explained by the prevalence of studies focusing on curriculum design and student competencies or skills, which were typically conducted to determine the educational needs and knowledge demands of college students preparing to enter the workforce. In K-12 education, curricula are typically prescribed or more static than in higher education and are less likely to be decided through deliberation between faculty and other experts. Studies that were conducted to contribute to the development of a research agenda were also focused on topics in higher education in greater numbers. Examples include Wright’s (2007) international study to determine research priorities around sustainability; Schmidtke (2010) identified areas where additional research was needed to better understand factors influencing American Indian student success in college.

Figure 3.

Number of Delphi-based studies by level, 2003-2012.

Realizing Additional Potential for Research and Improving Practice Using the Delphi Technique

There remains untapped potential for using the Delphi technique in educational research. Certainly, the types of Delphi studies that have been conducted in education (e.g., developing the best curriculum, identifying areas for research) can continue. However, by expanding our application of the method to Policy and Change-Oriented approaches, educational researchers can make new contributions to help solve major perennial and emerging challenges. Decision making in education rarely involves a choice between one alternative and another (Kennedy, 1999); rather, it involves making accommodations to integrate numerous constraints, priorities, and ideals. This yields a key opportunity for the Delphi technique to make contributions to educational practice. The knowledge that could be created by making better use of the Delphi technique has the potential to foster a better understanding about the nature of key problems, identify obstacles for policy design and implementation, and help to advance potential solutions and reforms in our field.

We propose that the use of the Delphi technique in educational research can be expanded in two key ways to address these and other concerns. The first is by utilizing the technique for studying topics or problems that have divided education stakeholders, where the true nature of problems and effects of proposed reforms are not fully understood, or when changes are still emerging. Researching these topics could make important contributions to improving practice, but is too often passed over for simpler or less controversial studies; studies could also effectively draw on practitioner knowledge of tensions and changing conditions on the ground. A few examples are included below to illustrate the sorts of challenges that could be studied, as well as how the Delphi technique might be applied. The second way to expand use of the Delphi technique is by embracing Policy and Change-Oriented Delphi approaches that engage a wide range of practitioners, content experts, policymakers, and other stakeholders through Delphi approaches. By including as many groups that have knowledge about a topic, are affected by policies and practices, or are responsible for executing plans, we can evaluate how conflict and other factors influence policy design and implementation, and effectiveness in achieving intended goals. We can also seek to engage practitioners and policymakers more thoughtfully as partners in the creation of knowledge that can be applied toward improvements in the field (Cross, 2000).

Below, we identify some areas where new Delphi studies in educational research could generate new data and a better understanding of challenges to facilitate change. These do not represent the full range of possibilities but are offered as a few examples to stimulate thinking about how the Delphi technique could be applied to researching a larger set of challenges. In selecting examples, we have tried to be attentive to highlighting topics that have typically not been studied using the Delphi technique. We also illustrate examples of different Delphi approaches—Policy and Change-Oriented Delphi—that have been applied less frequently to illustrate their potential. Although we do not fully outline the research design due to space constraints, we think the examples demonstrate the potential.

A Policy Delphi Approach for Examining Views on Charter Schools

One area where there is great potential for using the Delphi technique in education is the study of topics that have divided stakeholder groups. Over the last 20 years, politicians, policymakers, educators, parents, unions, interest groups, and others have been engaged in vigorous, public debate about the implications of charter schools (Bulkley & Fisler, 2002; Henig, 2008; Vergari, 2007). This debate has been a part of a much larger reform movement to expand school choice (Bulkley & Fisler, 2002; Vergari, 2007). The introduction of these alternatives to traditional public schools has aggravated tensions over who should control the provision of publicly financed education and choices available to families. Increased competition for limited resources and perennial questions about the effectiveness of each model in ensuring students’ success have sustained this debate for many years. Groups on both sides of this issue have utilized data and research to support their claims, as well as to discredit opponents. So, policymakers and others are confronted with conflicting evidence about the effects of the rise of charter schools; the polarization of views has affected how laws about charter schools are designed and how they operate.

Research on charter schools is an area where a Policy Delphi study could be used to develop a more complete understanding of opposing views and evaluate the full range of alternatives that are available to decision makers. For example, a Policy Delphi study on charter schools could tap into the knowledge and experiences of a wide ranging set of stakeholders who hold dissenting views about the issue or who are affected in various ways. Iterative rounds of anonymous surveys would allow these stakeholders to set aside political and rhetorical arguments typically engaged in public debate and contribute to a more complete understanding of the main concerns, needs, and demands of different groups who are affected by this issue. By providing feedback to participants between rounds, researchers could help groups with divergent views not just to examine those views that characterize their differences but also to begin exploring the sometimes-overlooked points of consensus that can be leveraged as the basis for change. This process could also facilitate policy design that is more responsive to various needs and concerns, actively involving parents, teachers, and other groups whose voices are often left out of the policymaking process and look past competing rhetoric to consider how sources of conflict are affecting policies’ effectiveness.

Examining Teacher Training and Evaluation Through a Change-Oriented Delphi Study

Delphi studies are also helpful for researching problems that are not well understood and to contribute to designing solutions. One such area of practice where the method could be applied is teacher training and evaluation. The Delphi technique could be used to facilitate a structured examination about how to best prepare primary and secondary teachers and evaluate their effectiveness. These are complicated questions, which are a regular source of consternation for teachers, administrators, unions, school boards, and others who all seek, but sometimes disagree about which models are pursued and how they are implemented.

Although groups within education debate these issues, outside groups have questioned the ability of educators to make changes to the existing education system themselves. For example, in a Washington Post opinion article, Bill Gates (2013) warned that “as states and districts rush to implement new teacher development and evaluation systems, there is a risk they’ll use hastily contrived, unproven measures.” Some change seems inevitable given the amount of attention being paid to these challenges. However, whereas some of the external groups (e.g., foundations) pushing for reform have been criticized for excluding contrary views from discussions, educational researchers can counter those efforts by involving all of the opposing groups in a deliberative, open examination using a Policy or Change-Oriented Delphi approach. For example, researchers working collaboratively with leaders from a state board of education could conduct a Change-Oriented Delphi study with the goal of engaging representatives of key stakeholder groups to design and implement a new model for training, supporting, and evaluating teachers.

Change-Oriented Delphi studies are designed to address a social problem, involve a variety of different stakeholder groups, and help groups to understand one another’s perspectives—engaging them in developing a collective commitment to change. Participants are sometimes involved in designing and planning studies; their perspectives can also contribute to the creation of resources to guide implementation and ongoing efforts. In a study, participants—including an inclusive sample of state- and district-level policymakers, teachers, principals, faculty development and assessment experts, union leaders, and others—would collectively set goals for the research, develop survey protocols in the earlier rounds, and frame agendas for in-person meetings in later rounds, which would be focused on developing plans for implementing the group’s ideas. The group might decide to conduct two rounds of surveys before convening meetings to review findings and discuss an action plan for implementation. This strategy would help researchers to provide decision makers with a more complete assessment of the demands for teacher preparation and evaluation, and obstacles arising from intra- and intergroup tensions, and determine points of consensus to facilitate and inform policy making and implementation; it would also bring groups to the table who are not always involved in policy discussions, further engaging participants in creating change.

Using a Modified Policy Delphi to Understand the Emerging Effects of Disruptive Technologies

Researchers have found using the Delphi technique to be beneficial for studying new phenomena and trends while they are still emerging. New technologies and the quickening pace of technological change are causing uncertainty about the future of K-12 and higher education among educators, policymakers, foundations, and for-profit education enterprises (e.g., institutions and education service providers). Delphi studies of educational technology have typically utilized a Conventional Delphi approach, but Policy Delphi studies could also make strong contributions. During a period of unprecedented advancements in technology, when new tools and programs seem to be obsolete as soon as they hit the market—and certainly before we have introduced them in the classroom—the Delphi technique could be helpful in bringing together diverse perspectives to attempt to get ahead of changes (Nworie, 2011). There have been a few studies in this area such as Holden and Wedman’s (1993) research on potential uses of computer-mediated communications technology in higher education and Pollard and Pollard’s (2004) study, which forecasted research priorities in educational technology; other curriculum and program design studies have also included changing technology as an area of focus.

Researchers could consider expanding the scope of existing studies by integrating a broader range of stakeholder perspectives. An example of a Modified Policy Delphi study would draw on the knowledge and experience of diverse stakeholders such as teachers, students, professional staff from a sample of schools serving diverse student populations, policymakers, and technology researchers and experts such as software, video game, and mobile application designers. Focus groups of the technology experts could be convened to make projections about emerging technologies, followed by several rounds of online surveys involving the full group to explore potential opportunities, benefits, and challenges that are associated with new products and platforms. Such a study could help to forecast not only which types of technologies might emerge in the future but also how they will affect changes or alter the needs of teachers and students. Researchers and policymakers could more completely consider the potential implications of emerging technological change for our institutions, student learning, and the work of faculty and professional staff to design policies and research agendas that respond more quickly to emerging challenges.

Much like the charter school debate described above, individuals and groups with different views or interests are making competing claims about the implications and what should be done. Educational researchers can help to inform decision making about emerging changes by making greater use of the Policy, Change-Oriented, and other Modified Delphi approaches; generating new and current data about changing conditions in education can help researchers to narrow gaps in knowledge that hamper policymakers’ and practitioners’ efforts to come up with timely solutions for addressing a changing K-12 and higher education landscape.

Implications

The examples above—illustrating how an expanded application of the Delphi technique could be used in studying and improving practice around charter schools, teacher training and evaluation, and emerging educational technologies—help us to imagine some of the possibilities for enhancing educational research and its potential to improve policy and practice. The method has the capability to facilitate an in-depth examination of problems that are not easily researched, engage experts’ and other stakeholders’ opinions, and surface and disseminate ideas and knowledge that may not be included in important discussions about policy design and the future of K-12 and higher education. The potential applications of Delphi-based studies in educational research are diverse. And, the Delphi technique’s different permutations can accommodate studies with wide-ranging research questions and epistemological foundations.

The method is certainly not a panacea for the problems faced by educational researchers and practitioners, nor will it eliminate all of the criticisms of educational research that exist. It is only one of many approaches available for conducting effective educational research and developing potential improvements for policy and practice. However, the alignment between the utility of the Delphi technique and the type of problems that educational researchers are increasingly trying to address position it as a useful tool. When utilized effectively, it can help to enhance the quality, reach, and impact of our work, contributing to efforts to narrow the pervasive research–practice gap.

Conclusion

Our hope is that educational researchers will consider how they might make greater use of the Delphi technique—and the full set of its permutations—to help find solutions to the major problems facing our field. The challenges that we face demand our best work; we need to leverage the best tools that are available to us. Educational researchers’ interests in applying their work to support changes in policies and practices to best serve the needs of students and society will be advanced as we expand our use of a method that is organized around a deliberative process and has the potential to engage diverse stakeholders—many whose voices are heard too infrequently—to be a part of creating solutions with us.

Footnotes

Appendix

Topic or Purpose Categories, Descriptions, Examples, and Percentage of Studies, 2003-2012.

Main topic or purpose of studies	Description of topic or purpose	Percent
Curriculum and Student Competencies or Skills	Determining the appropriate curriculum or method of instruction in disciplines or sub-disciplines, as well as skills or competencies for students.	27.6
	Example : Rossouw, Hacker, and de Vries (2011) drew up perspectives of a panel of international experts to identify key concepts that should be taught to students in engineering and technology courses.
Research Agenda Development	Identifying gaps in existing knowledge or needs for research about emerging topics.	10.0
	Example : Schmidtke (2010) identified areas where additional research was needed to better understand factors influencing American Indian student success in college.
Educational Technology Tools and Application	Examining instructional technologies, opportunities for their use, and changes affected by their use.	9.7
	Example : Sangra, Vlachopoulos, and Cabrera (2012) developed a comprehensive definition of the characteristics of e-learning models.
Student Evaluation	Determining approaches for evaluating student performance or identifying needs and challenges encountered by populations of students.	9.2
	Example : Lu, Hsu, Chan, Cheen, and Kao (2012) developed a tool for assessing “life stress” experienced by college student athletes.
Future Forecasting	Estimating the future landscape and emerging issues that will shape systems, institutions, or educators’ work.	8.2
	Example : A study by Westerheijden, Huisman, and de Boer (2004) evaluated current changes to produce three potential representations of the future of higher education in the Netherlands.
Faculty Member Competencies, Knowledge, or Skills	Identifying necessary skills or content knowledge for educators, often in specific fields.	6.6
	Example : Koster, Brekelmans, Korthagen, and Wubbels (2005) developed a list of task-related skills and competencies that teacher educators should possess.
Academic Program or Service Evaluation	Establishing factors that determine program or service quality, as well as studies that identify changes influencing design needs or delivery.	6.1
	Example : Shelton (2010) involved 43 experienced administrators of online education programs to create a list of 70 quality indicators and a process for scoring program quality.
Faculty Member Evaluation	Determining approaches for evaluating faculty performance or identifying needs and challenges encountered by certain groups of faculty members.	5.1
	Example : Dutta, Kundu, and Chan (2010) identified factors that limit the involvement of minority rehabilitation faculty members in scholarly pursuits.
Methods	Describing the use of the Delphi technique in educational research, often for specific purposes.	5.1
	Example : Franklin and Hart (2007) described the Delphi technique, as well as benefits and limitations of the research method.
Staff Competencies, Knowledge, or Skills	Identifying necessary skills or content knowledge for staff practitioners in specific roles or fields.	4.1
	Example : Shriberg et al. (2008) created a list of important social justice knowledge, skills, and strategies for school psychologists.
Policy Analysis	Establishing the effects of policies at the federal, state, or institutional level on outcomes or the need for changes in policies to meet changing needs.	3.6
	Example : Manley (2011) examined how state policy amendments affected the infrastructure of secondary school’s career and technical education programs.
Staff Member Evaluation	Determining approaches for evaluating performance or identifying needs and challenges encountered by certain groups of staff practitioners.	2.0
	Example : Graham (2010) evaluated the ways university general staff can contribute to student success at Australian institutions.
Administration Evaluation	Determining approaches for evaluating performance or identifying needs and challenges encountered by administrators.	1.5
	Example : Clark, Farmer, and Welch (2010) examined challenges encountered by administrators of career and technical programs that have implications for quality.
Administration Competencies, Knowledge, or Skills	Identifying necessary skills or content knowledge for administrators in specific roles.	1.0
	Example : Plinske and Packard (2010) surveyed a community college system’s trustees to identify characteristics, competencies, and desired professional experience of community college presidents.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Daniel Maxey is a PhD candidate and dean’s fellow in Urban Education Policy in the University of Southern California’s Rossier School of Education. He is interested in bringing his experience in policy analysis, politics, and public affairs to research on issues related to the public roles and responsibilities of colleges and universities, as well as change movements in higher education. Specifically, he is interested in examining the politics of higher education institutions and organizations, public roles and responsibilities of colleges and universities, civic engagement, and change movements in higher education.

Adrianna Kezar is a professor of Higher Education, University of Southern California and co-director of the Pullias Center for Higher Education. Kezar is a national expert of faculty, change, governance, and leadership in higher education and her research agenda explores the change process in higher education institutions and the role of leadership in creating change. Kezar is well published with 15 books, over 75 journal articles, and over a 100 book chapters and reports.

References

Adler

Ziglio

(1996). Gazing into the Oracle: The Delphi method and its application to social policy and public health. London, England: Jessica Kingsley.

Beamish

Bryer

(1999). Practitioners and parents have their say about best practice: Early intervention in Queensland. International Journal of Disability, Development and Education, 46, 261-278.

Birnbaum

Shushok

(2001). The “crisis” crisis in higher education: Is that a wolf or a pussycat at the academy’s door? In Altbach

P. G.

Gumport

P. J.

Johnstone

D. B.

(Eds.), In defense of American higher education (pp. 59-84). Baltimore, MD: Johns Hopkins University Press.

Brown

(1992). The definition of a profession: The authority of metaphor in the history of intelligence testing, 1890-1930. Princeton, NJ: Princeton University Press.

Brydon-Miller

Kral

Maguire

Noffke

Sabhlok

(2011). Jazz and the banyan tree: Roots and riffs on participatory action research. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), The SAGE handbook of qualitative research (pp. 387-400). Los Angeles, CA: Sage.

Bulkley

Fisler

(2002). A review of the research on charter schools. Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.

Burkhardt

Schoenfeld

A. H.

(2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3-14.

Cancian

Armstead

(1990). Participatory research: An introduction. Unpublished manuscript.

Clark

R. W.

Farmer

E. I.

Welch

S. M.

(2010). An examination of leadership issues facing Pennsylvania career and technical administrators. Career & Technical Education Research, 35, 47-62.

10.

Clayton

M. J.

(1997). Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educational Psychology, 17, 373-386.

11.

Coady

M. H.

Galea

Blaney

Ompad

D. C.

Glidden

Vlahov

(2007). Increased interest in vaccination following a community-based intervention in NYC [PowerPoint slides]. Retrieved from https://apha.confex.com/apha/135am/recordingredirect.cgi/id/17434

12.

Cohen

Barnes

C. A.

(1999). Research and the purposes of education. In Lagemann

Shulman

L. S.

(Eds.), Issues in Education Research (pp. 17-41). San Francisco, CA: Jossey-Bass.

13.

Cohen

Garet

(1991). Reforming education policy with applied social research. In Anderson

Biddle

(Eds.), Knowledge for policy: Improving education through research (pp. 123-140). London, England: Falmer Press.

14.

Colbeck

C. L.

(2000). Reshaping the forces that perpetuate the research–practice gap: Focus on new faculty. New Directions for Higher Education, 2000(110), 35-47.

15.

Conrad

C. F.

(1989). Meditations on the ideology of inquiry in higher education: Exposition, critique, and conjecture. Review of Higher Education, 12, 199-220.

16.

Cross

K. P.

(2000). The educational role of researchers. New Directions for Higher Education, 2000(110), 63-74.

17.

Cuban

(1990). Reforming again, again, and again. Educational Researcher, 19(1), 3-13.

18.

Dalkey

Helmer

(1963). An experimental application of the Delphi method to the use of experts. Management Science, 9, 458-467.

19.

Dee

(2006). Institutional autonomy and state-level accountability: Loosely coupled governance and the public good. In Tierney

W. G.

(Ed.), Governance and the public good (pp. 133-155). Albany: SUNY Press.

20.

Delbecq

A. L.

Van de Ven

A. H.

Gustafson

D. H.

(1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott Foresman.

21.

Dutta

Kundu

Chan

(2010). The conduct of socially valid investigation by culturally diverse researchers: A Delphi study. Rehabilitation Education, 24, 113-121.

22.

Ellett

C. D.

Teddlie

(2003). Teacher evaluation, teacher effectiveness, and school effectiveness: Perspectives from the USA. Journal of Personnel Evaluation in Education, 17, 101-128.

23.

Floden

R. E.

(2011). Speaking truth to policy and practice. In Conrad

C. F.

Serlin

R. C.

(Eds.), The SAGE handbook for research in education: Pursuing ideas as the keystone of exemplary inquiry (pp. 29-42). Thousand Oaks, CA: Sage.

24.

Fox

M. F.

(1985). Publication, performance, and reward in science and scholarship. In Smart

J. C.

Tierney

W. G.

(Eds.), Higher education: Handbook of theory and research (pp. 255-282). Bronx, NY: Agathon Press.

25.

Franklin

K. K.

Hart

J. K.

(2007). Idea generation and exploration: Benefits and limitations of the Policy Delphi research method. Innovative Higher Education, 31, 237-246.

26.

Gates

(2013, April 3). A fairer way to evaluate teachers. The Washington Post. Retrieved from http://www.washingtonpost.com/opinions/bill-gates-a-fairer-way-to-evaluate-teachers/2013/04/03/c99fd1bc-98c2-11e2-814b-063623d80a60_story.html

27.

Glass

J. H.

Scott

A. J.

Price

M. F.

(2013). The power of the process: Co-producing a sustainability assessment toolkit for upland estate management in Scotland. Land Use Policy, 30, 254-265.

28.

Goldhaber

D. D.

Brewer

D. J.

(2008). What gets studied and why: Examining the incentives that drive education research. In Hess

F. M.

(Ed.), When research matters: How scholarship influences education policy (pp. 197-217). Cambridge, MA: Harvard Education Press.

29.

Graham

(2010). Hearing the voices of general staff: A Delphi study of the contributions of general staff to student outcomes. Journal of Higher Education Policy and Management, 32, 213-223.

30.

Hargreaves

D. H.

(1999). Revitalising educational research: Lessons from the past and proposals for the future. Cambridge Journal of Education, 29, 239-249.

31.

Hasson

Keeney

McKenna

(2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015.

32.

Helmer

(1977). Problems in futures research: Delphi and causal cross-impact analysis. Futures, February, 1977, 17-31.

33.

Hendrickson

R. M.

Lane

J. E.

Harris

J. T.

Dorman

R. H.

(2013). Academic leadership and governance of higher education: A guide for trustees, leaders, and aspiring leaders of two- and four-year institutions. Sterling, VA: Stylus.

34.

Henig

J. R.

(2008). Spin cycle. How research is used in policy debates: The case of charter schools. New York, NY: Russell Sage Foundation and Century Foundation.

35.

Herrington

C. C.

Gadsden

V. L.

(2013). Inaugural editorial. Educational Researcher, 42(7), 7-8.

36.

Hill

(1996). Education: How research affects policy and how both affect education. In Galster

(Ed.), Reality and research: Social science and U.S. urban policy since 1960 (pp. 131-155). Washington, DC: Urban Institute Press.

37.

Hirsch

(2000). Practitioners as researchers: Bridging theory and practice. New Directions for Higher Education, 2000(110), 99-106.

38.

Holden

M. C.

Wedman

J. F.

(1993). Future issues of computer mediated communications: The results of a Delphi study. Educational Technology Research and Development, 41(4), 5-24.

39.

Hsu

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.

40.

Hussler

Muller

Ronde

(2011). Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technological Forecasting & Social Change, 78(9), 1642-1653.

41.

Judd

R. C.

(1972). Use of Delphi methods in higher education. Technological Forecasting and Social Change, 4, 173-186.

42.

Kaestle

(1993). The awful reputation of education research. Educational Researcher, 22(1), 23-31.

43.

Keeney

Hasson

McKenna

H. P.

(2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38, 195-200.

44.

Keller

(1985). Trees without fruit: The problem with research about higher education. Change: The Magazine of Higher Learning, 17(1), 7-10.

45.

Kennedy

(1999). Infusing educational decision making with research. In Cizek

G. J.

(Ed.), Handbook of Educational Policy (pp. 54-76). New York: Academic Press.

46.

Kezar

(2000a). Still trees without fruit? Higher education research at the millennium. The Review of Higher Education, 23, 443-468.

47.

Kezar

(2000b). Understanding the research-to-practice gap: A national study of researchers’ and practitioners’ perspectives. New Directions for Higher Education, 2000(110), 9-19.

48.

Kezar

Eckel

(Eds.) (2000). Moving beyond the gap between research and practice in higher education. San Francisco: Jossey-Bass.

49.

Kezar

Maxey

(2014). The Delphi technique: An untapped approach of participatory research. International Journal of Social Science Research Methods, 2014, 1-18.

50.

Koster

Brekelmans

Korthagen

Wubbels

(2005). Quality requirements for teacher educators. Teaching and Teacher Education, 21, 157-176.

51.

Labaree

D. F.

(1998). Educational researchers: Living with a lesser form of knowledge. Educational Researcher, 27(8), 4-12.

52.

Labaree

D. F.

(2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(4), 13-22.

53.

Labaree

D. F.

(2004). The trouble with ed schools. New Haven, CT: Yale University Press.

54.

Lagemann

E. C.

(1999). An auspicious moment for education research. In Lagemann

Shulman

L. S.

(Eds.), Issues in education research (pp. 17-41). San Francisco, CA: Jossey-Bass.

55.

Landeta

(2006). Current validity of the Delphi method in social sciences. Technological Forecasting & Social Change, 73, 467-482.

56.

Levin

(2004). Making research matter more. Educational Policy Analysis Archives, 12(6), 1-20.

57.

Lincoln

Y. S.

Guba

E. G.

(2000). Paradigmatic controversies, contradictions, and emerging influences. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (2nd ed., pp. 163-186). Thousand Oaks, CA: Sage.

58.

Linstone

H. A.

Turoff

(1975). The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley.

59.

F. J.

Hsu

Chan

Cheen

Kao

(2012). Assessing college student-athletes’ life stress: Initial measurement development and validation. Measurement in Physical Education and Exercise Science, 16, 254-267.

60.

Makridakis

S. G.

Wheelright

S. C.

(1978). Interactive Forcasting: Univariate and Multivariate Methods. Holden-Day, Incorporated.

61.

Manley

R. A.

(2011). The Decentralization of Perkins: History, impact, and recommendations for future CTE legislation. Career and Technical Education Research, 36, 119-152.

62.

Marshall

(2014). Exploring the ethical implications of MOOCs. Distance Education, 35, 250-262.

63.

McNiff

(2013). Action research: Principles and practice. New York, NY: Routledge.

64.

Millar

Thorstensen

Tomkins

Mepham

Kaiser

(2006). Developing the ethical Delphi. Journal of Agricultural and Environmental Ethics, 20, 53-63.

65.

Nworie

(2011). Using the Delphi technique in educational research. TechTrends, 55(5), 24-30.

66.

Pill

(1971). The Delphi method: Substance, context, a critique, and an annotated bibliography. Socio-Economic Planning Sciences, 5(1), 57-71.

67.

Plank

D. N.

(2011). Minding the gap between research and policy making. In Conrad

C. F.

Serlin

R. C.

(Eds.), The SAGE handbook for research in education: Pursuing ideas as the keystone of exemplary inquiry (pp. 43-58). Los Angeles, CA: Sage.

68.

Plinske

Packard

W. J.

(2010). Trustees’ perceptions of the desired qualifications for the next generation of community college presidents. Community College Review, 37, 291-312.

69.

Pollard

(2004). Research priorities in educational technology: A Delphi study. Journal of Research on Technology in Education, 37, 145-160.

70.

Powell

(2003). The Delphi technique: myths and realities. Journal of Advanced Nursing, 41(4), 376-382.

71.

Ramaley

J. A.

(2006). Governance in a time of transition. In Tierney

W. G.

(Ed.), Governance and the public good (pp. 157-177). Albany: SUNY Press.

72.

Reason

(1994). Three approaches to participative inquiry. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (pp. 324-339). Thousand Oaks, CA: Sage.

73.

Reeves

Jauch

L. R.

(1978). Curriculum development through Delphi. Research in Higher Education, 8, 157-168.

74.

Robinson

V. M.

(1998). Methodology and the research–practice gap. Educational Researcher, 27(1), 17-26.

75.

Rossouw

Hacker

de Vries

M. J.

(2011). Concepts and contexts in engineering and technology education: An international and interdisciplinary Delphi study. International Journal of Technology and Design Education, 21, 409-424.

76.

Rowe

(1994). Enhancing judgement and decision making: A critical and empirical investigation of the Delphi technique (Unpublished PhD thesis). University of Western England, Bristol, UK.

77.

Rowe

Wright

(1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353-375.

78.

Sackman

(1975). Delphi critique: Expert opinion, forecasting, and group process. Boston, MA: Lexington.

79.

Sadhasivam

(2014). Educational paradigm shift: Are we ready to adopt MOOC? International Journal of Emerging Technologies in Learning, 9(4), 50-55.

80.

Salkind

N. J.

(2010). Delphi technique. In Salkind

N. J.

(Ed.), Encyclopedia of research design (pp. 344-347). Thousand Oaks, CA: Sage.

81.

Sangrà

Vlachopoulos

Cabrera Lanzo

(2012). Building an inclusive definition of e-learning: An approach to the conceptual framework. The International Review of Research in Open and Distributed Learning, 13(2), 145-159.

82.

Schmidtke

(2010). Math and science instructor perceptions of their American Indian students at a sub-baccalaureate technical college: A Delphi study. Journal of Career and Technical Education, 25(2), 8-23.

83.

Shelton

(2010). A quality scorecard for the administration of online education programs: A Delphi study (Doctoral dissertation). University of Nebraska, Lincoln, NE.

84.

Shriberg

Bonner

Sarr

B. J.

Walker

A. M.

Hyland

Chester

(2008). Social justice through a school psychology lens: Definition and applications. School Psychology Review, 37, 453-468.

85.

Sroufe

G. E.

(1997). Improving the “awful reputation” of education research. Educational Researcher, 26(7), 26-28.

86.

Strauss

H. J.

Zeigler

L. H.

(1975). The Delphi technique and its uses in social science research. The Journal of Creative Behavior, 9, 253-259.

87.

Tierney

Rhoads

(1994). Enhancing promotions, tenure, and beyond (ASHE-ERIC Higher Education Report Series). San Francisco, CA: Jossey Bass.

88.

Tyack

Cuban

(1995). Tinkering toward utopia: A century of public school reform. Cambridge, MA: Harvard University Press.

89.

Udoh

I. A.

Mantell

J. E.

Sandfort

Eighmy

M. A.

(2009). Potential pathways to HIV/AIDS transmission in the Niger Delta of Nigeria: Poverty, migration, and commercial sex. AIDS Care, 21, 567-574.

90.

Vergari

(2007). The politics of charter schools. Educational Policy, 21, 15-39.

91.

Westerheijden

D. F.

Huisman

de Boer

(2004). Ground force does the Dutch higher education gardens. Three scenarios revisited. Policy Futures in Education, 2, 374-387.

92.

Wright

T. S. A.

(2007). Developing research priorities with a cohort of higher education for sustainability experts. Journal of Sustainability in Higher Education, 8, 34-43.

93.

Zemsky

(2009). Making reform work: The case for transforming American higher education. New Brunswick, NJ: Rutgers University Press.