Scientific-Based Research and Randomized Controlled Trials,the “Gold” Standard? Alternative Paradigms and Mixed Methodologies

Abstract

This article addresses three controversial issues related to mixed methods research and policy. First, “Scientific-Based Research” promoted by “No Child Left Behind” (NCLB) reinforces diametrically opposed paradigmatic views and research methodologies. As policy, NCLB prioritizes specific methodologies prescribing what counts as scientific evidence. Second, from a critical stance, federal policies shape and control decisions that funding agencies make regarding methodologies (Randomized Controlled Trials—Gold Standard). Third, top-down policies are currently framed in postpositivist ontological and epistemological conceptions and should include constructivist, critical, transformative, and emancipatory paradigms supporting alternative methodologies. This article challenges current practices of prioritizing specific research methodologies used to evaluate interventions. As an alternative, logical purpose statements and research questions should be the standard used to guide decisions about appropriate methodologies and procedures.

Keywords

randomized controlled trials critical realism pragmatism mixed methods paradigms

Introduction

This article addresses three controversial issues related to mixed methods research and policy. First, the No Child Left Behind (NCLB) act of 2001 promotes “Scientifically-Based Research” (SBR) that shapes the problems in education that are prioritized, the funding strands that are released, and the methodologies that are suggested in “Request for Proposals” (RFPs; cf. Feuer, Towne, & Shavelson, 2002; National Research Council, 2002). When the National Research Council defined the meaning of SBR, it prioritized specific methodologies (Creswell & Plano Clark, 2011; Teddlie & Tashakkori, 2009), methods (Gorard, 2010), and prescribed what counts as scientific evidence (Feuer et al., 2002).

Second, from a critical/transformative stance (Creswell, 2009; Greene, 2007; Mertens, Bledoe, Sullivan, & Wilson, 2010), federal policies including NCLB are designed to shape and control decisions that funding agencies make in regard to prioritizing Randomized Controlled Trials (RCTs) as the “Gold Standard” in education-based research. RCTs, in essence, are studies that measure an intervention’s effect by randomly assigning individuals to an intervention or control group. This practice of promoting RCTs clearly delineates a shift in power from local and state level as was originally indicated when the U.S. Constitution was written to the federal government who create the policies that affect education practice (Alexander & Alexander, 2009).

Third, emerging controversies inherent in social science research have lead to a broader paradigmatic view inclusive of constructivist (Christ, 2013), critical-emancipatory (Mertens, 2010), and transformative (Greene, 2007) methodologies. Consequently, a case will be made for contesting current policies and practices that reinforce methodological singularity when conducting research.

Funding from federal agencies such as the Institute of Education Science (IES) and the Office of Special Education and Rehabilitation Services (OSERS) prioritize RCTs that may be appropriate for medical trials, large-scale efficacy, replication, and scale-up studies and are designed to “test” well-developed programs in multiple locations, but prioritizing specific methodologies regardless of the purpose of the research or the questions is inappropriate. From a critical stance, there is a great inequality when specific methodologies are promoted by grant funding agencies regardless of the intent of the research (Christ, 2012b). Despite efforts by the National Research Council (2002) to inform educational policy makers about the nature of scientific research and recommending the careful linkage of research questions and methodologies, review of federal grant RFP reveals the continued prevalence of methodology prioritization.

This article argues for alternatives most applicable to developmental-level funded research that, if published, can influence the way “think tank” members, research scholars, and political activists view what methodologies are appropriate when funding research. This article also challenges the unilateral prioritization of specific research methodologies by government agencies to evaluate interventions, teaching praxis, and student performance. As an alternative, paradigms other than postpositivism should be considered and combined with logical well-defined problem and purpose statements linked to research questions when guiding decisions about appropriate methodologies and methods (Christ, 2007; Creswell & Plano Clark, 2011; Maxwell, 2012; Teddlie & Tashakkori, 2009).

American pragmatism (Johnson & Gray, 2010) and critical realism (Christ, 2010; Maxwell, 2011; Maxwell & Mittapalli, 2010) are two theoretical paradigms that would be quite useful as a way to conduct “scientific” education-oriented intervention and policy research. A case will be made that policy makers and reviewers of funded research proposals should be aware of alternative paradigmatic stances and methodologies, and the inherent value they bring to social science research.

Mixed methodologies (Chatterji, 2005; Creswell, 2009; Greene, 2008; Teddlie & Tashakkori, 2009) including action-oriented research (Billies, Francisco, Krueger, & Linville, 2010; Christ, 2010) are particularly useful for studying interventions, teaching praxis, student, and policy performance. This article first defines and focuses on the use of mixed methodologies that blend and merge multiple forms of data to gain a deeper understanding of the strengths of relationships demanded of funded research, and then a justification is given of the value inherent in “how” and “why” explanations often left out of SBR.

Introducing pragmatism and critical realism as potential research paradigms presents a viable alternative to narrow postpositivist RCT methodologies and broadens the options available to address research questions and the accompanying methods that policy researchers and funding agencies demand. Establishing this argument first requires defining postpositivism, a metatheoretical stance that seeks to verify or refute preexisting theories. Those who ascribe to the postpositivist paradigm recognize that background, values, and knowledge influence research. Karl Popper’s views were the foundation for postpositivism advancing the concept of falsification rather than relying only on verification. Thomas Kuhn’s “worldviews” as first indicated in “Structures of Scientific Revolution” (Kuhn, 1962) also promoted a shift in response to evidence that set the stage for modern beliefs about postpositivist epistemology. Postpositivists continue to pursue and value objectivity, but they recognizing that bias is ever present and knowledge is fallible; thus, it can be modified or withdrawn when sound evidence emerges and is verified probabilistically.

SBR and RCT, the “Gold Standard”

RCTs as a methodology are predominantly required by funding agencies when conducting large-scale confirmatory research projects (Hesse-Biber, 2012; Plano Clark, 2010). Experimental research designs and lately RCTs have been labeled the “Gold Standard,” although this claim is repeatedly challenged in the literature (Cartwright, 2007; Denzin, 2008; Scriven, 2008). SBR (Denzin, 2008) and RCTs (Scriven, 2008) continue to dominate large-scale federally funded grants and are seen as the preferable means of establishing intervention impact (Brady & O’Regan, 2009) according to the governing bodies who determine funding. RCTs are difficult to conduct, expensive to implement, and may not be suitably matched to the identified problem or purpose of the research. The claim that RCTs are the “Gold Standard” rests on deductive assumptions and accompanying methods that require stringent internal validity standards. If any of the assumptions of a preconceived RCT are compromised, positive results that indicate a causal conclusion must be questioned (Brady & O’Regan, 2009). Clearly, deductive methods come at a great cost; RCTs inherently narrow the project scope and reduce many aspects of what can be learned. Fidelity of implementation requirements also precludes the opportunity to modify interventions, research questions, or procedures (Christ, 2007, 2013) once the study is underway even if warranted as it would violate the research design. In terms of research conducted in the field of education used to influence policy makers and government funding agencies, the reliance upon RCTs provides a distorted view of what is occurring. This is due to limited information that can be presented about specified variables which by the nature of studying humans as they interact with their environment are neither stable, free from influences, or confounding variables.

Shortcomings of SBR and RCT

Philosophical, methodological, and ethical considerations have impact upon all stages of “SBR.” More specifically, paradigmatic, methodological, and ethical considerations influence the conduct of RCTs at every stage from planning, implementing, evaluating, and disseminating results designed to indicate the strength of causal relationships. The challenge faced by those interested in promoting SBR and RCT is that unless tightly controlled experimental conditions are put into place, human-based research rarely follows preordained paths. Furthermore, when placed into experimental conditions, humans rarely act as they normally would. In terms of deductive research and RCT designs, anomalies in intervention administration or working with humans in natural conditions often introduces “confounding variables” and “error.” Creating a priori research questions and following stringent implementation and analysis procedures might reduce error, but severely limits knowledge gained to specific “operationally defined” measurable variables. Unless the intent of the research project is a large-scale replication study, the methods associated with RCT are quite limiting at best and inappropriate at worst.

Constructivist Research and the RCT

When qualitative data are collected and compared with quantitative data, the term “triangulation” is often cited (Denzin, 1970, 2012; Greene, 2007; Teddlie & Tashakkori, 2009). Flick (2002) first voiced concern that “triangulation is not a tool or strategy of validation but an alternative to validation” (p. 227) that is best “understood as a strategy which adds rigor, breadth, complexity, richness” (p. 229). Themes that emerge from qualitative data do not always match, and may contradict statistical results that bring into question the epistemological soundness of RCT as the “Gold Standard” (Creswell, 2009). Dahlberg, Wittink, and Gallo (2010) also recently questioned the absolute stance of a priori postpositivist-oriented research questions citing that they make experimental designs inflexible. Knowing if an intervention is statistically different from a control group is no more important than understanding the qualities, usefulness, and challenges inherent in the intervention to the participants in the study. Constructivist-oriented research used to analyze various forms of qualitative data allows the voice and experiences of the participants to compliment an understanding of the effectiveness of the intervention. Traditional RCTs often leave out information that is not directly related to answering preconceived research hypothesis that considerably limits the value of an often expensive and time-consuming analysis of human interactions.

Brady and O’Regan (2009) succinctly stated that “at a fundamental level, the application of postpositivist laboratory experimental design to the field of social research is criticized on the basis of its incompatibility to the open complex reality that is the social world” (p. 269). Christ (2010), Teddlie and Tashakkori (2009), Greene (2007), and Creswell (2009) have all questioned the concept of determining absolutes or the ability to separate facts from values. Johnson and Gray (2010), Biesta (2010), and Maxwell and Mittapalli (2010) also address epistemological issues in their perspectives related to the nature of how knowledge is produced that challenges the wisdom of relying upon numerical data statistically analyzed as the primary source of answering research hypothesis so prominent in funded research. Brady and O’Regan first pointed out the questionable external validity of RCT as participants cannot be randomly selected from an entire population as is the intent of a true experimental design. Those who participate in RCT are randomly assigned for intervention inclusion calling into question how representative this subpopulation is of the wider sample, which in turn limits the degree, to which findings can be generalized.

RCTs that are funded often rely heavily upon observations and survey data that may not capture intended constructs or adequately measure intervention outcomes. Validated instruments, common to RCT interventions, often are administered to a population dissimilar for which the measure was norm referenced. Ethical issues associated with RCTs also present an axiological challenge as depriving a control group of the known benefits of an intervention is clearly inappropriate. Brady and O’Regan (2009) also bring to light numerous technical issues inherent in RCT research including how well the intervention has been developed, if the intervention will change over the course of the study, statistical equivalency of the groups, length of time necessary to determine if changes have occurred, and attrition. With these and other challenges facing those who conduct RCT, why does it continue to hold a “Gold Standard” in terms of funded research? Clearly, the IES (2009) promotes RCTs, especially under goal 3 efficacy and replication projects:

Studies using random assignment to intervention and comparison conditions have the strongest internal validity for causal conclusions and thus are preferred whenever they are feasible. When a randomized trial is used, the applicant should clearly state the unit of randomization; choice of randomizing unit or units should be grounded in a theoretical framework. Applicants should explain the procedures for assignment of groups or participants to intervention and comparison conditions. (p. 68)

Are there viable alternatives to how RCTs are commonly conducted that improves, for example, social validity (Schwartz & Baer, 1991), and how useful are the results for intended stakeholders? Furthermore, how does one adequately address concerns about internal validity for causal conclusions (Chatterji, 2005) attempting to demonstrate that a program or treatment worked?

Using mixed methods when conducting RCTs may offer a way to broaden the knowledge gained when conducting intervention research. By expanding the focus to include how the intervention functions and if it is accepted, tolerated, or rejected by the participants goes well beyond the purpose of determining if the random assignment to intervention and comparison condition indicates internal validity for causal conclusions. Determining if a statistically significant difference exists on specified constructs for intervention and control groups, and if fidelity of implementation was acceptable, understandably, is of concern for funding agencies, but these findings might not address justifiable concerns stakeholders possess such as if the intervention is user friendly, maintains over time, or is accepted by practitioners. Using mixed methods in RCTs allows for collecting and analyzing a variety of data sources resulting in inferences that are applicable to a much broader spectrum of stakeholders. Simply stated, mixed methodologies framed in alternative or blended paradigms make good sense. Unfortunately, many who determine which form of research is funded do not appear to grasp that paradigms other than postpositivism are useful.

Alternative Paradigms: Pragmatism and Critical Realism

SBR and more specifically RCTs have their roots in a postpositivist paradigmatic stance. This stance promotes the idea that human knowledge is never infallible as it is based on human conjectures and unprovable assumptions. As knowledge is based on assumptions that change and are modified when other evidence emerges, the knowledge that is created is never perfect. Yet, postpositivism is not synonymous with relativism (points of view have no absolute truths or validity) and retains the concept that objective truths exist, although imperfect. Kuhn (1962) first indicated the idea that individual theories are incomplete, and a better way to understand how knowledge is created, is to recognize that worldviews are influenced by social conditions. Knowledge created shifts, often in response to what is considered acceptable evidence at the time. Postpositivism embraces the use of a “scientific method” and striving for objective truth, often through the use of experimental methodologies, and it also incorporates many basic assumptions of positivism including ontological realism (beliefs are approximations of reality and new observations add to understanding what is reality).

What makes little sense when examining the logic behind policy makers and funding agencies preference for a single paradigm and methodological approach is that it does not coincide with emerging views that alternative paradigms and research methodologies are appropriate for social science research. Simply stated, a singular or preferential view about the prominence of a particular research methodology restricts how knowledge is created and used to promote improvements to social conditions including education processes. For this reason, expressing how alternative paradigms and methodologies are useful when studying humans in their local conditions might be one of the few steps that productively can inform those in charge of funding strands.

Alternative paradigms, or what some call the researchers’ “worldview” (Christ, 2009; Onwuegbuzie & Combs, 2010), should not be so rigid as to restrict how research is conducted. Labels and categories including “paradigms” may be useful tools for communication, but when these descriptions are conceived as rigid and “law like,” they have a tendency to promote an either–or stance. Mixed Methods authors (Biesta, 2010; Christ, 2013; Creswell & Plano Clark, 2011; Greene, 2008; Maxwell & Mittapalli, 2010; Mertens, 2007; Teddlie & Tashakkori, 2009) have all promoted pragmatism (Greene & Hall, 2010) as a viable alternative to the narrow postpositivism constructivism dichotomy. Critical realism (Christ, 2010; Lipscomb, 2010; Maxwell, 2011; Maxwell & Mittapalli, 2010) has also been proposed as a viable paradigmatic alternative especially when used to conduct intervention-oriented research.

Paradigms as interpretations are intrinsically tied to the researchers’ worldview that influences how research is conducted. Unfortunately, with the current climate of accountability, funding agencies that release RFPs expect researchers to demonstrate causal relationships that have a tendency to restrict how research is planned. The challenge researchers face when applying for grants is to allow the purpose of the research and accompanying questions to determine the methodology and procedures. Researchers should be aware that alternative paradigms including pragmatism and critical realism are available and can be used when demonstrating causal relationships.

Clearly, methodologies are fluid and ever changing, and only the labels are rigid, similar to the tension concerning paradigms. Still, policy makers and funding agencies continue to promote an archaic dichotomous stance with preference for specific methodologies. For example, pragmatism and critical realism have much in common with postpositivism when considering the intent of demonstrating causality and the understanding that there are imperfect levels of truth that help determine how knowledge is created and judged (Christ, 2010, 2013). One only needs to consider Frauley and Pearce’s (2007) definition of critical realism to see similarities in pragmatism and postpositivism. Critical realism, as they indicate, have many levels of objective truths, and different levels of reality exist ranging from the objective, independent of human understanding, to subjective truths, that we understand in the process of meaning making and interpretations of causality (Goff, 2004).

Pragmatism, critical realism, and postpositivism as paradigms have in common the belief that there are levels of truths that can be discerned, but finding absolute truths about social phenomenon is impossible (see Table 1). Some proponents of critical realism promote interpretations of “subjective” and “objective” data combined using “abduction” and “retroduction” to formulate “conceptualizations” (Sayer, 1992, p. 50). When considering funded research project methodologies, the practice of analyzing data to demonstrate causality can be conducted in other ways than through statistical means. Quoting Maxwell (2012):

Table 1.

Worldview Matrix.

Worldview	Postpositivism	Critical realism	Pragmatism	Constructivism	Transformative/action
Ontology: (Reality) is what “exists” and ways in which it can be represented.	Etic-outsider singular reality exists although imperfectly represented in terms of probabilities.	Different levels of reality exist ranging from objective that is independent of human understanding to subjective truths that we understand and grasp in the process of meaning making.	Multiple forms of reality. Follows a Deweyan “What works” action-oriented view of reality.	Emic-co-construction of meaning. Each person’s reality is different. No absolute truths.	Inequalities exist. Exposing power structures and bringing the voice of the oppressed to light can lead to social change.
Epistemology: (Knowledge-theory) constructed using various sources of data. Analysis depends on the research question, data sources, and purpose of the study.	Researcher distances self from “subjects” to gain an “accurate,” “valid,” and “reliable” representation.	There are levels of objective truths that can be discerned, but finding absolute truths about social phenomenon is impossible.	Etic- and emic-perceptions can coexist in one study. Both data strands form knowledge to be compared and combined increasing the credibility.	Co-construction of knowledge about events occurs as a result of closeness: Researcher and participants work together to create knowledge.	Exposing inequalities and representing the voice of the oppressed to the masses broadens social awareness. Participants are collaborators.
Axiology: (Values in research) How the role of values influences the way research is conducted.	Researcher attempts to be unbiased and not introduce own values/prior experiences about when conducting and analyzing data.	Researchers’ worldview influences how knowledge is produced, differentiated, stratified, and changed into meanings and representative realities inferred.	Multiple stances: Values at the forefront and recognized as influencing the research process.	Researcher and participants recognize bias and negotiate their shared interpretations and their views about the value of the research process.	Oppression exists in the world, and the role of research should be to work toward increasing social justice.
Methodology: How the processes of research are used.	Deductive approach: Test and verify a priori theories. Determine significant difference among groups or strength in relationships among variables.	Construction and interpretation of “subjective” and “objective” data, processes are called “abduction” and “retroduction” that are used to formulate “conceptualizations.”	Mixed approach: Various forms of qualitative and quantitative data are blended to create a representative model.	Inductive approach: Researcher uses constructivist grounded data analysis approaches to building patterns, themes, and general concepts.	Participatory- and action-oriented research is designed to enhance individual, social, and societal well-being.

Source. Adapted from Christ (2013).

Both regularity/variance theory and process theory are forms of causal explanation.

Process theory is not merely “descriptive,” as opposed to “explanatory” variance theory;

it is a different approach to explanation. Variance theory typically involves a “black box”

approach to the problem of causality in the social sciences. Lacking direct access to

social and cognitive processes, researchers must attempt to correlate differences in output

with differences in input, and control for other plausible factors that might affect the

output. In contrast, process theory, which is much more suitable for qualitative methods,

can often directly investigate these causal processes through observation of social settings

and interviews with participants. (p. 37)

For example, interviews and observations can be coded for themes (Charmaz, 2006; Glaser, 1978; Weiss, 1994), triangulated (Denzin, 2012; Torrance, 2012), reflected, and merged with numerical data when making inferences that are the “conclusions or interpretations drawn from the separate quantitative and qualitative strands of a study” (Creswell & Plano Clark, 2011, p. 412). Unfortunately, as Maxwell indicates, RCTs are based on the premise of a regularity theory of causality that ignores process (Maxwell, 2004).

Bhaskar’s Critical Realism as promoted by Danermark, Ekström, Jakobsen, and Karlsson (2002); Sayer (1992); and Frauley and Pearce (2007) relies on three forms of knowledge, the “empirical” that is directly observable actions, the “actual,” which is the sum of inferences that when combined produces knowledge that is said to be the “real” not the directly observed or theorized, but what helps explain the structures and processes of making knowledge. Critical realists and pragmatist proponents agree that combining multiple sources of data influences how knowledge is produced, differentiated, stratified, and changed into meanings. This seems to be the main area where disagreements emerge with those who promote postpositivism.

The incompatibility thesis, what Rossman and Wilson (1985) deemed a purist view since the late 1980s, stipulates postpositivism and constuctivism are inherently dissimilar in their ontological, epistemological, and axiological stances; therefore, knowledge gained from each strand is essentially incompatible. Although rigid interpretations of preferential methodologies have faded in many of the social sciences, policies such as NCLB that promote SBR keep the incompatibility thesis alive and well in the world of federally funded research.

One viable way to alleviate this practice is to promote awareness of alternative paradigmatic views and methodological practices that meet the stringent requirements of funding agencies RFPs, yet provide researchers the flexibility required to choose an optimal methodological design based on the purpose and research question rather than the suggested designs often published in grant RFPs that limit design options of the project. Two such research approaches are particularly attractive. They are mixed methods and action research designs.

Mixed Methodologies

Tashakkori and Creswell (2007) defined mixed methods in their first editorial of the Journal of Mixed Methods Research (JMMR) as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry” (p. 4). This approach is particularly attractive when designing intervention studies for funded grants or conducting large-scale program evaluations that prioritize regularity/variance theory as the way to demonstrate causal inferences as both forms of data can address different questions. Recently, mixed methods publications now address combining multiple paradigms (e.g., pragmatism, transformative, critical realism, and dialecticalism) in studies as a way to address issues, including knowledge, epistemic and social values, and causality (Christ et al., 2011). The use of mixed methods as a research design has potential to change the narrow interpretation of optimal methodologies prevalent in funded grant research. One way to promote change in how policy makers and funding agencies write RFPs, and how reviewers rate applications, is to challenge epistemological dualism (objectivism/positivism and subjectivism/constructivism) and engage in “paradigm dialog” about the value of blending deductive and inductive research in a complimentary strength mixed methods stance (Johnson & Onwuegbuzie, 2004; Teddlie & Tashakkori, 2009). Using mixed methods in confirmatory research designs as required in most federally funded grants allows the expansion of knowledge about interventions. For example, various quantitative approaches can be used to determine strength of relationships between variables, significant group differences, whether an intervention demonstrated performance change, and whether the intervention was administered with adequate fidelity (Abry, Rimm-Kaufman, Larsen, & Brewer, 2011). Qualitative approaches are particularly effective at addressing how participants value interventions, if they are viable, useful, and economically feasible.

Action research can be one form of mixed methods (Christ, 2010; Christ & Elmetaher, 2012) that is a particularly useful alternative in development grants to RCTs where stringent control over environmental factors is virtually impossible (Christ, 2012a). Kemmis and McTaggart’s (1988) definition of action research highlights advantages of this methodology for use in education:

a form of collaborative self-reflective enquiry undertaken by participants in social situations in order to improve the rationality and justice of their own social or educational practices, as well as their understanding of these practices and the situations in which these practices are carried out. (p. 1)

Kurt Lewin (1946/1948) first indicated that Action Research “proceeds in a spiral of steps, each of which is composed of a circle of planning, action, and fact finding about the results of the action” (p. 206). This definition appears to have been influenced by Dewey’s theory of knowledge, concepts of indefinite interactions, knowing as the mode of experience, and the relationship between actions and consequences. In particular, replication of small-scale social science research studies can lead to valid evidence concerning outcomes of intervention research.

Conclusion

Philosophical, methodological, and ethical considerations have impact upon all stages of RCTs, including planning, implementing, evaluating, and disseminating results that indicate a causal relationship. Unless tightly controlled experimental conditions are put into place, human research rarely follows preordained paths as planned. Furthermore, when placed into experimental conditions, humans rarely act as they normally would. In terms of deductive research and RCT designs, anomalies in intervention administration or working with humans in natural conditions often introduces “confounding variables” and “error.” Creating a priori research questions and following stringent implementation and analysis procedures may reduce error, but environmental constraints often change the behaviors humans would normally exhibit. Fortunately, blending deductive and inductive research in a single study, a complimentary strength mixed methods stance (Teddlie & Tashakkori, 2009) can offset the error inherent in RCT findings and expand the ability to impart knowledge by collecting and analyzing multiple forms of data to answer emergent research questions (Christ, 2007).

Epistemological and axiological considerations, an etic-outsiders perspective that is value free, as the basis for the original logic behind promoting deductive RCT research designs have changed. The very nature of funding agencies promoting RCTs that prioritizes an etic-research perspective, and quantitative rather than qualitative methods as promoted by Weisner (2005), is counterintuitive to the nature of the study of human behavior. RCTs as promoted by organizations such as the IES are seen as most successful when based upon well-developed theories that dictate operationally defined constructs and carefully articulated intervention components that use valid and reliable measures that adequately capture the constructs that they purport to measure. The intervention components and accompanying measures, according to IES criteria, should be administered in a controlled environment to reduce the possible introduction of unwarranted confounding variables that contaminate findings. Sample and effect size must also be calculated, taking into consideration participant attrition. Unexpected changes in the intervention components or participant attrition reduce the likelihood that if change was detected, it was not the result of a Type I or Type II error or confounding variables.

Even when a statistically significant difference is noted through appropriate statistical analysis, the majority of published RCTs primarily answer if there was a significant group difference or the strength of the relationship, but do not show how or why the intervention affected change in the participants, nor if the intervention was applicable, or desirable to the stakeholders. To understand the potential for future applicability of an intervention, an emic-insiders perspective should be incorporated with the traditional etic-outsider approach common to RCT intervention evaluations. Combined, the deductive and inductive research approaches provide a more complete and holistic understanding of the strengths and challenges associated with human intervention research. The only way that policy makers and funding agencies will change their rigid interpretations of what constitutes evidence will be when numerous sound intervention studies that incorporate alternative methodologies and paradigms are successfully published in reputable journals. This will result in changes to the way standards and policies are created, and methodologies are used in the social sciences. These changes may be slow to take hold, but one only has to look back two decades to see how research methodology used to study human behavior has vastly diversified.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Thomas W. Christ, PhD, chairs the Education Leadership doctoral program at the University of Bridgeport and the American Education Research Association Mixed Methods Special Interest Group. His current research focus includes mixed methods and action research pedagogy; student, program, and organization evaluations; international education reform; and curricula theory. His publications include mixed and action research methodologies, postsecondary leadership, disability support services, technology, and literacy program evaluations.

References

Abry

Rimm-Kaufman

Larsen

Brewer

(2011, September 9). Applying new methods to the measurement of fidelity of implementation: Examining the critical ingredients of the responsive classroom approach in relation to mathematics achievement. Paper presented at the Society for Research on Educational Effectiveness, Washington, DC.

Alexander

M. D.

(2009). American public school law. New York, NY: West Publishing Company.

Biesta

(2010). Pragmatism and the philosophical foundations of mixed methods research. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 95-118). Thousand Oaks, CA: SAGE.

Billies

Francisco

Krueger

Linville

(2010). Participatory action research: Our methodological roots. International Review of Qualitative Research, 3, 277-286.

Brady

O’Regan

(2009). Meeting the challenge of doing an RCT evaluation of youth mentoring in Ireland: A journey in mixed methods. Journal of Mixed Methods Research, 3, 265-280. doi:10.1177/1558689809335973

Cartwright

(2007). Hunting causes and using them: Approaches in philosophy and economics. New York, NY: Cambridge University Press.

Charmaz

(2006). Constructing grounded theory: A practical guide through qualitative analysis. Thousand Oaks, CA: SAGE.

Chatterji

(2005). Evidence on “what works”: An argument for extended-term mixed-method (ETMM) evaluation designs. Educational Researcher, 34(5), 14-24. doi:10.3102/0013189X034005014

Christ

(2007). A recursive approach to mixed methods research in a longitudinal study of postsecondary education disability support services. Journal of Mixed Methods Research, 1, 226-241. doi:10.1177/1558689807301101

10.

Christ

(2009). Designing, teaching and evaluating two complementary mixed methods research courses. Journal of Mixed Methods Research, 3, 292-325. doi:10.1177/1558689809341796

11.

Christ

(2010). Teaching mixed methods and action research: Pedagogical, practical and evaluative considerations. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 643-676). Thousand Oaks, CA: SAGE.

12.

Christ

(2012a, April 14). Grant funding strategies for action research designs. Paper presented at American Education Research Association, Vancouver, British Columbia, Canada.

13.

Christ

(2012b, April 16). Policy, scientifically based research, and randomized control trials: The “Gold Standard?” Paper presented at American Education Research Association, Vancouver, British Columbia, Canada.

14.

Christ

(2013). The worldview matrix as a strategy when designing mixed methods research. International Journal of Multiple Research Approaches, 7, 86-94. doi:10.5172/mra.2013.2458

15.

Christ

Elmetaher

(2012). Research teaching pedagogy: Lessons learned from an Arabic language intervention pilot study. Research in the Schools, 19, 62-75.

16.

Christ

Greene

Johnson

R. B.

Lipscomb

Maxwell

Tashakkori

. . .Onwuegbuzie

(2011, April 11).Evolving paradigms in mixed methods research. American Education Research Association Symposium, New Orleans, LA.

17.

Creswell

(2009). Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3, 95-108. doi:10.1177/1558689808330883

18.

Creswell

Plano Clark

(2011). Designing and conducting mixed methods Research (2nd ed.). Thousand Oaks, CA: SAGE.

19.

Dahlberg

Wittink

Gallo

(2010). Funding and publishing integrated studies: Writing effective mixed methods manuscripts and grant proposals. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 775-802). Thousand Oaks, CA: SAGE.

20.

Danermark

Ekström

Jakobsen

Karlsson

J. C.

(2002). Explaining society: Critical realism in the social sciences. London, England: Routledge.

21.

Denzin

(1970). The research act. Chicago, IL: Aldine.

22.

Denzin

(2008). The new paradigm dialogs and qualitative inquiry. International Journal of Qualitative Studies in Education, 21, 315-325. doi:10.1080/09518390802136995

23.

Denzin

(2012). Triangulation 2.0. Journal of Mixed Methods Research, 6, 80-88. doi:10.1177/1558689812437186

24.

Feuer

M. J.

Towne

Shavelson

R. J.

(2002, November). Scientific culture and educational research. Educational Researcher, 31(8), 4-14. doi:10.3102/0013189X031008004

25.

Flick

(2002). An introduction to qualitative research. Thousand Oaks, CA: SAGE.

26.

Frauley

Pearce

(Eds.). (2007). Critical realism and the social sciences: Heterodox elaborations. Toronto, Ontario, Canada: University of Toronto Press.

27.

Glaser

(1978). Theoretical sensitivity: Advances in the methodology of grounded theory. Mill Valley, CA: Sociology Press.

28.

Goff

(2004). Critical realism, post-positivism and the possibility of knowledge. London, England: Routledge.

29.

Gorard

(2010). Research design, as independent of methods. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 237-252). Thousand Oaks, CA: SAGE.

30.

Greene

(2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

31.

Greene

(2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2(1), 7-22. doi:10.1177/1558689807309969

32.

Greene

Hall

(2010). Dialectics and pragmatism: Being of consequence. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 119-144). Thousand Oaks, CA: SAGE.

33.

Hesse-Biber

(2012, April 15). Enhancing the credibility of evidence based practice: Moving RCT towards a mixed methods design. AERA SIG Distinguished Business Meeting Speaker, Vancouver, British Columbia, Canada.

34.

Institute of Education Science. (2009). Request for applications: Education research grants CFDA Number: 84.305A (pp. 1-91). http://ies.ed.gov/funding/pdf/grantsgov_application.pdf

35.

Johnson

R. B.

Gray

(2010). A history of philosophical and theoretical issues for mixed methods research. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 69-94). Thousand Oaks, CA: SAGE.

36.

Johnson

R. B.

Onwuegbuzie

(2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. doi:10.3102/0013189X033007014

37.

Kemmis

McTaggart

(Eds.). (1988). The action research planner (3rd ed.). Geelong, Victoria, Australia: Deakin University Press.

38.

Kuhn

T. S.

(1962). The structures of scientific revolutions. Chicago, IL: University of Chicago Press.

39.

Lewin

(1948). Action research and minority problems. In Lewin

G. W.

(Ed.), Resolving social conflicts (pp. 201-216). New York: Harper & Row. (Original work published 1946)

40.

Lipscomb

(2010). Events and event identity: Under-explored topics in nursing. Nursing Philosophy, 11, 88-99. doi:10.1111/j.1466-769X.2010.00435.x

41.

Maxwell

J. A.

(2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3-11. doi:10.3102/0013189X033002003

42.

Maxwell

J. A.

(2011). A realist approach for qualitative research. Thousand Oaks, CA: SAGE.

43.

Maxwell

J. A.

(2012). Qualitative research design: An interactive approach (3rd ed.). Thousand Oaks, CA: SAGE.

44.

Maxwell

J. A.

Mittapalli

(2010). Realism as a stance for mixed method research. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 145-168). Thousand Oaks, CA: SAGE.

45.

Mertens

D. M.

(2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1, 212-225. doi:10.1177/1558689807302811

46.

Mertens

D. M.

(2010). Divergence and mixed methods. Journal of Mixed Methods Research, 4(1), 3-5. doi:10.1177/1558689809358406

47.

Mertens

D. M.

Bledoe

K. L.

Sullivan

Wilson

(2010). Utilization of mixed methods for transformative purposes. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 193-214). Thousand Oaks, CA: SAGE.

48.

National Research Council. (2002). Scientific research in education ( Shavelson

R. J.

Towne

, Eds.). Committee on Scientific Principles for Education Research. Washington, DC: National Academy Press.

49.

No Child Left Behind Act of 2001, Pub. L. No. 107-110.

50.

Onwuegbuzie

Combs

(2010). Emerging data analysis techniques in mixed methods research: A synthesis. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 397-430). Thousand Oaks, CA: SAGE.

51.

Plano Clark

. (2010). The adoption and practice of mixed methods: US trends in federally funded health-related research. Qualitative Inquiry, 16, 428-440. doi:10.1177/1077800410364609

52.

Rossman

G. B.

Wilson

B. L.

(1985). Numbers and words: Combing quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9, 627-643.

53.

Sayer

(1992). Method in social science: A realist approach. London, England: Routledge.

54.

Schwartz

Baer

(1991). Social validity assessment: Is current practice state of the art? Journal of Applied Behavior Analysis, 24, 189-204. doi:10.1901/jaba.1991.24-189

55.

Scriven

(2008). A summative evaluation of RCT methodology and an alternative approach to causal research. Journal of MultiDisciplinary Evaluation, 5(9), 11-24.

56.

Tashakkori

Creswell

(2007). Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research, 1, 207-211. doi:10.1177/1558689807302814

57.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: SAGE.

58.

Torrance

(2012). Triangulation, respondent validation, and democratic participation in mixed methods research. Journal of Mixed Methods Research, 6, 111-123. doi:10.1177/1558689812437185

59.

Weisner

T. S.

(Ed.). (2005). Discovering successful pathways in children’s development: Mixed methods in the study of childhood and family life. Chicago, IL: University of Chicago Press.

60.

Weiss

R. S.

(1994). Learning from strangers: The art and method of qualitative interviewing. New York, NY: Free Press.