Culturally Responsive Experimental Intervention Studies

Rationale for CRR

Researchers have recorded long-lasting disparities in academic, social, and economic opportunities and outcomes between nondominant youth from historically marginalized communities and their peers from dominant groups (e.g., Anyon, 2005; Darling-Hammond, 2010; Ferri & Connor, 2005; Losen & Orfield, 2002; Wagner, Newman, Cameto, & Levine, 2006). The U.S. Department of Education and Office for Civil Rights (2012) issued a report detailing how quality of educational opportunities were diminished based on race, gender, language, and disability status, thus contributing to the marginalization of nondominant communities. The findings of this report highlight several key affronts to equal access to education, including the ongoing obstacles to desegregate U.S. schools: limited access to college preparatory and advanced placement secondary courses, limited resources to instructional technology, and racial disproportionality in school disciplinary actions involving suspensions and expulsions (U.S. Department of Education & Office for Civil Rights, 2012).

For students who receive special education, these disparities are pronounced and complicated by disproportionate identification of disability and service delivery in restrictive settings. In what has been identified as a historical pattern, the most recent report to Congress on the implementation of the Individuals With Disabilities Education Improvement Act of 2004 documented an elevated risk of disability identification in what are considered judgmental disabilities such as learning disabilities (LDs) and BDs for Native American and African American students as compared to all other race/ethnicities combined. Furthermore, the risk of disability identification was 1.63 and 2.28 times higher, respectively, in the category of BD, a disability category with some of the lowest indicators of academic and postschool success, such as 56% rate of high school completion (U.S. Department of Education & Office of Special Education and Rehabilitation Services, 2011).

Additionally, minority students with disabilities have diminished access to inclusive education settings that are generally associated with positive outcomes, such as enrollment in postsecondary education (U.S. Department of Education & Office of Special Education and Rehabilitation Services, 2011). This problem is compounded by overrepresentation of Native Americans and African Americans when disability category is the focus of analysis because they are both overrepresented in BD. Only 35% of youth identified with BD are served in the most inclusive settings (i.e., spending 80% or more of the school day in a general education classroom). Fully 17% of youth identified with BD are in separate settings such as residential and treatment settings, the third highest percentage of students served in separate settings by disability category after students who are deaf and blind (30%) and those who have multiple disabilities (25%; U.S. Department of Education & Office of Special Education and Rehabilitation Services, 2011).

When students experience similar behavioral difficulties—with or without a disability label—those from nondominant backgrounds more often face exclusionary disciplinary actions. These outcomes also have a historically racialized presence in schools. Nearly 40 years ago, researchers found that African American students were up to 3 times more likely than their White peers to be suspended (Children’s Defense Fund, 1975). These disparities hold today, with African American, Latino, and Native American youth significantly more likely to experience exclusionary school discipline correlated with academic failure and involvement in juvenile justice system (Krezmien, Leone, & Achilles, 2006). Recent national data show that nondominant students are punished more often and more severely, for example, being subjected to mechanical restraints for less serious incidents, such as disrespectful behavior and dress code violations (for more detailed results, see Office for Civil Rights, 2012).

Demographic projections show that students from nondominant backgrounds will make up more than half the student population nationwide in the near future. Each year, millions of U.S.-born cultural and linguistic minority students, as well as newly arrived immigrant and refugee youth, enter school with diverse strengths, needs, interests, and experiences. These youth attend predominantly urban schools that significantly lack adequate academic, linguistic, and social supports and fall short of providing positive social climates (Anyon, 2005; Suárez-Orozco & Suárez-Orozco, 2001). There is a critical need to generate valid knowledge to understand these challenges to educational equity and to design culturally relevant, academically rich, and sustainable education systems that disrupt and transform the inherent conditions reproducing inequalities in the United States (Ladson-Billings & Tate, 2006; Paris, 2012).

The increasing diversity in schools, the ever-widening outcome and opportunity gaps, and the historical marginalization of nondominant communities constitute a systemic tension and requires a paradigm expansion for developing new theoretical models and intervention methodologies considering dynamic contexts of education systems and goals, practices, histories, and political interests of participants and researchers (Artiles, 2011; Darling-Hammond, 2010; Klingner et al., 2005; Ladson-Billings, 2006).

Marginalization of Nondominant Communities and the Role of Research

For too long, nondominant communities have been excluded from educational and psychology research (Graham, 1992). In a meta-analysis of the 180 intervention studies of students identified with LDs, Swanson and Hoskyn (1998) found that the majority of the intervention studies did not report participants’ race or ethnicity. In addition, the results are rarely disaggregated by race/ethnicity (Artiles, Trent, & Kuan, 1997; National Research Council, 2002a; Trainor & Bal, 2014). Students identified as English language learners, in particular, are often excluded from experimental studies to establish internal validity (Solano-Flores, 2008). Although this limits external validity or generalizability of the findings, the outcomes of intervention studies are often presented as objective or culture- or race-free and their results as evidence supporting practices for all students, including those from nondominant communities who were excluded from participation in research (Klingner et al., 2007).

When researchers focus on youth and families from nondominant backgrounds, representation of those participants is deficit in orientation. Deficit-oriented representations are associated with the study of participants’ sociodemographic background based on acontextualized, static individual characteristics (e.g., individuals’ social skills) or overly generalized categories and loosely identified group traits such as learning styles (e.g., racial learning style; Gutiérrez & Rogoff, 2003). In education research, nondominant youth and families have long been positioned as being at risk for academic failure and behavioral problems via theories such as the culture of poverty or eugenics (Baker, 2002; Erickson, 2009; Gieryn, 1995; Snyder & Mitchell, 2010). Furthermore, research methodologies and training are culturally responsive to dominant cultural group practices (e.g., White males with able bodies) and have inherited biases and prejudice toward nondominant groups including racial minorities, people with disabilities, people living in poverty, and immigrants (Scheurich & Young, 1997).

In psychology and education research, complex constructs such as race, disability, and class are treated as independent variables and innate characteristics of individuals—often via problematic proxy indicators such as free or reduced-price meals for socioeconomic status. The historical misuse of race to study the effect of race, for example, has been promulgated as an explanation and justification of the racial hierarchy via culturally biased tools (e.g., IQ tests or behavioral checklists) and often-inadequate statistical analyses (e.g., using single-level models that did not allow to examine the nested structure of student-level and school-level determinants; Bonilla-Silva & Zuberi, 2008).

The privileged use of culturally biased research tools as objective and culture-free is exemplified by the founding figures of psychology in the West. Francis Galton, G. Stanley Hall, and others who developed personal trait measures such as intelligence and personality tests established psychology as a field of individual difference. These influential figures offered self-governing, isolated individuals whose abilities and disabilities were mainly determined by hereditary characteristics (e.g., race) as the unit of analysis. They were also strong supporters of White supremacy and the eugenics movement, which sought to improve genetic quality and purity of a society (i.e., dominant group) through the practices of selective breeding and sterilization (Baker, 2002). These ideologies and interests were embedded in the foundational constructs (e.g., intelligence, self-esteem, self-determination, motivation, and personality traits), measurements, and knowledge hierarchies (e.g., statistical analyses of individual differences) and became the dominant paradigm of psychology and education in the United States. And eventually, these ideologies and tools created a well-oiled machine in formal schooling and in education research that identify and “fix” deficits within individual children (Erickson, 2009).

To additionally complicate those issues, academies of higher education, responsible for research and its dissemination for the purpose of solving education’s greatest challenges, reflect their own history of marginalization of scholars from nondominant groups (Acker, Webber, & Smyth, 2012; Barclay, 2012; Diggs, Garrison-Wade, Estrada, & Galindo, 2009; Lather, 2004). First, research-intensive universities continue to struggle to support and retain scholars from historically marginalized groups (e.g., scholars of color and disabled scholars), thus contributing to marginalization of the work of nondominant scholars (Alex-Assensoh, 2003). Second, nondominant scholars are likely to have heavy service loads and function as resources for their peers and students from similarly marginalized backgrounds, thus facing obstacles to tenure and promotion in institutions that value productivity over collegiality (Jayakumar, Howard, Allen, & Han, 2009). Third, nondominant scholars who choose to pursue lines of research that investigate culture, race, or racial inequalities are often viewed negatively in academic settings where dominant-group scholars perceive such scholarly pursuits as racially motivated, biased, and unscientific (Barclay, 2012; Fenelon, 2003). In particular, methods used to critique and research questions that focus on power/privilege may be infrequently employed by researchers from dominant groups and lead to questions about, or defense of, rigor during the implementation and dissemination of research (DeCuir & Dixson, 2004; Milner, 2007; Stanley, 2007).

In short, the marginalization of students, families, and researchers from nondominant groups contributes to the systemic tension in special education research and its inadequacy for addressing complex issues and inequalities, calling for paradigmatic expansion regarding the culture of knowledge production. Systemic tensions are a key to understand the sources of problems as well as the innovative and generative potential for systemic transformation in a scientific field (Engeström, 2008). The purpose of the CRR rubric is to expand current paradigms of knowledge-production and develop CRR, with a focus on experimental interventions because experimental design is privileged and touted as the most effective methodology of knowledge production in special education.

Paradigmatic Expansion

Kuhn (1962) noted, “Paradigms prove to be constitutive of science . . . [they] provide scientists not only with a map but also with some of the directions” (p. 22). According to Kuhn, progress in a scientific field takes place as dialectical episodes. The conceptualization of scientific truth is not universal but bounded by time and space. Scientific truth cannot be established solely by objective criteria; it is defined by a collective conceptual agreement of a given field as situated in specific social, historical, and spatial contexts (e.g., political interests; Gieryn, 1995). The relative continuity of foundational conceptual agreements in all scientific fields can be interrupted and challenged by fundamental changes in its content of the study, thus potentially creating systemic tensions (Kuhn, 1962). Each scientific discipline solves systemic tensions via paradigm expansion to establish a consensus about a more encompassing paradigm for understanding and addressing those tensions (Kuhn, 1962). For example, in the field of physics, moving from Newtonian mechanics to relativistic mechanics is an example of paradigm expansion. Such paradigm expansions require a discovery period and a revolutionary reconfiguration that lead to the expanded paradigm with an updated conceptual map directing new methodologies. Gieryn (1995) asserted expanded paradigms change the culture of scientific fields directing new inquiries and asking novel questions of existing data.

Experimental psychology, most specifically its behaviorist conceptualization, has been the main influence in the mainstream special education research paradigm in the United States (Kauffman & Landrum, 2006). This view takes the individual as the unit of analysis and looks for universal laws of human behaviors. Culture is often regarded as an extraneous variable associated with homogenous, static group characteristics (e.g., values, customs, or behaviors). Theories highlighting the complex nature of human learning and development and the role of culture in, for example, Bourdieu’s social reproduction theory (e.g., concept of habitus), Erikson’s developmental theory of identity, or Bronfenbrenner’s ecological systems theory have been adapted by social scientists in accordance with the unit of analysis: self-governing, morally directed, rational selves in a universal time and stable spaces (Popkewitz, 1997).

Criticism of narrowly focused epistemologies and ontologies in psychology and education research is not new. Some have argued the current experimental studies employing highly controlled laboratory-based experimental methodology (stimulus-response) are overly deterministic and not well suited to understanding higher psychological processes that involve reasoning, language, and goal-oriented actions (M. Cole, 1996). Although highly controlled laboratory-based research contributes to the knowledge base of elementary (physiological) psychological functions, a developmental-historical methodology such as ethnological analyses of cultural group practices is necessary for understanding the higher psychological functions. Wilhelm Wundt, widely recognized as the founding figure of experimental psychology, acknowledged this limitation on the field’s knowledge production, asserting, “Only through a synthesis of their respective insights could a full psychology be achieved” (M. Cole, 1996, p. 29). Against Wundt’s explicit warnings, however, proponents of experimental methodology complemented a narrow view of epistemology with behaviorism, inferential statistics, and standardized aptitude and achievement tests to study both elementary and higher psychological functions.

Alternative conceptualizations of experimental methodology focusing on cultural mediation and dynamic interactions of human and context include Vygotsky and his followers, who developed sociocultural theory, a historical materialist theory of learning, and experimental methodology aligned with Wundt’s earlier work and encompassed both the study of elementary and higher psychological functions. Vygotsky (1976) called this experimental methodology “the functional method of double stimulation” (p. 74). In double stimulation experiments, “The subject is put in a structured situation where a problem exists . . . and the subject is provided with active guidance towards the construction of a new means to the end of a solution to the problem” (van der Veer & Valsiner, 1991, p. 169). Sociocultural conceptualization of experimental psychology and its intervention methodology are increasingly used in education research, organizational studies, and learning sciences for designing expansive learning contexts inside and outside of schools (Sannino, Daniels, & Gutiérrez, 2009; for the use of the functional method of double stimulation in special education, see Bal, Kozleski, Schrader, Rodriguez, & Pelton, 2014).

An intended contribution of the CRR rubric as a research tool is to facilitate a paradigmatic expansion in special education interventions that involves examining the critical and generative role of culture and contexts to provide fuller and dynamic understanding of human learning, development, and ability. The focus on intervention experiments is deliberate because these designs are privileged and have a defined and peer-reviewed initial set of quality indicators. We use the term culturally responsive research to refer to scholarship that applies the tenets of cultural responsiveness in inquiry, communication, and interaction in education. Addressing persistent problems of equity requires critical attention to the cultural notion of knowledge production in the field of special education (Artiles, Kozleski, Trent, Osher, & Ortiz, 2010; McDermott & Varenne, 1995; National Research Council, 2002b).

The Cultural Practice of Knowledge Production

All stages of intervention research, from question inception to dissemination, are “culturally and socially mediated and negotiated” (Arzubiaga et al., 2008, p. 310). Research tools mediate scientific processes of knowledge production. As cultural artifacts, those tools are heavily instantiated with constantly shifting cultural practices, ideologies, identities, and interests that evolve over time and are negotiated and orchestrated in local sociopolitical contexts (Scheurich, 1997). To illustrate, with its dominant conceptualization, intervention itself has been used as a linear, ableist tool to fix differences constructed as deficits within an individual or a cultural group such as newly arrived immigrant families or deaf people (Snyder & Mitchell, 2010). Grounded in sociocultural theory, CRR takes intervention as a cultural practice that is open for negotiation and adaptation:

An on-going transformational process that is constantly re-shaped by its own internal organizational and political dynamic and by the specific conditions it encounters or itself creates, including the responses and strategies of local and regional groups who may struggle to define and defend their own social spaces, cultural boundaries and positions within the wider power field. (Long, 2001, as cited in Engeström, 2011, p. 603)

In our view, universalist, culture-free perspectives that do not critically consider disabled, racialized, gendered, or classed experiences and enduring structural inequalities are products of culture and history through local performances of researchers, participants, and consumers of research. The significance and implications of research results can be fully understood if and when the specific contexts of the researchers, the participants, and scientific fields frame the work. Conceptualizing research as situated cultural practice acknowledges the regimes of power/knowledge as central players in the reproduction of disparities in access, influence, and thus, predictable variance in outcomes (Leonardo, 2009; Soja, 2010; Young, 1990).

Research requires a constellation of analytic tools as cultural artifacts—both ideal and material—for examining the individual and social processes of learning and development, including the examination of disparate education opportunities and outcomes. Researchers risk contributing to the reproduction and obfuscation of structural inequalities when only certain methodological tools are promoted as the most effective way to answer all research questions or when the quality indicators of any given methodology fail to acknowledge research as a situated practice, and both may also lead to routinized implementation and dissemination decisions (Trainor & Bal, 2014). The establishment of experimental designs as a “gold standard” and the development and application of quality indicator rubrics in special education without attention to diversity and equity have overlooked and continue to overlook complicated research questions and contributed to the persistence of inequalities.

Gutiérrez (2006) used the term White innocence to explain how researchers, including those from nondominant communities, may reproduce dominant models of racial hierarchy. White innocence is not a simplistic reference to the racial identification of researchers; rather, it refers to “the dominant subject position that preserves racial subordination and the differential benefits for the innocent who retains her own dominant position vis-à-vis the ‘objects’ of study” (Gutiérrez, 2006, p. 4). From this perspective, researchers are “all implicated in some way in maintaining White innocence” (p. 4). Bonilla-Silva and Zuberi (2008) expanded the conceptualization of White innocence to White logic and examined the taken-for-granted reasoning and methodologies about empirical reality: “White logic assumes a historical posture that grants eternal objectivity to the views of elite Whites and condemns the views of non-Whites to perpetual subjectivity” (p. 17). Gutiérrez (2006) and Bonilla-Silva and Zuberi (2008) used a dialectic view toward knowledge production and emphasized the role of structure (institutions), ideologies, individual factors, and race-based interests in scientific knowledge production.

Working with nondominant communities and the associated conceptualizating of culture and cultural difference play critical roles in understanding salient education problems and their solutions. Culturally responsive researchers should be reflexive toward mundane research activities and develop a critical understanding of research as a cultural practice that is loaded with deep struggles over power/privilege. Similar to practitioners who often struggle to identify how culture may shape their views and daily practices of teaching/learning (Craig, Hull, Haggart, & Perez-Selles, 2000), education researchers must interrogate the normalized, culturally situated practices and perspectives purported to be objective and culture-free.

Following Latour (1993), we posit that researchers using experimental design must examine what they are actually producing, that is, the amalgams of nature and culture instead of what has been conceptualized as “working within a purified realm of knowledge” (p. 48). As such, we include factors related to researchers (e.g., training, prior experiences, and positionality) and to research processes (e.g., negotiation of sites, methods of recruitment, and data collection) in the CRR rubric (see, e.g., the CRR Rubric Item 6 in the section “Design and Logic”). Attention to these details is critical to making visible the taken-for-granted processes of knowledge production, at times resulting in continued marginalization of nondominant people.

For the development of the rubric, we eschewed static and overly deterministic conceptualizations of culture such as those often associated with the inheritance of values, beliefs, traditions, norms, and interactional styles (e.g., making eye contact). Instead, we take culture as a dynamic, multifaceted, and generative process that cannot be reduced into a set of overgeneralized traits and outcomes that are frozen in time and space (Bal, 2011a). We employed sociocultural theory of culture, examining the dynamic interactions of individual, institutional, and interpersonal factors (Rogoff, 2003). The guiding principle of this view is that culture mediates all human activities: “Humans develop through their changing participation in the socio-cultural activities of their communities, which also change” (Rogoff, 2003, p. 11). A process-oriented model of culture can

[1] inform future research priorities and policy making in general and special education; [2] document how special education practice, research, and policy [are] enacted in racially and economically stratified schools and communities; and [3] lead to significantly improved educational outcomes for students from historically underserved groups. (Artiles et al., 2010, p. 296)

Conceptualizing Culture as a Dynamic Process

M. Cole (1996) aptly stated, “Culture is very difficult for humans to think about. Like fish in water, we fail to ‘see’ culture because it is the medium within which we exist” (p. 8). Because the study of culture is interdisciplinary, we focused the operationalization of CRR on recent and comprehensive scholarship relevant to learning, development, and collective knowledge production activities, drawing from work in education, psychology, and science studies (e.g., APA, 1990, 2003; Banks et al., 2007; Bonilla-Silva & Zuberi, 2008; M. Cole, 1996; Latour, 1993; Rogoff, 2003; Tillman, 2002). For the purposes of the CRR rubric, we operationalized culture as the residue of collective problem-solving activities and collections of cultural artifacts perfected over generations and through history that reflects a social group’s efforts to adapt, survive, and thrive in ever-changing local and global circumstances (Bal, 2011a; Gallego, Cole, & the Laboratory of Comparative Human Cognition, 2001). Four key considerations in our operationalization are mediation through artifacts, cultural activity as unit of analysis, intersectionality, and ecological validity.

CRR Rubric Development and Application

We aim to expand—not to replace—commonly accepted standards and quality indicators in special education experiential research. Although areas of overlap between the CRR rubric and existing quality indicator rubrics of experimental research (Chard et al., 2009; Gersten et al., 2005) exist, such as presenting sociodemographic information and prior training about interventionists, we have expanded issues in design and implementation, such as sampling, that have implications for generating evidence in addressing strengths, needs, interests, and experiences of youth identified with disabilities. Basing our argument and rubric items on the principle that research is itself a cultural practice, our ultimate goal is to provide a conceptual tool to enhance researchers’ reflexivity and responsivity during conceptualization, design, implementation, and dissemination of research.

Rubric Development

We created rubric items by reviewing the literature base, examining tenets associated with CRR, and identifying criteria from extant theoretical and empirical scholarship. Findings from the literature review were organized into domains for rubric item development following the American Education Research Association (AERA) standards for reporting on empirical research (AERA, 2006). These domains are (a) problem formulation, (b) design and logic, (c) sources of evidence, (d) measurement/assessment process, (e) analysis and interpretation, and (f) dissemination. We created ratings for the levels of rigor for each criterion (Chard et al., 2009).

We used a 3-point Likert-type scale (0–2) to represent the variance in cultural responsiveness ratings for each criterion. Ratings of 0 are indicative of an absence of documentation of the role of culture, employing a culturally neutral approach. Ratings of 1 are indicative of documentation that culture was viewed merely as a categorical, static variable via proxy indicators (e.g., race) determining participants’ perceptions and behaviors. Ratings of 2 are indicative of documentation that an approach focusing on the affordances and constraints of the contexts in which individual, institutional, and interpersonal factors intersect. Following an iterative process, we sent the original rubric to two experts in the area of measurement and evaluation and four experts in the areas of special education, culturally relevant pedagogy, and education research. We revised rubric items based on those experts’ feedback. The final 15-item rubric is included in Table 1. We then applied the rubric to a set of experimental studies identified by Test et al. (2009) as rigorous of evidence-based practices in special education transition (see Trainor & Bal, 2014, for a detailed account of this application). We made minor modifications in this final version of the CRR rubric.

OPEN IN VIEWER

Table 1

Rubric for culturally responsive experimental intervention research

Rubric Item	0	1	2
1. Foundational construct(s) of the study	The construct under examination (e.g., learning disabilities, social skills, intelligence, self-determination, and at-risk behaviors) is implied but not explicitly discussed.	The construct under examination is explicitly discussed as universal. It is based on a norm-referenced sample representative of people from dominant backgrounds. Evidence of alternative conceptualizations of the construct is not presented based on the studies with nondominant groups.	The construct under examination is addressed comprehensively and critically; multiple perspectives and/or competing ideas are discussed with evidence of alternative conceptualizations.
2. Relevancy of the research problem	The relevancy of the research problem to the interests and needs of participants and local communities or to the researchers and the research field is not explicitly discussed.	The relevancy of the research problem is discussed, as it relates to the research field and/or the researcher’s interests or line of inquiry.	The relevancy of the research problem addresses the research field and/or the researcher’s interests or line of inquiry and the participants’ and local communities’ strengths, needs, and interests.
3. Critical and comprehensive review of the relevant literature	The review of extant literature results in a narrow rationale for the study that does not address what is known with multiple and/or conflicting views about the concept(s) under investigation and research problem.	The review of extant literature includes scholarship as it relates only to the concept(s) under investigation and research problem.	The review of extant literature is critical and creates a dialogue with studies employing alternative methodologies and perspectives on the concept(s) under investigation and research problem.
4. Justification of the theoretical framework	The theoretical framework of the study (e.g., behaviorism, cognitivism, or social constructivism) is not discussed explicitly. It is taken for granted.	The theoretical framework of the study is discussed abstractly, only as it relates to the construct under examination.	The theoretical framework is discussed comprehensively and critically as it relates to material, social, and historical contexts of the study and participants’ lives, experiences, and preferences. The framework is justified with its limitations and strengths to study the topic, participants, and the specific context of the study.
5. Description of participants	Description of participants’ sociodemographic characteristics includes two or fewer characteristics (e.g., gender, race, income level, education, gender, and disability).	Description of participants includes more than two individual characteristics (e.g., gender, race, income level, formal education, and disability).	Description of participants includes both individual characteristics and the institutional and social dimensions (e.g., institutional and cultural group histories, rules, and power differentiations) as related to the context of the intervention for both the control and intervention groups.
6. Description of researchers and interventionists	Description of researchers and interventionists (e.g., teacher, translator) includes two or fewer individual characteristics (e.g., gender, race, and language)	Description of researchers and interventionists includes more than two individual characteristics (e.g., race, economic background, gender, and disability); however, the description is limited to the dimension of the individual.	Description of researchers and interventionists includes individual characteristics and the contextualized institutional dimensions and relational positions among and between the participants and interventionists (e.g., status and insider/outsider positions).
7. Description of sampling procedures	Recruitment and sampling methods are not discussed.	Recruitment and sampling methods are discussed but lack detail about the rationale for the inclusionary criteria (e.g., exclusion of English language learners) and the congruency between recruitment methods and participants’ experiences and preferences (e.g., language preference or method of contact).	Recruitment and sampling methods include differentiation based on participants’ living conditions, experiences, and preferences and maximizing the potential to include nondominant communities (e.g., people with disabilities or English language learners) into intervention studies as participants and/or researchers.
8. Description of research setting	Description of the research setting is not discussed.	Description of the research setting includes the school and community characteristics relevant to the construct under examination.	Description of the setting includes political, material, social, and historical factors and their interaction within the context of the intervention process.
9. Description of data collection strategies	A rationale for the data collection strategies is not discussed.	A rationale for the data collection strategies is provided; however, it is limited to a technical discussion of the methodology.	A rationale for the data collection strategies includes consideration of participants’ preferences, needs, strengths and interests. Multiple methods of data collection are used and maximize accessibility (e.g., providing interviews and measurement tools in participants’ language, using participant-selected locales). Description of evidence includes interactions between the researchers, interventionists, and participants.
10. Ecology of the intervention	The intervention includes a contrived context, task, and control for variables to the extent that its application in outside of the research contexts (i.e., real life) is unlikely.	The intervention includes controlled variables and experimental conditions and that generally represent participants’ experiences and preferences in nonresearch contexts, yet the properties of the intervention align more closely with those of the interventionists.	The intervention is aligned with participants’ experiences and preferences in nonresearch contexts (i.e., real life). The integrity of the participants’ experiences and contexts is balanced with the intervention.
11. Intervention design	Culture-free approach: The intervention does not consider fundamental aspects of cultural and linguistic diversity that participants, interventionists, and researchers bring to the study. The diversity of the material, social and historical contexts, is not discussed.	Culturally sensitive approach: The intervention incorporates methods and procedures to integrate individual and within-group diversity. These include a combination of the following: Researchers’ and interventionists’ training for working with historically marginalized groups, including participants’ interests, perspectives, and practices; inviting community representation; ensuring availability and accuracy of translation of translation of intervention materials and procedures; considering the implications of legal issues (e.g., citizenship status); and/or examining the applicability of interventions to participants’ lives.	Culturally relevant approach: The intervention study incorporates methods and procedures to address diversity but also meets all three fundamental criteria for culturally responsive interventions: to improve academic achievement, skills and knowledge, and social outcomes; to affirm participants’ cultural group and personal identities; and to facilitate the development of participants’ critical perspectives to develop both an awareness of and the capacity to challenge inequities that they experience.
12. Assessment of intervention efficacy	The validity, reliability, and language of the measurement(s) are not discussed.	The validity, reliability, and language of the measurement(s) are discussed, but the measurements are standardized and norm-referenced for a population other than the sample.	The validity, reliability, and language of the measurement(s) are inclusive of the range of populations representative of participants, or the limitations of the measurement(s) and the lack of availability of normed-referenced tools for use with the sample is explicitly discussed.
13. Presentation of findings	The results are not disaggregated according to the participant sociodemographic characteristics.	The results are disaggregated according to participant characteristics between and within the intervention and control groups, but are limited to gender, race, income level, or language or disability.	The results are disaggregated according to participant between and within the intervention and control groups and include intersections of participants’ sociodemographic characteristics (e.g., race, class, and disability identification).
14. Analysis and interpretation	Culture-free approach: Participant’s cultural, linguistic, and economic backgrounds and contextual factors are not included in data analysis and interpretation.	Cultural deterministic approach: Participant’s backgrounds and contextual factors are analyzed as categorical and static variables. Differences among the participants are interpreted based on the dis/advantages associated with living conditions, sociodemographic characteristics, or participants’ lack of competencies in mainstream skills and knowledge.	Cultural instrumentalist approach: Participant’s backgrounds and contextual factors are analyzed as dynamic, complex, and dialogical. Differences within the participants are interpreted as situated in affordances and constraints of the physical, social, and historical relations of the context. Factors under consideration include organizational structures, power distribution, and participants’ identities and preferences.
15. Discussion of dissemination	Dissemination strategies are limited to the presentation of data in the article.	Dissemination strategies extending beyond the article are discussed.	Dissemination strategies are strategically selected to maximize sharing of knowledge (e.g., the data were shared with practitioners, policymakers, and nondominant families) with clear practical benefits to participants’ communities writ large.

Below we provide a discussion of each rubric item and rating to illustrate the relationship between extant literature, rubric criteria, and rating schema in six domains of the AERA (2006) standards for reporting on research: problem formulation, design and logic, sources of evidence, measurement/assessment process, analysis and interpretation, and dissemination.

Discussion

In the present article, we explore the cultural responsiveness of experimental intervention research and its significant implications, specifically for working with nondominant youth experiencing academic and behavioral difficulties in schools. Conducting intervention research employing methodologies that are culturally responsive is a priority posited by scholars concerned with the enduring problems of inequity in special education identification and service delivery such as disproportionate representation of nondominant students (e.g., Artiles et al., 2010; Haager et al., 2007; Vincent et al., 2011). At the same time, special education scholars have published several seminal works operationalizing rigor in research methodologies, tethering ongoing problems to a lack of a strong empirical knowledge base (Brantlinger et al., 2005; Chard et al., 2009; Gersten et al., 2005; Lane et al., 2009; Odom et al., 2005). Operationalizing CRR, however, has received less attention than the operationalization of rigor in experimental research, often conceptualized as culture-free or largely absent consideration of race, class, and gender.

The development of this rubric addresses the knowledge gap resulting from definitions of rigor operationalized without attention to cultural practices embedded in and resulting from scientific inquiry. The application of this rubric for the purposes of ascertaining the extent to which evidence is culturally responsive can be found in other publications (Trainor & Bal, 2014). Our review of the literature did not evidence a rubric for the consideration of CRR. However, criteria for the concept of cultural responsiveness in research did surface. Our work was to identify these criteria and to bring them together in the form of a rubric, a commonly used format in our field, special education, for the purpose of examining rigor as pertains to operationalizing CRR.

We recognize some of our rubric items may challenge the dominant applications of positivistic and postpositivistic methodologies. We value experimental designs and have conducted intervention research. Hence, our aim is to contribute to the development of culture-conscious experimental research by paying closer attention to contextual factors and interaction effects producing ecologically valid and sustainable interventions that address complex issues. We support CRR, even if doing so necessitates decisions in design, implementation, and dissemination that may not strictly adhere to historically privileged approaches to experimental research. Our work essentially blends notions of responsivity and reflexivity, a construct typically associated with qualitative methods, into experimental research. This may require researchers to be open and critical and to report limitations more transparently than is currently common practice. This rubric may make an important contribution through the expansion of identified criteria for rigor and broader notions of what counts as evidence.

We also acknowledge that the dominant conceptual framework of special education intervention research may presuppose a deficit in need of fixing, and that this position is ableist (Campbell, 2008; Ferri & Connor, 2005). The tension between intervening and efforts to normalize people with disabilities illustrates yet another dialogue about oppression and equity. Viewing disability through a scope focused on measurement and intervention is foundational to special education rarely interrogating and challenging the underpinnings of ableism as addressed by disability studies and critical special education scholars who examine historical and structural manifestations of dominant views of normalcy and dis/ability. Dialogue about how interventions may be beneficial for all has an important role in CRR, and we acknowledge that our rubric only opens the door to such a discussion for a paradigm expansion. Working within experimental design, this rubric highlights the importance of people with disabilities as decision makers and their situated knowledge in design, implementation, and dissemination. We invite intervention researchers and disability studies scholars to further this dialogue.

In agreement with Bonilla-Silva and Zuberi (2008), we think a research field is at its best when it is reflexive via continuous critical examination of its tools and methods. Without such examination, perspectives, experiences, and goals of dominant social groups are accepted as natural, logical, and normal. The rubric is our continuing effort in responding to the call for strengthening rigor, applicability, and effectiveness of experimental intervention research in special education (Chard et al., 2009; Odom et al., 2005; Ortiz & Yates, 2010) and for creating race-conscious, culturally responsive practices in education research in general (Erickson & Gutiérrez, 2002; Ladson-Billings, 2006; Leonardo, 2009).

Science “is not about certainty, but about uncertainty” (Erickson & Gutiérrez, 2002, p. 22). Our hope is that the rubric as a living artifact will be negotiated and adapted by practitioners in and through the cultural activity of education research that is complex, ever-shifting, and ambiguous in nature. One goal of the CRR rubric is to increase our potential as researchers to address the disparities in access, outcome, and opportunity. Another goal is to address dialectical tensions within education research because the dominant research paradigm for knowledge production has not been able to adequately address and perhaps even reproduce the status quo inadvertently increasing the disparities.

The use of the CRR rubric can facilitate praxis, a continuous critical reflection and action, to challenge naturalized and thus taken-for-granted practices and ideologies of knowledge production and to design culturally responsive intervention studies. Our hope is that this rubric will be instrumental in the necessary paradigm expansion in special education to understand the cultural nature of learning and development and constructs such as race, class, gender, and disability. Doing so will provide additional ways to explore and address the historical exclusion and marginalization of nondominant communities in knowledge production activities and the widening and deepening inequalities.