Critical Quantitative Literacy: An Educational Foundation for Critical Quantitative Research

Author Positionality

Positionality statements aim to illuminate, to the reader and the author(s), how an author’s identities and professional background interface with the context of the research being presented. Such statements are common in qualitative research studies, yet they are scarce in quantitative studies due partly to the misconception that quantitative studies are objective and that author positionality plays no role (Castillo & Gillborn, 2022). I include a positionality statement in this manuscript because I believe that neither this work nor any other research is wholly objective. Moreover, I endorse the inclusion of such statements as an essential part of producing CQL research, and I encourage such practices in other quantitative studies. I write in the first person to underscore the personal and subjective nature of this statement.

I produced this manuscript from the position of numerous privileged social identities, including that of being a cisgender, heterosexual, White male. My academic and professional backgrounds consist of philosophy, mathematics, statistics, and educational studies. I have spent more than a decade teaching statistical methods to university students, and I have applied statistical methods to academic research for most of my career. Moreover, I have no intention to stop applying them to academic research, despite recognizing their flaws. Rather, I aim to acknowledge these flaws and revisit, revise, and repurpose quantitative methods from a critical and equity-focused perspective. My doctoral studies in a large school of education introduced me to numerous critical theories, some of which were predicated on philosophies familiar from prior studies. My familiarity with philosophy, mathematics, and statistics predates my growing knowledge of critical theories. I was not taught mathematics or statistics from a critical perspective, and I was surprised to learn of their history later in my career. I attribute my ignorance to my more privileged socialization and to the nonexistence of such a paradigm as CQL. Over time, I became increasingly familiar with the emerging fields of CritQuant and QuantCrit. As I read through the research in these fields, I had numerous points of agreement, disagreement, and confusion. Some of these reactions were due to my more privileged socialization that constructed systems of maintaining ignorance (Sullivan & Tuana, 2007), and some of them were due to differences in my understanding of the strengths, weaknesses, context, and limits of quantitative methods. The idea of developing CQL arose from these tensions, including the objective to improve the criticality of quantitative methods in its goals, design, assumptions, findings, and language.

Critically Informed

Every aspect of the quantitative research enterprise in the social sciences can have a direct mapping onto real consequences for real people in the real world. The outcomes of quantitative research may challenge the systems that marginalize individuals or perpetuate marginalization. Researchers cannot be tasked with omniscience, but they can be cognizant of the quantitative decisions they are making and consider how these decisions may translate to real people and real consequences. Such cognizance requires careful attention to detail and scrutiny at each stage of the quantitative research continuum. This scrutiny is supported by the insights of critical theories, such as feminist theory and CRT.

Understanding

The word understanding here is intended to encompass full and thoughtful consideration of the early, middle, and later components of quantitative research, including the logical throughline of the entire quantitative research process. Early axiological and ontological components include, for example, the research questions being asked, the research design and objectives, and the data being collected. The middle components of quantitative research include the data cleaning decisions, the analytical decisions (e.g., outlier removal, data dis/aggregation), and choice of analytic tools. Some later epistemological components of quantitative research include the purely quantitative interpretation of findings, the narrative around the interpretation of findings, and the dissemination of those findings. Important decisions are made at each step, and CQL requires critical awareness throughout. The potential impact of every decision should be given careful thought.

Statistical Research Design

Designing statistical research requires the awareness that in order to use quantitative methods to answer research questions, complex social phenomena must be distilled into measurable variables (ontology). In doing so, decisions must be made, and these decisions introduce external and researcher biases into the research design. Such introductions cannot be avoided, and CQL requires an analysis of the implications. Additionally, before a design is ever considered, research questions must be formulated, and these questions ought to be critical and equity-oriented in nature. These aspects of statistical research design must be met with the elements of the previously mentioned understanding. Key considerations of statistical research design include the questions of what, where, why, how, and for whom?

Definitions

From the beginning stages of formulating research questions to the final stages of presenting quantitative research findings, variables and terms are being defined (and sometimes redefined) by the researcher. Many of these, such as racial or gender categories, are crudely approximated and heterogenous monoliths that fail to reflect the diversity of reality (e.g., Garvey et al., 2019; Philip et al., 2016; Zuberi & Bonilla-Silva, 2008). Interpretation of quantitative findings relies on meaningfully defined terms. To the extent to which these definitions have not been considered and articulated, the findings do not merit a clear substantive interpretation (epistemology and ontology). CQL requires cautious awareness of how variable definitions limit or enhance quantitative research.

Variables

Variables encompass what has been included and omitted from the statistical model. Moreover, CQL investigates why chosen variables have been included and omitted and the potential impact on the research. CQL does not task researchers with the impossible task of including every relevant variable, but it does task them with being aware that variables’ exclusion does not preclude their influence on quantitative findings. Additionally, variables require measurement, which exists in multiple forms, such as Likert scales, self-identification, and open responses. Often, such as with Likert scales, these measures provide crude quantification of socially complicated phenomena. The way variables are measured carries over into model assumptions, performance, and critical interpretation.

Methods

Although considerable attention is paid to the choice of quantitative methodology, considerably less attention is paid to satisfying the mathematical assumptions of these methods, how these methods function (i.e., the internal mathematics), and how inattention to either of these affects social justice when interpreting the findings. Exploratory data analysis may reveal, for example, the (in)appropriateness of linearity assumptions or whether linearity holds for everyone in the sample. It may also reveal differential response patterns in measurement (e.g., Culpepper & Zimmerman, 2006). Exploring unmet assumptions may also reveal biases, often tilted toward whoever is the majority in the sample, such as masking heterogeneous effects between groups. Satisfying mathematical assumptions is essential for conducting any rigorous quantitative research, but it is doubly important for CQL because failure to meet these assumptions can have consequences that are counterproductive for critical and equity-focused work.

Findings

Understanding the findings includes not only recognizing the most precise mathematical interpretation of statistical results from the method chosen for analysis but also having a contextualized interpretation of these results, given the many decisions made around methods, variables, definitions, designs, and subjective biases. Knowing the mathematics of statistical analysis is important, but many other parts of the quantitative research pipeline also influence the findings. Awareness and careful interpretation provide rigor and criticality to the work, encourage caution toward overinterpretation, and underscore the need for replication in quantitative research.

Unpacking the Statistical Definitions

Perhaps the most familiar statistic in quantitative methods is the arithmetic mean (henceforth, mean). Often, the mean is one of the more intuitive statistics for learners in introductory courses and is defined as the sum of values within a set of numbers divided by the total number of entries in the set. Students (and educators) seldom critically interrogate the mean, which is unfortunate because much more can be said about it, particularly from a critical perspective. For example, the purpose of the mean is to estimate what is happening around the center of the data, but by its mathematical definition, and by virtue of being a point estimate, it obscures what is happening at the fringes of a set of values. Said another way, the mean, as a single “representative” number, effectively hides the distribution of the data. Indeed, the goal of the mean is to identify a point of central tendency. But in a school district where some students are performing exceptionally well or poorly in comparison to the rest of the district, for example, the mean has ontological and axiological implications. Appealing to the mean of the district masks real scores by real students who are having real and different experiences. Much more is happening in schools and districts than mean scores can convey, and we miss seeing the trees for the forest. Moreover, means can be increased in ways that benefit only some students in the set, which risks exacerbating disparities between privileged and marginalized students if the fuller distribution goes unexamined while focusing only on increasing means. Given the mean’s ubiquity throughout quantitative methods, this phenomenon is not limited to descriptive statistics, and critical quantitative scholars ought to be keenly aware of this obfuscation. Indeed, the mean forms the basis of most statistical analyses (e.g., linear regression models) and serves as the typical point of comparison for hypothesis tests, such as t tests. The purpose of this discussion, from a CQL perspective, is to develop an awareness of what the number does, what it represents, what it does not tell you, and why these pieces of information are important and, accordingly, to identify useful information to supplement the mean.

A related definition to consider might be statistical variance. It is, by definition, a sample’s average squared deviation from its mean. Perhaps because the value itself is squared and superficially fails to intuitively convey much about the distribution of the data, variance is often treated as a mere means to the standard deviation. But again, this view is unfortunate because the way in which the variance is defined makes ontological claims about meaningful ways of describing diversity and heterogeneity within a sample. ¹ Variance is defined in relation to the mean, as opposed to another measure of central tendency, such as the median or mode. And, as already suggested, the mean obscures the general distribution of a sample, despite its relative sensitivity to highly deviant values. This definition of variance is, like the mean, propagated throughout the breadth of statistical methods, thereby normalizing quantitative methods’ focus on the mean. For example, variance serves as the basis for the F test used in analysis of variance. As with the mean, the goal from a CQL perspective is to place the definition and interpretation into a critical context, recognizing the critical implications of the variance being defined around the mean and supplementing understanding the mathematical definition with a critical understanding of its implications.

Unpacking the Math

Slowing down to contextualize the mathematical formulae found in quantitative research methods is essential for building CQL and for reimagining how these tools should be used. The goal is to read the mathematical machinery through a critical lens. For example, an educator might ask which part of the mean’s equation led it to obscure the values found in the tails of a distribution. The answer may be twofold. First is the invisible (and unnecessary) equal weighting of each observation in the data set. Second, the sum obtained in the numerator is divided by the total number of observations n. Coupled with the equal weighting, division by n accentuates the most dense regions of the data. Responses farther from the mean, although influential, are fewer in number and thus are less represented by the point estimate that is the mean. Moreover, division by n (or n – 1) is common in other statistics, including the variance and some effect sizes. Careful consideration of who is included in that n is a suggested practice in QuantCrit (Castillo & Gillborn, 2022).

Other opportunities to unpack the math arise and illuminate ways in which biases, inequities, and hasty generalizations can seep in. For example, it is well known that the n in the denominator of standard error calculations, coupled with overreliance on p values as arbiters of statistical significance, can lead to trivial differences in means whose epistemological “significance” is not placed in context (Nuzzo, 2014; Ziliak & McCloskey, 2008). Moreover, differences with p values that fall above conventional thresholds for statistical significance (α = .05) are not unimportant. There are trivial differences in p values of .049 and .051, and many factors can affect p value calculations (Gelman & Stern, 2006). Alternatively, consider that most regression models make such assumptions as additivity and linearity. These assumptions are fine but represent a very specific relationship between variables that is taken for granted and often unexplored. In psychometric measurement, subsets of eigenvalues and eigenvectors are used to approximate complex relationships between variables, represented by shared (mean-centric) variance, often on limited-range Likert-scale data, and carry traditional assumptions, such as linearity and residual normality. Yet we often uncritically give these variables labels and obscure the underlying mathematics. This practice is part of what is referred to as the jingle-jangle fallacy (Kline, 2016).

Building CQL also means building the awareness that quantitative methods do not have to be limited by many of these assumptions. Exploratory data analysis is a powerful tool that can help uncover, for example, differential response patterns between groups (e.g., Culpepper & Zimmerman, 2006). Moreover, despite their heightened difficulty in interpretation, nonlinear models exist and can be adopted by researchers. Similarly, effect sizes exist to help contextualize mean differences, and awareness of their mathematical functioning can help scholars critically discuss mean differences in scientific research. In psychometrics, such tools as robust estimators and multiple group modeling help mitigate unmet statistical assumptions or analyze differences in measurement along group-based lines. The mathematics undergirding quantitative methodology are riddled with definitions and assumptions that are important for building insightful CQL. Importantly, these definitions and assumptions render epistemological claims from quantitative research more ambiguous and uncertain than often believed.

Unpacking the Assumptions

Quantitative research methods include at least two types of assumptions: mathematical assumptions, such as the homoscedasticity of residuals in linear regression models, and philosophical assumptions, such as the intrinsic value in the variables being used as part of the quantitative inquiry. Mathematical assumptions are often discussed in quantitative methods classrooms, but the philosophical assumptions are often taken for granted. Both have important implications for building CQL.

There are too many mathematical assumptions in quantitative research methods to address here. Moreover, these assumptions vary, depending on the method being discussed. For the sake of illustration, consider the assumptions of homoscedastic and normally distributed residuals in a linear regression model. Besides simply knowing that these are mathematical assumptions of the linear regression model, emphasizing their importance from a critical perspective can help develop CQL. The interpretation of non-normal residuals, for example, may change, depending on the residual distribution’s shape, but in general, non-normal residuals imply that the model is performing poorly for some individuals (observations). This result could mean that the model is making poor predictions for some individuals (given by standardized residuals far from zero), or it could mean that the model is systematically over- or underpredicting for individuals within some range of the data if the residual plot is skewed. Augment this information with the interpretation of heteroscedastic residuals, which imply that the regression model is systematically performing better for some ranges of the data than for others. In both cases, axiological issues around equity arise when we ask such questions as Who is being poorly modeled? To the extent to which model diagnostics go unexplored and unreported, researchers bury the answers to such questions. Unpacking the math also provides an opportunity to discuss these questions. The definition of a residual—the difference between an observed and predicted value—makes it clear that residuals systematically above or below zero, residuals many standard deviations above or below zero, or residuals exhibiting disparate variances within different regions of the data come from a model that is favoring some members of the data over others. CQL places these diagnostics into critical context to underscore their importance.

Philosophical assumptions in quantitative research methods often go unexamined because they are not mathematical and, therefore, can evade discussion in quantitative methods classrooms. However, they are crucial for meaningful statistical inference and, therefore, a fundamental part of developing CQL. For example, if a researcher wants to compare students’ performance on a statewide assessment along racial and ethnic lines, they are making implicit ontological assumptions about the meaningfulness of the statewide assessment and the classification mechanism for racial and ethnic groups. If such assumptions about the meaningfulness of these things were not made, there would be no point in asking the question. The answer would be irrelevant, ambiguous, and/or nonsensical. For example, if it is held that the statewide assessment fails to provide important, accurate, or relevant information about the examinee, and/or if the classification mechanism for racial and ethnic groups is unacceptably crude or inconsistent, then performance differences between groups provide no useful information, irrespective of statistical significance. Yet we, as quantitative researchers, make these assumptions all the time, such as with meaningful racial silos, despite substantial arguments to the contrary and alternative recommendations (James, 2001; Sen & Wasow, 2016; Zuberi & Bonilla-Silva, 2008). Whether in one’s own research or in the consumption of others’, what is important for CQL is that quantitative scholars are cognizant of these assumptions and explicit about how they function.

Other philosophical assumptions are also dormant in quantitative research. Taking all prior assumptions for granted, if a researcher were to claim statistical significance between racial or ethnic groups on a statewide assessment, then some authority is afforded to the confidence level α to arbitrate what is statistically relevant. This assumption is epistemological in that if a p value falls on one side of α, we believe that we have learned something new about the world. But if the p value falls on the other side of α, we believe that we have learned nothing. Reframing statistical inference outside this arbitrary classification can help build CQL in that it contextualizes the epistemology of statistical inference for what it really is: a probability claim predicated on numerous mathematical and philosophical assumptions, each of which needs investigation and scrutiny.

Unpacking the Design

Designing a study to answer questions by using quantitative methods is difficult and requires ontological sacrifices to translate a complex reality into numbers. These challenges have been suggested elsewhere in this manuscript, yet other important considerations for CQL include the two closely related issues of the data collection mechanism and the analytic sample. These issues are farsighted in that they have generalization in mind and are concerned with for whom results will be valid (other CQL considerations notwithstanding). Random sampling from a population of interest is often the preferred approach to ensure that findings are generalizable, and deviations from a random sample can reconfigure and obfuscate the population being represented. Yet in many social settings, random sampling is not feasible, if not altogether unethical. Self-selection and convenience sampling can become the norm, thereby limiting, or altering, the generalizability of a statistical analysis.

Other important considerations of research design include the mechanism by which data are collected, and how this mechanism may relate to participants’ responses. For example, psychological phenomena, such as stereotype threat, are well known for their downward impact on evaluation scores for more marginalized individuals (Nguyen & Ryan, 2008). Alternatively, other psychological phenomena, such as desirability bias, are known to skew responses to survey questions pertaining to sensitive topics (Grimm, 2010). The insight of CQL is to recognize that numbers are not generated in a vacuum. Consideration must be given to the influences on these numbers when using them for statistical analysis. These ideas are shared with the data literacy scholarship, within which some scholars have called for an increased criticality around the production and consumption of data itself (e.g., Irgens et al., 2020; Pangrazio & Selwyn, 2019). The key realization is that whether explicitly or implicitly, decisions are made about what data to collect and how to collect it. These decisions are not neutral and have implications for quantitative output.

Unpacking the Language

An easy way in which educators can help build CQL is by paying careful attention to the language used in quantitative research. This focus often requires picking apart the words used within the findings and discussion sections of empirical research. For example, it is common for interpretations of regression models to invoke causal language, such as the “effect” of X on Y. Many scholars have articulated the problem with causal language, especially with noncausal variables, such as race (e.g., Holland, 2003). Causal language is often invoked when what is really being observed are probability-based mean differences or nonzero (linear) relationships between two variables conditioned on numerous other assumptions being met. Loose language provides an opportunity for educators to contrast written or spoken words with everything else underlying the quantitative methods, thereby reinforcing CQL when reading and producing quantitative findings. Relatedly, quantitative findings are often placed into a larger discussion in scientific research by being cited in other studies. CQL encourages readers (and authors) to be mindful of how language is propagated and thus to situate their language within the broader scientific community.

Other important considerations of the language in quantitative research regard how some individuals may be implicitly excluded from the study and how deficit frameworks may be introduced. Limiting an investigation of educational outcomes to boys and girls, for example, assumes a gender binary that alienates and discredits the experiences of individuals with other gender or sex identities (Garvey et al., 2019). Alternatively, deficit language is often used to describe the results of statistical analyses, especially in conversations around student achievement (Ladson-Billings, 2007). It could even be argued that deficit language comes naturally to a system of epistemological inference formulated around analyzing the probability and magnitude of mean differences between two populations. This work is an opportunity to discuss how quantitative research ignores the underlying sociopolitical systems that produce and contribute to deficit narratives (Russell et al., 2022). Deficit framings are not necessary to illuminate mean differences, and with care and attention, such framings can be avoided entirely (Ro & Bergom, 2020). Moreover, those with CQL can contextualize findings with critical explanations for why such differences may occur.

Design

Fundamental questions for constructing a quantitative research design include What does this research aim to reveal? and Whom will the findings be about? Alternatively, if reading research produced by others, the question might be reframed as Whom is the research about? and Do the sampling methodology and definitions accurately map the sample to the intended population? Where, if anywhere, are the discrepancies? As noted, the variables selected, the variables not selected, and the supporting arguments for the research hypothesis ought to match the design and narrative. This approach, then, provides other lines of inquiry: Which variables were included? Which variables were not included? How do the included variables relate to one another, and how do omitted variables affect these relations? If, for example, the research hypothesis is that socioeconomic status (SES) correlates with social mobility, then does the research design isolate the effect of SES, or does it allow it to be confounded with unmeasured factors, such as systemic racism? What instrumentation is used in the data collection process, and what are the implications of this decision? Is the collection process designed to facilitate accurate and relevant responses, and have such potential biases as social desirability been mitigated? The instrumentation used invokes specific definitions. At each step, the narrative should match the answers to these questions.

Measurement

Quantitative research findings should be interpreted in the context of specific measurement definitions for the variables used in a study. Accordingly, interrogating definitions and placing them in context offers a broad space for inquiry. SES is commonly measured in a variety of ways (such as household income or free and reduced lunch), yet these definitions are not the same thing and are often described under the umbrella term of SES. Interrogating how variables—independent and dependent—are measured is an essential part of CQL. Contrasting the interpretation of quantitative findings with the variable definitions while paying close attention to the language is also an essential part of CQL. Interrogating the appropriateness of variable definitions for answering the proposed research questions is, again, central to CQL. For example, broad racial categories, such as Asian, notably fail to capture important sociocultural and historical heterogeneity often siloed by this label. Studies that contrast Asian success, say, with that of other minoritized communities often fail to recognize challenges facing Hmong, Vietnamese, Cambodian, or other Southeast Asian communities included in the broader Asian label (Her, 2014). On a more fundamental level, it has also been noted that most quantitative methods operate on means, variances, and covariances. Each of these has a specific definition that represents a choice made by the researcher. Developing CQL requires asking questions about how variables are measured and placing the answers to those questions in the context of the research questions, implications, language, and discussions.

Methodology

Just as developing CQL is theory-agnostic, it is also (quantitative) methods-agnostic in that developing CQL applies to any quantitative methodology. No matter the method, CQL encourages one to ask why this method is being used, whether an appropriate alternative exists, precisely what is conveyed by this method, and how the results of this method map onto the substantive research questions. Relatedly, one might ask about how outliers are being defined and accommodated by the chosen method and what assumptions are being made about them. Quantitative methods require a variety of mathematical assumptions to be met, some of which can be quite difficult. Knowing and understanding these assumptions are central parts of CQL, as is inquiring about who might be affected, and how.