Abstract
Classroom practices and approaches often rely on anecdotal evidence for implementation and effectiveness. Conducting small-scale, quasi-experimental studies can provide empirical evidence for the effectiveness of a classroom practice. In technical and professional communication, quasi-experiments tend to be underused compared to other research methods. This article introduces quasi-experimental research as a tool for instructors to use in their teaching approaches and practices by addressing two common fears that prevent them from conducting such research: the fear of doing it wrong and the fear of wasting time. The authors use case studies to explain key concepts, including the difference between quasi and true experiments, selection bias, and confounding factors, and discuss principles of quasi-experiments related to ethical considerations, data collection, and statistical analysis.
This article introduces quasi-experimental research as a tool that instructors can use to evaluate their own pedagogical approaches and practices. While quasi-experimental research is not unique to instructors and scholars in technical and professional communication (TPC), we contextualize the application of quasi-experimental research specifically for TPC instructors by providing scenarios, applications, and contexts that are common in TPC instruction. Therefore, this article will be especially useful for TPC instructors who are interested in benchmarking existing or innovative pedagogical practices, but it will also be helpful for scholars who have not used quasi-experiments in their own research but would like to include it in their toolkit.
Here are two scenarios of how quasi-experiments can apply to TPC instructors:
You are excited to have come up with a new strategy, grounded in leadership research and theory, to support collaboration and teamwork in your class's team projects. But you want to establish that your enthusiasm is warranted: That is, you want to ensure that you are not so enamored with your new idea that you are just seeing what you want to see. You give a workshop on how to design PowerPoint presentations, advocating the assertion–evidence model (Alley et al., 2005; Nathans-Kelly & Nicometo, 2014). An engineering professor attending the workshop states that while this model might be useful for nonexpert audiences, it is not helpful for communicating research to peers. Although you are not certain you agree with this claim, you have no actual evidence for your advocacy of this model in this context.
Experienced TPC instructors have likely encountered some version of these scenarios: cases in which they wanted evidence that something works—that a teaching method grounded in pedagogical theory is in fact effective in practice. Both scenarios can be answered by quasi-experimental research designs.
Quasi-experimental research is, at heart, a simple research design that allows the investigator to evaluate a hypothesis by comparing two or more groups on any number of outcome variables. This kind of research is particularly useful for instructors because it can help individuals identify blind spots—places where they see what they want to see or places where they are unduly swayed by a single compelling anecdote rather than a pattern of more subtle evidence. Moreover, such research is highly valuable to a community of practice because multiple researchers can pool their results, allowing the community to gradually aggregate evidence from multiple small studies into a more complete, nuanced picture of effects. Recent TPC scholarship has called for more aggregable approaches to research in a variety of ways, including calls for a more consistent body of knowledge (Coppola, 2010), a shared set of research questions (Rude, 2009), rigorous and replicable approaches to research (St.Amant & Melonćon, 2016), and a move beyond anecdotal evidence to inform our practice as technical and professional communicators (Graham, 2017).
Despite its potential, quasi-experimental research is underutilized. A survey of TPC research found that less than 7% of all TPC scholarship published in 1992 through 2011 used a quasi-experimental or true experimental design (Boettger & Lam, 2013). Further, a more recent study of TPC journal articles published in 2012 through 2016 shows that the percentage of experimental research actually decreased, with just 3.4% of articles using experimental designs (Lam & Boettger, 2017). And a still more recent study (Melonćon & St.Amant, 2019) of empirical research methods in five technical communication journals also found that there is a dearth of experimental research in TPC. Specifically, just 19 empirical research articles in a period of 5 years used an experimental method—and removing IEEE Transactions on Professional Communication from the data set lowers this number to just two experimental studies. This paucity of research is reflected by the field of writing studies at large. An overview of writing research published in 1999 through 2004 (Juzwik et al., 2006) found that out of 4,739 articles related to writing studies, only 151 (3%) were classified as experimental or quasi-experimental (the majority of the studies used interpretative methods such as discourse analysis, interviews, or focus groups). Similarly, an analysis of 270 articles published in the Writing Center Journal in 1980 through 2009 found that only 1% were experimental or quasi-experimental designs (Driscoll & Perdue, 2012).
Wolfe (2021) has suggested that this lack of quasi-experimental research can in turn hinder TPC instructors and scholars from receiving research funding. In an informal survey of grants received by members of the College Composition and Communication group, Wolfe found that of the 67 grants reported, 19% were for studies using quasi-experimental designs—a percentage that is far higher than that of published studies with such a design. Moreover, when the analysis was narrowed to just those studies receiving grants of more than $50,000, 35% used quasi-experimental designs. While this survey is far from conclusive, it does suggest that at least some funding agencies value quasi-experimental designs.
So why is published quasi-experimental research, which can provide funding opportunities for innovative pedagogical approaches, so scarce? Although Wolfe (2021) has attempted to address ideological reasons that prevent scholars from engaging in quasi-experimental research, we assume here that TPC instructors are at least theoretically favorable to what Haswell (2005) has called RAD (replicable, aggregable, and data driven) scholarship and instead focus on two different obstacles that prevent instructors from conducting quasi-experimental research: (a) the fear of doing it wrong and (b) the fear of wasting time on an unsuccessful hypothesis.
In this article, we address some of the key concerns that instructors or scholars might have about conducting quasi-experimental research. To accomplish this, we return to the two scenarios that begin this article as case studies for discussing the challenges and rewards of this research method. Quasi-experimental studies are not unambiguously successful. They are flawed—as is all research—and the results do not always point to straightforward conclusions. We have chosen to highlight the mixed nature of these studies in order to confront the fears that likely keep many researchers from engaging in this useful methodology. Before we present these two case studies, we will first define quasi-experimental research and make a case for quasi-experimental research in TPC. But the heart of our discussion addresses the driving concerns with conducting quasi-experimental research. We provide specific examples from our case studies to address issues that could prevent researchers from engaging in quasi-experimental research. By addressing many of the concerns or obstacles that instructors might have, we hope to make this research method less intimidating.
What Is “Quasi-Experimental” Research?
While there can be complex variants of quasi-experimental research, it is a simple research design: The researcher tests a hypothesis by dividing participants into groups that differ according to a specific variable or phenomenon of interest. For example, a researcher interested in testing whether a new strategy for supporting collaboration is effective would divide students into two groups: one group of teams would use the new strategy and another group of teams would use the status quo. Such research is “experimental” because the researcher actively tries to control or manipulate the variable of interest (e.g., the teaching strategy). Such research is also “quasi”—not truly experimental—because many variables are out of the researcher's control due to the logistics of the naturalistic research setting (e.g., a classroom rather than a lab).
Another key difference between true, or lab, experiments and quasi-experiments is in how research participants are assigned to the control or experimental group. In a true experiment, participants are assigned to a group by randomization (i.e., every study participant has an equal chance of being placed in the treatment or control group) whereas in a quasi-experiment, participants are typically assigned to the treatment or control group by either the researcher or self-selection. The lack of randomization could have a negative impact on research outcomes because it could introduce confounding variables. For example, differences between treatment and control groups could be influenced by demographic characteristics that were not considered due to nonrandom groupings. Conversely, randomization could positively influence external validity, or how well the sample that was studied accurately reflects a population. But for every research design, a researcher must make tradeoffs depending on the context, setting, and resources for the study.
While the term “quasi” may carry a negative connotation, the quasi nature of quasi-experimental research should not be perceived as a net deficit. Although experiments conducted in labs can control for more variables—and thus are more methodologically rigorous—they have the major disadvantage of artificiality. Lab experiments are conducted in settings where the experiment is foregrounded, and individuals might behave differently from how they would in the “real world.” By contrast, quasi-experiments are conducted in less controlled settings where participants are likely to behave more naturally, which lends itself to an increased ecological validity (i.e., ability to generalize to real-life situations). Thus, what quasi-experimental research loses in methodological rigor, it gains in naturalism. Table 1 lists some advantages and limitations of true experiments versus quasi-experiments.
Some Advantages and Limitations of True Experiments Versus Quasi-Experiments.
Why Do a Quasi-Experimental Research Project?
TPC instructors or scholars might want to engage in quasi-experimental research in their classrooms for three reasons: (a) to validate or challenge whether their methods of teaching are effective, (b) to validate or challenge whether they are emphasizing the right content in their courses, and (c) to find data that will establish pedagogical best practices and persuade those outside the field that these best practices are better than other methods.
Most dedicated instructors are likely already engaged in some form of nonsystematic quasi-experimental research. For instance, instructors who have tried out a new teaching technique in their classroom have probably subjectively compared student papers or student experiences before and after implementing this change. Quasi-experimental research is simply a more systematic form of this reflective practice. To illustrate the benefits and applications of quasi-experimental research, we elaborate on the two scenarios that begin this article to show what quasi-experimental research looks like.
Teamwork Case Study: Is a New Practice or Technique Better Than the Status Quo?
Chris (Author 1), like many other TPC instructors, requires team projects in many of his classes. But these team projects bring many challenges, some stemming from the lack of a solid working relationship between team leaders and team members. To reduce relational conflict and friction between leaders and members, he experimented with having team members provide leaders with data-driven feedback using leader-rapport management (Campbell & Lam, 2019). Leader-rapport management is a construct that measures how well a leader uses linguistic strategies to build relationships with team members. The theory behind this construct, then, assumes that if leaders receive an objective measurement of their ability to manage rapport with team members, they will have a chance to adjust or reinforce their leader-rapport management behavior depending on the nature of the feedback they receive. To test whether this experimental, data-driven feedback method was effective, Chris studied multiple sections of the same class with one crucial difference: In one set of classes, team leaders received data-driven feedback at the project midpoint (e.g., the experimental group); in the other set of classes, team leaders did not receive any feedback (e.g., the status quo, or control group).
Team members in both sets of classes completed a survey asking about the team leader's effectiveness in managing relationships at two points in the semester: the midpoint—just before the team leaders in the experimental class receive feedback—and at the end of the project. The hypothesis was that team leaders in the experimental class would experience greater gains in their relationship effectiveness than would those in the control group. Although this hypothesis was ultimately not supported, the data Chris gathered was still useful in supporting the pedagogical promise of the approach, as we will later discuss in more detail.
PowerPoint Case Study: Are “Best Practices” Really Best?
Joanna's (Author 2's) former writing center frequently gave workshops teaching the assertion–evidence model (Alley et al., 2005; Garner & Alley, 2013; Nathans-Kelly & Nicometo, 2014) of PowerPoint slide design. But this model and the evidence supporting it have been challenged by faculty and graduate students who claim that the model is ineffective or unneeded for advanced research. To test their claim, Joanna and her associate director, Nisha Shanmugaraj, found a consortium held in one of their school's engineering departments and persuaded the consortium leaders to have audience members complete feedback surveys commenting on presentation quality. They then contacted graduate students scheduled to present and invited them to participate in a research study in which they would receive a small cash stipend in exchange for learning about the assertion–evidence model and working with a writing center consultant to make sure they effectively implemented the model. Six students agreed to participate. The hypothesis was that conference attendees would rank more highly the quality of the presentations of the six graduate students who worked with the writing center to implement the assertion–evidence model than they would those of the other graduate students.
These two case studies show how instructors can use quasi-experimental research to address tangible, practical problems that they encounter in the classroom. We have chosen these examples because of their relative simplicity. We will come back to these case studies as we address two common fears of conducting quasi-experimental research and describe some key concepts of and approaches to such research.
“Did I Do It Right?” Addressing the Fear of Doing It Wrong
The first fear or objection to quasi-experimental research that we want to address is the fear of executing the research study incorrectly. The idea of manipulating a specific variable is concerning for many researchers. In quasi-experimental designs, the variables are often “muddy”: Many elements are outside the researcher's control. Researchers often worry, then, that their results will be challenged and that their study will be dismissed due to a flawed design. To ease their concern, researchers should keep in mind that
all research, no matter how carefully designed, is flawed; there is no such thing as a perfect study. their study is not the end of the story, but it is the beginning of a larger conversation, just one of many studies on topics that can provide a more compelling story about the larger research question. their goal is not to eliminate all flaws but to honestly and openly acknowledge them and then emphasize conclusions that can be put forward despite these flaws.
For instance, our two case studies have different flaws. But the goal of the quasi-experiments in both cases is not to eliminate these flaws but to persuade readers that there is something worth attending to despite the inevitable flaws.
In the Teamwork Case Study, for example, Chris studied team projects in two different classes: one that included data-driven feedback and one that did not. The potential flaw in this design is that the two classes could have two very different populations based on variables outside of Chris's control. Also, students in the experimental class received more feedback and instruction than did those in the control class. Thus, any gains that students had in the experimental class could be due to getting more attention than the control class did. In other words, the study might show that any intervention—not just the specific intervention of data-driven feedback—produces gains.
In the PowerPoint Case Study, the six graduate students who volunteered to participate in the study were almost certainly more motivated to improve their communication skills than were the graduate students who did not work with the writing center. Moreover, the use of the assertion–evidence model was also conflated with writing center feedback in this study. Thus, the study is potentially flawed because positive results might have less to do with the assertion–evidence model than they do with writing center assistance—or with the motivation of students who are willing to participate in a study on presentation skills.
Thus, both studies, to various degrees, suffer from selection bias—or a mismatch between the sample studied and the general population being considered. Both also suffer (again, to various degrees) from confounding factors that muddy the variable being studied (e.g, in study 2, the writing center visits confounding with the assertion–evidence model). And both suffer from contextual factors—participants know they are in a study and therefore may act differently from the way they would normally. So why are these flaws not fatal? How can these studies be implemented, let alone published, despite these flaws? The primary answer is that each study needs to be considered—and presented— as a drop in the bucket of knowledge. The findings simply edge us closer toward an answer to a larger, overarching research question.
For instance, in the Teamwork Case Study, the data-driven feedback method Chris described is innovative and makes sound theoretical sense. The pedagogical description of this method may have been worth publishing even without the quasi-experimental study supporting it. But the quasi-experimental data provide insights into the strengths and weaknesses of the approach. After reading this study, Joanna is eager to implement Chris's method in her class but with some key changes intended to overcome the drawbacks that his study helpfully highlights. Thus, even though Chris's study was flawed in terms of selection bias and confounding factors, the data he presented helped provide a relatively unbiased account that allows other researchers to refine and improve his ideas.
In the PowerPoint Case Study, the selection bias and confounding factors are particularly strong, and the sample size is very small. But the study was still worth conducting because these self-selecting participants who received extra help might not have outperformed their peers. If that had been the case, Joanna would have needed to reconsider her advice to use the assertion–evidence model in research contexts. But her hypothesis was upheld: The presentations of the six students receiving writing center help and using the assertion–evidence model were rated more highly than those of their peers, suggesting that the assertion–evidence model at least does not hurt and may help research presentations. And based on these findings, Joanna can now answer critics in her workshops by saying she has evidence that graduate students who worked with a writing center tutor on the assertion–evidence model were perceived by peers and faculty as having higher quality presentations than those of the graduate students who did not do so. Moreover, the success of this initial trial has encouraged her to repeat the experiment—but this time without the required tutoring appointment. After conducting additional small studies to further support her hypothesis, Joanna hopes to publish research making a strong claim that the assertion–evidence model is effective in research-based contexts.
The primary way to address the flaws of quasi-experimental research, then, is to carefully scope the claims that can be made, keeping in mind that each study is an incremental step forward that will have to be repeated with additional research. In our experience, writing instructors who are skeptical of quasi-experimental research perceive it as making much more positivist claims than an ethical researcher would advance. In fact, one reason that we both gravitate toward quasi-experimental research is because it requires researchers to make an extra effort to delineate the limitations of their studies with the expectation that future researchers will critique and improve on the methods and study designs of prior work. To us, this open recognition of flaws makes quasi-experimental research a more collaborative enterprise than many other forms of research.
“What If I Get ‘Bad' Results?” Addressing the Fear of Wasting Time
A common misconception of empirical research, particularly quasi-experimental research, is that there is a major risk in taking the time to design the study and collect, clean, and analyze the data only to find that the hypothesis is not supported. But such unsupported hypotheses not only are critical for teachers in rethinking their pedagogy; they can lead to new theories that challenge conventional wisdom or teacher lore.
Thus, it is useful for instructors who are thinking about a quasi-experimental project to imagine all potential outcomes of the study and determine whether a failed hypothesis would mean the end of their research—or the beginning of a new research agenda. Few studies lead to complete dead ends although many take us on much more complicated journeys than originally planned. For instance, Sir Alexander Fleming was originally studying the influenza virus when he discovered penicillin, which developed accidentally while he was on a 2-week vacation. Although Fleming did not plan or hypothesize such a finding, he was able to discover a world-changing medicine from this unexpected result. Likewise, many thwarted studies lead to new opportunities.
For example, the hypothesis in the PowerPoint Case study was supported, so we have evidence that the assertion–evidence model can work in technical research settings—a modest contribution that could help persuade STEM researchers to adopt this model. But if the hypothesis had not been supported—if the graduate students working with writing consultants and implementing the model had not produced presentations that the audience perceived as being of higher quality than their peers—then perhaps the engineers who questioned this model's utility for communicating research to peers were correct. Or perhaps the writing consultants gave misleading advice at some point in the process. Or perhaps our methods of teaching the assertion–evidence model were ineffectual. In any of these situations, we have fundamental assumptions about communication, genre, or pedagogy that need to be explored.
Thus, in the PowerPoint Case Study, there is no situation in which the research could “fail.” If the hypothesis is upheld, the researcher has a straightforward project that makes a modest contribution to the literature on PowerPoint design. If the hypothesis is not upheld, the researcher makes a much more complex—and potentially groundbreaking—contribution that could change practices.
By contrast, the Teamwork Case Study does contain a possibility of failure. If the researcher's hypothesis that his innovative method helps build rapport within teams is not upheld, the status quo would not be challenged. But the cost of such a failure would not be particularly high. The researcher would only have lost time spent collecting and partially analyzing data (stopping once he realized that the results were not positive). If he still believes in the method, he could make changes to his pedagogy and rerun the study the next semester (perhaps collecting different kinds of data). If he no longer believes in the method, he will change his pedagogical practice accordingly.
To help ensure that he would find useful information about the effects of his innovation, Chris collected a wide range of data to analyze from surveys of team members, interviews with team leaders, and students’ written reflections on their team experiences. Thus, even though the findings of his data analysis did not support his primary hypothesis (that the method would improve leader–member relationships), he did find that his data-driven feedback method positively increased students’ overall perception of the teamwork process, a major benefit given that teamwork is difficult to assess. Moreover, because of Chris's systematic data collection and analysis, he was able to pinpoint a particular problem that prevented his data-driven feedback method from realizing its full benefits: Team leaders often avoided any action that would require direct confrontation with team members despite having data that suggested such action. This finding suggests ways that Chris could improve his data-driven feedback method by combining it with instruction on how to constructively confront team members and provides direction for additional follow-up studies.
Principles for Making Quasi-Experimental Research Less Intimidating
Now that we have addressed two major fears of conducting quasi-experimental research, we describe some high-level principles that make quasi-experimental research more manageable.
Principle 1: Work With What You Have
As we pointed out, there is no research that is not flawed. Your ideal research study may involve collecting data that are logistically cumbersome to obtain. For instance, you might want to interview students about revisions made to their papers. But such interviews are difficult to schedule and time-consuming to collect and transcribe, so you might consider collecting less ideal but more feasible data about what you want to find out, such as asking students to write a reflective memo about their revisions. While such reflections are less rich than interviews and have the downside of being written for a grade, they still tell us something about what students perceive as an ideal writing performance or process. Reflections may not tell us what students really think, but they at least tell us if students understand our goal for them and what our instruction aims to teach.
As much as possible, try to use data, such as formal papers, reflections, or short surveys at the end of a unit, that organically fit into your research setting. For instance, the Teamwork Case Study collected student reflections on their teamwork processes that Chris had already asked students to provide. In the PowerPoint Case Study, Joanna asked conference attendees to rate and comment on presentations and promised to share this feedback with the presenters. Although these ratings were a new task, conference attendees saw the value of providing graduate student presenters with feedback. The research study thus complemented participants’ preexisting goals.
When designing a study, then, think about what resources you have access to and work with those. If you know of an ongoing assessment in a department or program, explore whether you can use some of those data for your own purposes. Tell close colleagues about your hypothesis and study, and see if they are interested in trying out your technique in their classroom or would be willing to have their students serve as a control group. If a department or colleague is surveying students, see if you can piggyback on that survey by adding one or two short questions for your purposes.
Principle 2: Imagine the Final Results at the Outset and Work Backward
When you start a new project, envision what you would like your results to look like. Imagine both the kind of results you would get if your hypothesis is supported and the kind of results you might get if your hypothesis is not supported. You might even sketch out the graphs or tables that you would ideally like to report. Then work backward to make sure you are capturing the types of data that will allow you to support your hypothesis.
For example, in the Teamwork Case Study, Chris wanted to support his hypothesis that data-driven feedback would positively influence team leaders’ rapport with their team members. Thus, his survey included Likert-scale questions measuring the quality of the relationship between leaders and team members (e.g., “On a scale from 1–5, how would you characterize your working relationship with your leader?”) Chris imagined what would happen if his hypothesis was not supported. He would want data that might help explain why it was not supported, so he also collected data concerning a variety of other potentially explanatory constructs, including communication quality, social loafing, and group effectiveness. If the quality of the leader–member relationship did not significantly change due to the data-driven feedback, these additional constructs might provide additional insight into the influence and impact of data-driven feedback. Of course, these explanatory constructs were not arbitrarily selected; they were included because prior theoretical and empirical research showed that the constructs were correlated with each other (Lam, 2015). Specifically, prior research has indicated that communication quality is negatively correlated with social loafing, and social loafing is negatively correlated with group effectiveness. Thus, imagining different outcomes and drawing from prior literature helped Chris plan his data collection.
Principle 3: Do Not Reinvent the Wheel
Read examples of other studies in similar areas to yours, and replicate as much of their methodology as makes sense. For instance, in designing the Teamwork Case Study, Chris looked through other studies that measured leadership effectiveness and research that asked students to evaluate other team members. He then reused these questions on his surveys and reflections. Reusing survey items, interview questions, or prompts that others have found productive not only increases the likelihood that your measures will be effective, but it makes your results more credible and useful by allowing researchers to directly compare your findings with those of previous research.
Principle 4: Consider a Trial Run
Collecting trial data could determine if a more formal study is worth the effort. For instance, if you have a last-minute idea for a study—perhaps you tried out a new strategy and it went unexpectedly well—consider doing a trial run even without obtaining approval from your institutional review board (IRB) or formal consent from students. In the United States, such trial runs, conducted without the intent of publishing the data, are considered internal assessments and do not require IRB permission. Although you cannot publish data from this trial run, it will help you find flaws in your data collection instruments and determine if your hypothesis merits further investigation. You can then obtain IRB approval; improve your surveys, interview protocols, or other forms of data collection; and conduct the official study.
Keep in mind that data collection is the easy part of a study. If you spend an hour or two collecting data that you cannot use but that teaches you more about the problem you are investigating, your time has not been wasted, but be mindful of your participants’ time, and avoid asking them to do activities that you do not analyze. A trial run of the study, then, will prepare you to execute your full-fledged study with fewer flaws or hiccups.
Principle 5: Work With Your IRB to Weigh Concerns About Using Your Students’ Data Against the Good of Improving Your Own and Others Instruction
If you intend to publish or present your results, you must obtain approval from your school's IRB. Working with IRBs at U.S. universities, we have found it useful to speak to the IRB officer assigned to review our study protocol before submitting our IRB application. A short phone call allows the IRB officer to better understand your goals and constraints and then advise you on the best way to balance the often competing considerations of pedagogical effectiveness, student privacy, and research validity. This phone call up front will ultimately save time for both you and the IRB officer.
Even after obtaining IRB approval, you may still feel that it is uncomfortably self-serving to conduct a study in your own classroom. But most researchers do classroom studies that they think will ultimately improve their pedagogy, so asking students to help you learn and grow as an instructor is not inherently self-serving. We try out new strategies and probe our assumptions all the time as teachers: A quasi-experimental study is simply a deeper, more formal method for assessing our pedagogy, and asking students to engage in this teacher and curricular assessment is both reasonable and ethical. In fact, we argue that instructors should always be engaging in curricular and instructional self-assessment, whether or not they are conducting formal research. In the United States, conducting programmatic and curricular assessments are not considered research for IRB purposes—only collecting data with the intent to publish is research. Thus, the activities you are asking students to do (e.g., write a reflection or take a short survey about their impression of a unit) are ones that you typically ask your class to do anyway, you could ask your IRB about saving consent for the end of the semester when you are simply asking students for their consent to let you include their data anonymously in your research.
Advanced Knowledge of Statistics Need Not Be Extensive to Analyze Quasi-Experimental Data
In most cases, it will be helpful to conduct statistical analyses comparing your control and experimental groups. These analyses help you decide which differences between the groups are meaningful and which ones may be spurious results due to random chance differences between individuals. The good news is that most quasi-experimental research studies can be completed using just two statistical tests: the t-test and the chi-square test. And even better, many online calculators produce reliable results and walk you through the statistical process. We recommend GraphPad, which has a very intuitive interface for calculating t-tests, and Social Science Statistics, which has a simple interface for calculating chi-square tests. You can find other online calculators by googling “online calculator t-test” or “online calculator chi square.”
One prevailing question for many new to quasi-experiments is this: How do you decide whether to use a t-test or a chi-square test? A good practice is to first analyze your data using simple percentages and averages and experiment with different ways to display differences between your control and experimental groups. Find a way to present your data that tells a clear story about what is different between your control and experimental groups. If that story focuses on averages, you most likely will want to use a t-test. If that story involves raw numbers or percentages, you most likely will want a chi-square test.
Often, the same data can be analyzed with either type of test, depending on how you frame your research narrative. For example, in the Teamwork Case Study, Chris asked participants to rate the quality of their relationship with their team leaders using a 5-point Likert scale in which 1 indicated strongly disagree and 5 indicated strongly agree. This construct yields statistical averages; therefore, Chris used a t-test to compare the two groups.
But Chris could have also decided to further combine the data. Specifically, he could combine Likert scale scores by grouping any score of 3, 4, or 5 (i.e., neither agree nor disagree, agree, or strongly agree) as a “positive relationship” and any score below three as a “negative relationship.” He would then report the percentages of students reporting positive and negative relationships. So if Chris had chosen this method to present the data, he would have used a chi-square statistical test to compare the two groups.
Although there are, of course, other factors to consider, the learning curve for statistical analysis is not steep. Table 2 provides a simple matrix to help with statistical decision making. There are multiple correct ways to analyze the same data, and most analyses can be done using free online calculators.
Matrix for Choosing a Statistical Test According to the Quasi-Experimental Research Design.
Suggestions for Finding Additional Help
A number of resources can help you with your quasi-experimental studies. First, many institutions have a statistical consulting service for researchers. The staff at such offices can help you not only analyze results but plan your studies. If your institution does not have such an office, you could seek help from faculty in your social sciences departments: Perhaps a faculty member in sociology, for instance, would be willing to help you with study design or analysis in exchange for your help with a writing workshop or grading? Alternatively, you could hire a student who has completed the required statistics courses for their social science or education major to provide a few hours of tutoring. And there are online statistical consulting services available for a fee. Joanna has had good experience with The Analysis Factor (https://www.theanalysisfactor.com/).
For more fundamental training in data collection, coding, and analysis, you could apply for the annual Dartmouth Summer Seminar for Composition Research (https://writing-speech.dartmouth.edu/research/summer-institutes-seminars), for which Joanna has been a regular session leader. Similarly, Chris has begun creating a series of YouTube articles on basic statistical tests for quasi-experimental research that can be found at (https://www.youtube.com/c/ChrisLam1). Finally, the WAC clearinghouse has a number of online guides for conducting research in writing studies, such as a guide on experimental and quasi-experimental research (see https://wac.colostate.edu/resources/writing/guides/experiments/).
Implications for Practice
We have intentionally written this article in a practical way so that those unfamiliar with quasi-experimental research can easily apply these principles to their own teaching or research agendas. We described several research studies that have revealed a lack of experimental research in TPC. An overarching implication of this article, then, is that the field needs more quasi-experimental research. Quasi-experiments, as we have described here, allow researchers to validate or invalidate accepted best practices and to propose and test novel approaches to TPC teaching and practice. We hope that this article will impact not just academic researchers but also TPC practitioners by encouraging them to seek out evidence-based arguments for best practices or conduct their own mini quasi-experiments.
We also hope that this article will lead to more systematic self-reflection and improvement in pedagogical practices. As we have described, many instructors are already conducting mini quasi-experiments in their classrooms. They are trying a new approach, gathering feedback, and informing subsequent pedagogy in these mini experiments. This article, then, provides instructors with a simple on-ramp for conducting more rigorous, potentially publishable quasi-experimental research. What comes with the publication of more quasi-experimental research, though, is an increase in robust, evidence-based studies for TPC instructors.
In summary, this article was not intended to replicate the style of traditional statistics textbooks. Instead, it provides a simple overview of the concepts and lessons needed to implement quasi-experimental research. We hope that this article provides readers with a clear roadmap for exploring new research questions with this underutilized method.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
