Abstract
In this paper, we consider Tracy’s proposal for universal criteria to judge the goodness of qualitative studies, and we explore her criteria by applying them to our own work. As a test of Tracy’s claim to universality, we situate her ideas within a womanist caring framework, using it as an exemplar to theorize deliberately Tracy’s criteria. We do this to evaluate just how theoretically limber Tracy’s criteria actually are. We conclude that Tracy’s criteria are useful because they are universal but not fixed and we build on her thinking to foreground ethics as an overarching framework for criteria rather than a standalone category. We highlight the urgency of returning the debate about criteria for qualitative work to the qualitative research community during a period when the legitimacy of qualitative work is under assault.
In this paper, we join the conversation about evaluating the quality of qualitative research by responding to Tracy’s 2010 article published in this journal. In her article, Tracy proposes universal criteria to assess the quality of qualitative research, breaking with the view (Creswell, 2007; Denzin, 2008; Golafshani, 2003; Guba & Lincoln, 2005) that the variety of qualitative methodologies requires a similar variety of criteria. We were intrigued by Tracy’s claim to universality and use our paper to explore her criteria by applying them to our own work.
A few years ago at AERA, we presented a paper about the disjuncture between the theoretical sophistication of qualitative research methodology and the persistence of traditional criteria used to evaluate qualitative work. At the time, we were both assistant professors who were having difficulty getting qualitative research articles published and we wondered how our work, which used alternative epistemologies and research methodologies, was being evaluated. In the years since that presentation, the issue of goodness in qualitative research has continued to be debated and new work on quality in qualitative work has been published (Cho & Trent, 2006; Freeman, 2011; Freeman, deMarrais, Preissle, Roulston, & St. Pierre, 2007; Tracy, 2010).
Currently, however, the important internal debate about quality among the diverse community of qualitative researchers has been overshadowed by the larger discussion that questions whether qualitative research has merit at all. Now, well into our careers, we regularly conduct qualitative studies and teach qualitative methods courses. We rejoin the conversation about qualitative research quality to return the center of gravity about goodness in qualitative research to the qualitative research community.
Like other scholars (Cho & Trent, 2006; Freeman, 2011; Freeman et al., 2007; Seale, 1999; Tracy, 2010), we have sought some sort of a usable framework that could help reviewers/readers determine what qualitative work is good, what the weaknesses and strengths of a given study are, and which studies are fatally flawed. Like others (Bochner, 2000; Scheurich, 1996; Schwandt, 1996; Smith & Deemer, 2000), given the multitude of perspectives enriching qualitative inquiry, we are reluctant to propose fixed criteria. On the other hand, we believe there should be some guidelines to evaluate the goodness of qualitative work.
There seemed to be little escape from the tension between the desire for some guidance to evaluate qualitative studies and the skepticism of universal criteria designed to span the wide range of qualitative studies. Then we read Tracy (2010) who proposes eight “big tent” 1 criteria for qualitative research. We were intrigued and drawn to Tracy’s claim that her criteria can work for all genres of qualitative studies and her distinction between “common end goals of strong research (universal hallmarks of quality) and the variant mean methods (practices, skills, and crafts) by which these goals are reached” (p. 839). All the same, we worried about the claim to universal criteria, which might position methods as “neutral” and divorced from epistemology and ethics (St. Pierre, 2006). Are Tracy’s means and end goals as flexible as they appear or will they lead to new regimes of truth (Foucault, 1980) that would marginalize some kinds of qualitative research and privilege others?
In this paper, we consider Tracy’s proposal for universal criteria to judge the goodness of qualitative studies, and we test her criteria by applying them to our own work. Do Tracy’s “variant means” for achieving quality criteria really make room under the “big tent” to include alternative epistemologies and methodologies? As a test of Tracy’s claim to universality, we situate her ideas within a womanist caring (Beauboeuf-Lafontant, 2002) framework. We have used womanist caring theory collaboratively and individually to frame our work, so we summon it here as an exemplar to theorize deliberately Tracy’s criteria that, because of their proposed universality, can appear to be neutral. We do this to evaluate just how theoretically limber Tracy’s criteria actually are. Our objectives in this paper are (a) to use womanist caring as an ethical framework for theorizing Tracy’s criteria for qualitative research using our own work as an example and (b) to explore Tracy’s claim that the criteria she proposes are, in fact, universal. We begin with a brief overview of the criteriology literature.
Brief History and Current State of the Field
In her assessment of the field, Tracy notes the long-standing debate within the qualitative research community on what good research is. Starting with Lincoln and Guba (1985), differing ideas of what makes quality qualitative research has been hotly debated (Altheide & Johnson, 1998; Cho & Trent, 2006; Dadds, 2008; Freeman, 2011; Freeman et al., 2007; Garratt & Hodkinson, 1998; Lather, 1993; Lincoln, 1995; Marshall, 1985; Richardson, 2000; Tracy, 2010). Indeed, Tracy’s own paper and our response to it are examples of the continuing conversation. The more complex and varied qualitative methodologies become, the more discussions there are about qualitative goodness and the more elusive a set of universal criteria seems to be.
The burgeoning of qualitative methodologies and discussions about criteria is situated in the context of an increasingly conservative political climate. As Tracy (2010) aptly notes, Despite the gains of qualitative research in the late 20th century, a methodological conservatism has crept upon social science literature over the last 10 years (Denzin & Giardina, 2008), evidenced in governmental and funding agencies’ preference for research that is quantitative, experimental, and statistically generalizable (Cannella & Lincoln, 2004). (p. 838)
Tracy argues that the debates about criteria within the qualitative research community have bewildered those outside of it and have served to further marginalize qualitative work. She cites Denzin (2008) who wrote, “We cannot afford to fight with one another. . . . We need to find new strategic and tactical ways to work with one another” (p. 321). Tracy’s call for universal criteria is an attempt to unify the qualitative research community in the face of political assault and the privileging of quantitative, experimental work.
These are the reasons that, in a departure from previous scholars (Bochner, 2000; Guba & Lincoln, 2005; Schwandt, 1996) who have argued against fixed, universal criteria and critiqued criteriology, Tracy (2010) proposed eight universal criteria (end goals) for evaluating qualitative research studies and delineates various means of achieving these goals. Her proposed criteria are (a) worthy topic, (b) rich rigor, (c) sincerity, (d) credibility, (e) resonance, (f) significant contribution, (g) ethics, and (h) meaningful coherence. This comprehensive list runs the gamut from technical considerations (e.g., how much time was spent in the field and following IRB procedures) to topic choice, uses of theoretical constructs, and importance. At face value, these criteria appear reasonable and strive to include all genres of qualitative research under the big tent. We will test this claim and explore how Tracy’s criteria (end goals) apply to two of our own projects and examine the variant means we used to attain those goals.
Our Projects
Jenny is conducting a longitudinal study of a charter school in post-Katrina New Orleans. Interested in educational reform, she spent the 2008-2009 academic year doing research and teaching at one start-up charter school. Since then she has continued to return to New Orleans four times a year to track the school’s progress primarily through interviews and reflective memos but also through observations.
Between 2005 and 2007, Jean and her doctoral students completed an oral history of Douglass school, an all-Black school that existed in Parsons, Kansas from 1908 to 1958. The school was closed in the wake of the Brown v. Topeka BOE ruling and was torn down in 1962. Almost 50 years later, Black citizens of Parsons still mourned its loss. Older citizens who had attended Douglass School told stories of their experiences at the school and expressed fears about subsequent generations of Black students in Parsons not knowing this part of their history. In the next section, we describe womanist caring theory (Beauboeuf-Lafontant, 2002) before using it to frame Tracy’s criteria.
Using Womanist Caring to Frame Tracy’s Criteria
Womanist caring consists of what Beauboeuf-Lafontant has described as three characteristics: embrace of the maternal, political clarity, and an ethic of risk. The first characteristic, embrace of the maternal, contrasts White conceptions of mothering as private and individual with community-oriented Black conceptions of mothers who “through feelings of shared responsibility, commit themselves to the social and emotional development of all children in a community” (Beauboeuf-Lafontant, 2002, p. 77). Embrace of the maternal emphasizes societal concerns over private individual ones. The second characteristic, political clarity, is the awareness that society, and by extension, its schools, is structured to ensure the success of some groups of children and the failure of others. In other words, it is the awareness that oppression is systemic. The third characteristic, an ethic of risk, is the ongoing commitment to fight injustice. As Sharon Welch (1990) wrote, “The fundamental risk constitutive of this ethic is the decision to care and to act although there are no guarantees of success. . . . It is an ethic in sharp contrast to the ethos of cynicism” (p. 80). The ethic of risk requires deep commitment and perseverance in the face of obstacles.
Embrace of the Maternal as a Frame for Tracy’s Criteria
Embrace of the maternal involves regarding others as your own responsibility, seeing everyone in the community as having a claim on your energy and affection. The notion of care and investment in others provides an ethical context for Tracy’s criteria in several ways. To offer a few examples, Tracy’s selection of a worthy topic, framed by embrace of the maternal, would involve selection of a topic that addresses the concerns of others, as well as the researcher’s own interests. For example, embarking on the study of educational reform in New Orleans, Jenny loosely framed her research questions. As an outsider, it felt presumptuous to impose specific areas of inquiry. Prolonged engagement that included taking on a teaching role at the school allowed her to see and feel the concerns of teachers and administrators. Her research questions developed through interactions with others as well as through her own lenses. Respectfully negotiating differences when they arose without privileging her own concerns and perspectives has become an ongoing commitment of her work.
Jean offered to facilitate completion of the oral history project on behalf of Parsons’ Black citizens. She held a community meeting to talk about the project and to assure them the oral history would occur on their terms and they would take the lead on reconstructing the story of Douglass School. Jean and her students were all White, and working with elderly Black citizens required them to earn the trust of the alumni and to be respectful listeners and learners. Given the troubled history of White researchers studying African American communities, the research team took extra care to ensure they came to the project with cultural sensitivity (Carter, 2003; Tillman, 2002).
Other examples of using embrace of the maternal to frame qualitative quality for the New Orleans project included maintaining Tracy’s end goal of rich rigor by means of spending generous amounts of time in the field and choosing participants who relay multiple perspectives. While prolonged engagement certainly is not a requirement of rich rigor framed by embrace of the maternal, in Jenny’s study, it is important to track the development of the charter school as it gains ground and faces new challenges after its inception. One year at a school would not portray fairly educational reform after the hurricane. Longevity is an important demonstration of embrace of the maternal in the New Orleans study precisely because it allows the reform movement to take root. Similarly, speaking to a variety of stakeholders illustrates embrace of the maternal because it is important to keep others’ concerns at the heart of the work and work against privileging the researcher’s perspective.
For Tracy, the end goal of credibility refers to the plausibility of the research findings; whether the researcher presented a persuasive account, a description and interpretation of reality that seems true, but also nuanced and complex. In Tracy’s framework, credibility is achieved by means of thick description, triangulation and crystallization, multivocality, and member reflections. In Jean’s oral history project, rich rigor and credibility were achieved through conducting in-depth interviews with 55 alumni of the school. Some alumni were interviewed several times. Many alumni lent photographs and other artifacts, and the research team scanned close to 60 photographs and newspaper articles about the school. One of the alums accompanied the team on a visit to the site where the school once stood. These activities and artifacts lent a depth and richness to the study that would not have been accomplished through only conducting interviews. Multiple perspectives and views were valued and encouraged, because embrace of the maternal requires the researcher to maintain high standards of accountability to the study and to care for the participants.
Other studies might demonstrate the end goal of rich rigor framed by embrace of the maternal through other means. Tracy would agree that, despite her call for eight big tent criteria, there is no one formula for how rich rigor would look. We argue that even framing the criteria with a specific theoretical lens requires that each study dictate the means for attaining rich rigor.
The end goal of sincerity, another of Tracy’s indicators of quality in qualitative research, would be expressed through reflexivity so that the researcher examines her own stake in the research. Embrace of the maternal requires providing support to meet others’ needs so they can be successful and have access to the world’s opportunities. Framed by embrace of the maternal, the researcher must interrogate her own positions and must take the realities of others into account. For example, during her work, Jenny, a long-time teacher educator, encountered many teachers and administrators who were trained through alternative certification programs such as Teach for America and Teach NOLA. Because teachers from alternative certification pathways are so prevalent in New Orleans charter schools, openness to the perspectives and contributions made by these educators is important to the New Orleans project. In addition, it has opened her eyes to the possibilities of these programs and shifted her attitudes about teacher education. Embrace of the maternal does not require researchers to forgo their own perspectives but does prohibit closed-mindedness that dismisses the situations of others or limits their opportunities.
Political Clarity as a Frame for Tracy’s Criteria
Political clarity, the awareness that oppression is systemic, is the second characteristic of Beauboeuf-Lafontant’s (2002) womanist caring framework. Within Tracy’s framework the awareness of systemic oppression would require that researchers choose topics that are worthy of study and perhaps even liberatory. For example, engaging participants’ time and energy cannot be undertaken frivolously so the research questions should lead to a study that is worth doing because it explores/exposes/redresses inequity. Applying a womanist caring framework to Tracy’s criteria, political clarity would require that the researcher’s contribution is significant and focuses on issues of equity and social justice. (We are not suggesting that all qualitative work needs to have this focus—although it would be nice! Rather we are suggesting that grounding Tracy’s criteria in womanist caring theory would require that studies with that particular theoretical perspective should adhere to social justice commitments.)
Jenny’s work looks at issues of race and social class in a post-Katrina context. What does it mean that so many of the teachers in the New Orleans charter school context are White, middle-class elites while so many of the students they work with are poor and Black who have been traditionally underserved? Beauboeuf-Lafontant’s framework requires what Case (1997) calls clear-sightedness about these inequities. Jenny foregrounds race and class in her analysis as a way of embodying political clarity in the heightened racialized context of post-Katrina New Orleans charter schools, focusing on a worthy topic according to a womanist reading of Tracy’s criteria. Similarly, she examines her own Whiteness and class privilege in an ongoing way to monitor how this affects her analysis demonstrating Tracy’s construct sincerity, in her self-reflexivity and transparency about methodological challenges.
Jean and her students, as White and middle class researchers also acknowledged and interrogated their White privilege through the course of the project as a way of embodying political clarity and meeting Tracy’s sincerity criteria. They had to be open to a new understanding of how race operates in the telling of history, particularly the history of school desegregation. The dominant narrative of school desegregation is one of racial progress and it was painful for the research team to come to terms with how school desegregation occurred on White terms and negatively affected Black schools and communities. Making their own racial positions explicit in the narratives created about the school was a way for the team to manifest political clarity in the project.
On another level, to enact political clarity, researchers must follow through on their commitments to participants and get their work out in the form of publications and presentations. Political clarity requires operating from a sense of collective responsibility to work against existing inequities through our research and the work should benefit the participants themselves. Therefore, it is unethical/oppressive to take people’s time without anything to show for it. In the Douglass School oral history, Jean has presented and published analyses of the oral history narratives for the academic audience, and she shared copies of the photographs with the alumni, the local public school, and the Parsons Historical Society. The interviews and photographs will eventually be archived at the Kansas Historical Society to give others the opportunity to learn about Douglass School and its significance to the education of African Americans in the state of Kansas as well as nationally. One motivation for completing the Douglass School oral history was to reclaim a past that was on the verge of being lost, and to allow the alumni to narrate their own story of Douglass School and to correct the incomplete and erroneous knowledge Whites had about the school, if they remembered it being present at all in the town. Both projects not only make significant contributions to our knowledge of these two situations, they are significant to the communities where the studies were conducted.
According to Tracy, quality research has resonance; that is, if affects, influences, or moves readers or multiple audiences through aesthetic or evocative presentations and through serving as a mirror of for others to see their own experiences. The story of Douglass School has resonated with many audiences, from senior citizens in Parsons to students to educational researchers. The oral history has particularly resonated with African Americans who experienced either segregated education or attended an HBCU, as they felt validated and affirmed and were appreciative of White researchers who were sincerely trying to understand their perspective.
Ethic of Risk as a Frame for Tracy’s Criteria
The third characteristic of womanist caring, an ethic of risk rooted in a “sense of an existential interdependence” (Beauboeuf-Lafontant, 2002, p. 81) in which the commitment to working for social justice is premised “on a concept of self that is part of rather than apart from other people” (p. 81). An ethic of risk is also concerned with taking action to promote not only social change, but self-change, which requires coming to the research process with humility. An ethic of risk additionally means persevering even with the knowledge that social change is slow in coming.
An ethic of risk calls for ongoing commitment even when a project seems hopeless and connects with the research process in a variety of ways. For example, when the ethic of risk is used to frame Tracy’s rich rigor, hit and run qualitative work would be seen as unethical because it does not probe deeply enough into research situations. In longitudinal work, researcher exhaustion, though it needs to be acknowledged, does not determine the end of the study. Perseverance becomes an important marker especially when energy wanes and initial excitement dwindles. Researchers must persist with their studies and not exit the field prematurely.
An ethic of risk requires researchers to acknowledge the interdependent nature of their work, recognizing they are not distant observers who collect and analyze neutral data. Instead they are responsible to the communities where research is conducted. They approach the research process with humility and do not position themselves as the experts on the lives of those who participate in their studies. For Jenny, taking on teaching responsibilities at the school blurred her role as researcher and participant and underscored the interdependency not only of her relations with others at the site but between the research and the process of teaching. As another example, in Jean’s oral history because she and her students presented and published papers from the project, they were often turned to as the experts when it came to knowledge of Douglass School instead of serving as the vehicle for ensuring their story was told in the way they wanted it told. It was important for Jean and her students to continually resist this positioning and to approach the project with humility, a manner consonant with Tracy’s criterion of sincerity.
Tracy’s Criteria: Universal But Not Fixed
Tracy’s universal criteria for qualitative research worked with the womanist caring frame. While the means for achieving her criteria manifested somewhat differently in each of our studies, they provided a useful guide for us to analyze and evaluate our own work. We believe Tracy’s criteria, her end goals, could work with other theoretical frames, as well, taking shape according to each study’s frame and purpose. When writing qualitative studies for publication, the criteria provide a tool for scholars to monitor the quality of their own work and we believe that scholars will strengthen their work if they make their use of Tracy’s criteria explicit.
We also acknowledge as academics with multiple demands on our time, we can fall short of fully meeting Tracy’s criteria framed by a womanist caring lens. For example, one way Jenny’s study might fall short of rich rigor framed by embrace of the maternal is her reliance on interview data over field notes. This was in part because of the teaching commitments she assumed during her year of immersion in the study. However, it is fair to say that her own experiences teaching at the school became powerful formative lenses through which she framed the study. We do believe striving to meet Tracy’s criteria framed by womanist caring theory is useful to keep us from being satisfied with qualitative research that takes short cuts or short changes the process.
Tracy’s criteria for quality in qualitative research are powerful because they are universal but not fixed. That is, researchers can work toward the end goals through variant means. Tracy (2010) has proposed a model that “leaves space for dialogue, imagination, growth and improvisation” (p. 837) and are meant to be limber adapting to different theoretical frames and perspectives. Her criteria can serve as a tent as long as qualitative researchers approach them with the openness with which they were intended. The danger lies in viewing Tracy’s criteria as fixed and inflexible, thereby reducing them to a checklist and defeating their purpose and utility.
Critique of Tracy: Big Tent Criteria and Ethics in Qualitative Research
Tracy’s criteria appear reasonable and strive to include all genres of qualitative research under the big tent; however, there is one area where we depart from Tracy. While we appreciate her bold efforts, we posit that what is lacking here is the grounding of criteria in an ethical framework. From our perspective, Tracy mistakenly treats ethics as a stand-alone category. We believe ethics is an umbrella construct that threads through every aspect of qualitative work (including what seem to be technical considerations). As the researcher is the instrument in qualitative inquiry, we believe that all aspects of the study from the initial conceptualization to the dissemination of findings must be guided by decision-making grounded in explicit ethical positioning.
In our examples, womanist caring theory outlines specific commitments for the researcher as she navigates her way through the research process, illustrating how embrace of the maternal, political clarity, and an ethic of risk frame research decisions and give ethical context to Tracy’s (2010) criteria. Using womanist caring (and we believe any theoretical construct) as a framework to guide research places ethics at the heart of the research process. The selection of a worthy topic, rich rigor, sincerity, credibility, resonance, significant contribution, and meaningful coherence are all steeped in ethical considerations. We depart from Tracy in that we do not believe that ethics can, or should, be bracketed into its own discrete category.
We reiterate, though, that Tracy’s big tent criteria work well for us, but they need to be framed ethically; they are not value-neutral. It is inevitable that every study is framed theoretically (whether the framework is explicit) and each theoretical framework has ethical implications. Each framework would lend differing shadings (variant means) to Tracy’s criteria (end goals) but our sense is that her criteria would still be a useful tool. It is critical that researchers make their theoretical frameworks and the accompanying ethical positions explicit so that their work can be evaluated appropriately.
Why Big Tent Criteria Are Important
In recent years, the debate about quality has taken on even more importance because much qualitative research has been devalued and its legitimacy as a research methodology has been called into question altogether (Lather, 2004; Lincoln & Cannella, 2004; St. Pierre, 2006; St. Pierre & Roulston, 2006). Although Tracy recognizes the political climate, she notes her motivation for identifying universal criteria is primarily pedagogical. We want to build on Tracy’s pedagogical motivation and emphasize as well the very real consequences that the “politics of evidence” (Denzin, 2009) have had on qualitative researchers. The conversation about qualitative research quality among qualitative researchers (Cho & Trent, 2006; Garratt & Hodkinson, 1998; Kvale, 1995; Lincoln, 1995; Scheurich, 1996; Schwandt, 1996; Seale, 1999; Smith & Deemer, 2000) has had little effect on the practice of journal editors and manuscript reviewers who are charged with determining whether a manuscript is worthy of publication or grantors who distribute funding. In the review process, traditional, aging but stubborn notions of trustworthiness criteria often go unexamined despite the theoretical sophistication of qualitative research methodologists and regardless of the epistemological and theoretical grounding of the work. As a result, there is a growing gap between evolving qualitative research methodologies and the criteria used to evaluate qualitative work.
The divide between methodology and criteria has very real consequences. While this gap affects all qualitative researchers regardless of the focus of their projects, those most affected are scholars of color and Whites who consider issues of race and/or advocate for equity and social justice (Carter, 2003; Foster, 1999; Stanley, 2007; Tyson, 2003) and use untraditional epistemologies and methodologies. This risky work often is dismissed as methodologically suspect, perhaps because there is no pretense of neutrality and because the researcher’s agenda is usually shared (St. Pierre, 2006). We reiterate, though, that the divide between methodology and criteria has consequences for all qualitative researchers regardless of whether the focus of their work is on equity and social justice.
It is important to recognize that the review process used to determine the quality of qualitative work is also an ethical process. Journal editors and reviewers need to be aware of their own ethical commitments when evaluating qualitative studies, especially those that use alternative epistemologies and methods. Ironically, assumptions of neutrality when evaluating qualitative research are often undergirded by reviewers’ unacknowledged political and ideological positions (Stanley, 2007).
Tracy’s universal criteria have the potential to provide a framework to help the qualitative research community evaluate qualitative work and fill an important gap in the field. We do not know whether Tracy’s criteria will pitch a bigger tent or if they will move the conversation about quality in qualitative research forward. It is important to acknowledge that other attempts to dislodge traditional trustworthiness criteria have not significantly changed the practices of authors, editors and reviewers. We are encouraged by the fact that her article is among the most read of those published in Qualitative Inquiry, which suggests the urgency and timeliness of developing criteria that are universal, flexible, and reflect the sophistication of contemporary methodology. We write this article to maintain a focus on Tracy’s criteria and hope that it will sustain the already robust interest that Tracy’s original article has generated.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.