Abstract
Quality assessment in qualitative research has been, and remains, a contentious issue. The qualitative literature contains a diversity of opinions on definitions of and criteria for quality. This article attempts to organize this diversity, drawing on several examples of existing quality criteria, into four main approaches: qualitative as quantitative criteria, paradigm-specific criteria, individualized assessment, and bridging criteria. These different approaches can be mapped onto the historical transitions, or moments, in qualitative research presented by Denzin and Lincoln and, as such, they are presented alongside the various criteria reviewed. Socio-political conditions that have led us to a fractured future, where the value and significance of qualitative work may be marginalized, support the adoption of bridging criteria. These broadly applicable criteria provide means to assess quality and can be flexibly applied among the diversity of qualitative approaches used by researchers. Five categories that summarize the language used within bridging criteria are presented as a means to move forward in developing an approach to quality assessment that fosters communication and connections within the diversity of qualitative research, while simultaneously respecting and valuing paradigmatic and methodological diversity.
Given the diversity of approaches to qualitative research and the plethora of criteria addressing quality, perhaps one of the most difficult tasks a reader of qualitative research has is to determine how to assess the quality of a piece of work. Studies can provide interesting insights into complex phenomena, but how can we assess the quality of the work done? The challenge of making sense of the divergent ways qualitative research can be assessed has been noted by many; for example, within health science research conceptions of quality have been called “mysterious” (Dingwall, Murphy, Watson, Greatbatch, & Parker, 1998) and at times even mocked (Thorne & Darbyshire, 2005). Within such divergence, qualitative research runs the risk of not being assessed appropriately and, consequently, devalued. Offering a means to organize existing criteria is, therefore, an important starting point to create greater clarity with respect to these criteria and the evaluation of qualitative research.
Our intent with this article is not to exhaustively cover the range of proposed criteria in the literature, given their expansiveness, but rather to summarize several overarching approaches. Thus, examples of existing criteria from the literature are organized into four approaches or modes of quality assessment: qualitative as quantitative criteria, paradigm-specific criteria, individualized assessment, and bridging criteria. These four approaches encapsulate various views expressed in the literature addressing conceptions of quality for qualitative research and provide an efficient means to understand the similarities and differences of proposed criteria. In addition, these modes are placed within historical and socio-political contexts in this article by aligning them with the historical moments in qualitative research discussed by Denzin and Lincoln (2005). Figure 1 provides a schematic summary of the mapping of modes to historical moments:
Categories of quality criteria for qualitative research organized using Denzin and Lincoln's (2005) historical moments in qualitative research.
Although criticisms of these moments have been raised, for example with regards to their lack of sensitivity to disciplinary diversity, disciplinary development, or geographic location (Delamont, Coffey, & Atkinson, 2000; Holt, 2003; Sparkes, 2002), they provide a useful heuristic device to link various approaches to quality assessment with socio-political contextual features. Considering that what is valued as “good” research is always socio-politically situated (Denzin, 2009; Seale, 1999b; Smith & Hodkinson, 2005), placing the approaches and moments together enhances understanding of how different views of quality have emerged and continue to exist, and also why a bridging approach provides the most useful direction to enable qualitative research to continue to flourish and diversify. Ultimately, it is argued that the current socio-political context characterized by a backlash against paradigmatic diversity and justice-oriented research (Denzin & Giardina, 2009) and a rise of neo-positivism (Cheek, 2008) means that a bridging approach to quality assessment is needed. As argued by Denzin (2011), “The interpretive community must mount an articulate critique of these external threats to our ‘collective research endeavour.’ We must create our own standards for quality, our own criteria” (p. 645).
Qualitative Research Paradigms
Understanding the four approaches to quality assessment and what they afford and marginalize requires consideration of various paradigmatic approaches underpinning qualitative research. Considering this, and drawing primarily on Lincoln and Guba's work (Guba & Lincoln, 1994; Lincoln & Guba, 2003, 2005), we provide a brief overview of major paradigms that inform qualitative research. Guba and Lincoln (1994) defined a paradigm as “basic belief systems based on ontological, epistemological, and methodological assumptions” (p. 107). We acknowledge that paradigms have been defined and organized in other ways (e.g., Creswell, 2007; Kuhn, 1996; Ponterotto, 2005), but set aside this philosophical debate to focus on considerations of approaches to quality assessment. To prevent misunderstanding, this discussion focuses on research paradigms as distinct from Kuhnian paradigms which reflect historical transitions or “revolutions” within disciplines.
Lincoln and Guba have presented five different paradigms: positivist, post-positivist, constructivist, critical theory, and participatory (Guba & Lincoln, 1994; Lincoln & Guba, 2003, 2005; Lincoln, Lynham, & Guba, 2011). Although presented as separate, Lincoln and Guba (2003) describe the “great potential for interweaving of viewpoints for the incorporation of multiple perspectives, and for borrowing or bricolage, where borrowing seems useful, richness enhancing, or theoretically heuristic” (p. 264). In general, this interweaving is proposed to be productive when done across commensurable paradigms, that is, between positivism and post-positivism or between constructivist, critical theory, and participatory paradigms. Indeed, positivism and post-positivism are often most closely aligned with quantitative work, but are also sometimes used to inform qualitative work (Ponterotto, 2005). Figure 2 presents a continuum of the research paradigms discussed by Lincoln and Guba, showing points of difference in relation to ontology, epistemology, and methodology that will be the focus of the paragraphs to follow:
Continuum of research paradigms and ontological, epistemological, and methodological assumptions.
Ontology
Among the most fundamental characteristics defining the paradigms is the belief in the nature of reality (i.e., ontology). Generally, a realist position holds that there is a “true,” “real,” and single reality of a phenomenon in the world, whereas relativists contend that multiple, equally valid and useful, views of a phenomenon exist. Positivists, post-positivists, and critical theorists are all considered realists, but in varying ways. Positivists are characterized as naïve realists, who believe that a true reality exists and can be discovered; post-positivists as critical realists, who believe that reality exists but can only be known imperfectly; and critical theorists as historical realists, who believe that historical factors (e.g., social, political, cultural, and economic) have shaped social reality which, in turn, is perceived as the “way things really are” (Guba & Lincoln, 1994; Lincoln & Guba, 2003, 2005; Lincoln et al., 2011). Those working within constructivism and the participatory paradigm, on the other hand, are considered relativists. Constructivists believe in a context-dependent created reality, whereas those within the participatory paradigm emphasize reality as co-created.
Epistemology & Methodology
Paradigms also differ epistemologically, that is, in relation to views regarding the nature of knowledge and how things can best be known. Paradigms span from objectivist to subjectivist epistemological stances. Although the term objectivity can be ambiguous (Kvale & Brinkmann, 2009), in general, objectivist epistemology holds that true knowledge exists apart from, and can be understood separate from, a researcher's values and beliefs. In turn, methodologically, this leads to a focus on techniques to eliminate or control the influence of a researcher, and other potential contaminants, in the knowledge generation process. Subjectivist epistemology emphasizes that knowledge is always generated from, and exists within, a particular perspective and holds that people act in the world on the basis of their subjective knowledge. Methodologically, instead of guarding against researcher influence, there is a focus on understanding how participants understand their worlds and how knowledge is generated through interactions amongst researchers and participants within particular contexts.
Epistemological variations also exist across the paradigms. For example, both positivists and post-positivists are considered objectivists but, again, to varying degrees. Stemming from their ontological assumptions, positivists strive to verify theories as true through the use of objective research methods, whereas post-positivists employ methods of theory falsification, approximation of truth, and probability. The remaining “alternative” paradigms incorporate subjectivist epistemological assumptions, with critical theorists often taking a stance that findings are always value-mediated and constructivist and participatory researchers emphasizing co-construction of knowledge.
Paradigms matter when considering how to assess quality because these basic assumptions guide what a researcher seeks to know, how they approach knowing, and what they believe are the best ways to know (Morrow, 2005). For example, if a researcher adopts an epistemological assumption consistent with post-positivism, then it is important for quality that techniques are rigorously used to minimize the influence of the researcher, such as member checking or intercoder agreement. On the other hand, if the researcher adopts a constructivist paradigm and emphasizes co-construction of knowledge, then techniques aimed at removing the researcher from the findings have a poor epistemological fit. Rather, quality in such a study would be enhanced through the use of techniques that enhance transparency regarding the researcher's contribution to the co-construction, such as reflexivity and a clear articulation of the researcher's standpoint.
About the Authors
In taking the stance that paradigms matter for quality assessment, we, as the authors of this article, have already begun to position ourselves within existing debates on how best to assess quality. Thus, consistent with the alternative paradigms, that is, those outside of positivism and post-positivism, we believe it is important we position ourselves to enable readers to assess how our assumptions may have influenced our thoughts and our writing. The first author places himself somewhere between constructivism and participatory research, and he has a background in kinesiology, psychology, and occupational therapy and is currently completing his PhD. The second author places herself primarily within a critical theory paradigm, with her work often drawing on critical discourse analysis and critical narrative inquiry. As such, both authors view paradigmatic and methodological diversity as important in the continued evolution of qualitative inquiry.
The impetus for this article is premised in the experiences of both authors, one being only recently introduced into the qualitative world while the other having more solid footing:
First Author: As a graduate student being introduced to qualitative research, it was only natural to strive for an understanding of how to do work that would be held to a high degree and respected by others, not only in meeting the requirements of my degree but in the uptake of my research by others both within and outside of the academy. Perhaps it is a consequence of many consecutive years in undergraduate and graduate education, where evaluation takes the form of reaching certain standards, but it is important for me to know that I am doing “quality” work. Is it not a responsibility of researchers to our participants, among other stakeholders, to ensure that we are reaching for quality? Working with Dr. Laliberte Rudman, and taking her course in qualitative research, gave me the opportunity to think critically about what “quality” in qualitative research means and to have discussions with her on how this has changed over time. The result being, of course, collaboration on this article.
Second Author: As a tenured professor who has had the opportunity to introduce Masters and Doctoral students in a multi-field Health and Rehabilitation Sciences program to qualitative methodologies and methods within a course over the past 7 years, I have attempted to foster an approach to appraisal of qualitative work that involves simultaneous consideration of paradigm, methodology, and ethics and that respects diversity in how qualitative work is done. Although I have been skeptical about the possibility of a set of criteria or considerations that could be used to “think about” (Cheek, 2008) quality across the rich and valuable diversity of qualitative research, I have also increasingly experienced the pressure to frame my own critical work in ways that suit external criteria imposed by others (e.g., grant agencies, journal reviewers, among others) that fail to respect diversity or my paradigmatic foundation. Given this, I do think there is a need to move forward via discussions amongst qualitative researchers regarding key quality considerations that connect us while at the same time provide the room we need to continue to diverge.
First Approach: Qualitative Criteria as Quantitative Criteria
The idea that qualitative and quantitative research should be held to similar criteria emerged within and became dominant in the earliest “moments” discussed by Denzin and Lincoln (2005), which lasted from the early 1900s to approximately 1970. Within the traditional period, “qualitative researchers wrote ‘objective,’ colonizing accounts of field experiences that were reflective of the positivist scientist paradigm” (p. 15). An emphasis was placed on creating “valid, reliable, and objective interpretations” (p. 15). Likewise in the modernist phase, researchers attempted to formalize qualitative research and developed techniques to fit conceptions of internal and external validity in quantitative research. Quantification was also included as a way to try to legitimize qualitative research, which at this time “clothe[d] itself in the language and rhetoric of positivist and post-positivist discourse” (p. 17).
Although chronologically these moments are depicted as ending long ago, quality criteria proposed by several contemporary authors can be seen as a continuation of this first approach. The reasoning provided by many of these authors echoes that used in the traditional and modernist periods, with an emphasis on legitimizing qualitative research often in traditionally quantitative fields. Through this legitimization, specific techniques are used that attempt to produce quality in ways largely consistent with objectivist epistemology. Table 1 provides a summary of the concepts and terms that will be discussed as part of this first approach:
Concepts and Terms Associated with Quantitative Criteria as Qualitative Criteria Mode of Assessment
For example, Collingridge and Gantt (2008) attempted to legitimize qualitative research within medical sciences by proposing quality criteria that parallel those used in post-positivist quantitative research. Viewing quantitative and qualitative research as separate “paradigms” with differences in rigor, these authors discussed qualitative research in terms of reliability, validity, and generalizability. They described the parallelism of different forms of validity (construct, criterion, content) across the two paradigms, but contended that different strategies are used to enhance the generalizability of qualitative research in comparison to quantitative research. Nevertheless, while arguing that qualitative inquiry employs a different approach to ensure rigor, Collingridge and Gantt located this difference in techniques and continue to locate qualitative research solely within the positivist and post-positivist modes of thought that pervaded the traditional period and modernist phase moments of qualitative research.
Within the psychology literature, Elliot, Fischer, and Rennie (1999) proposed seven common guidelines for evaluation of quantitative and qualitative work, as well as seven “evolving” criteria specific to qualitative research. The common criteria address explicating the context, purpose, and worth of the study; using appropriate methods and specifying methods; and respecting participants. The authors argued that criteria specific to qualitative research in psychology are needed to legitimize its rigor, ensure appropriate evaluation, and inform student supervision. These specific, evolving criteria included: situating the sample, grounding the study in examples, providing credibility checks, triangulation, developing a framework to explain the data, and ensuring data can support knowledge claims. Although the evolving criteria were purported as a tentative starting point for psychology to acknowledge the unique aspects of qualitative research, they for the most part are aligned with post-positivism in that they focus on procedures to ensure the right answers and minimize researcher influence. Indeed, these authors report that their colleagues have critiqued their work for its narrow epistemological location.
Morse and her colleagues provide another contemporary example of this approach; they have explicitly argued that quantitative ideas of reliability and validity should be maintained for qualitative work in order to sustain its legitimacy (Morse, 1999; Morse, Barrett, Mayan, Olson, & Spiers, 2002). As with the other authors reviewed above, Morse emphasized that reliability and validity in qualitative research is achieved through the use of particular techniques, or verification strategies, including: “methodological coherence” to ensure the appropriate matching of the research question to the methods used; appropriate sampling to ensure category saturation; concurrent data collection and analysis to ensure gaps in knowledge are addressed; “thinking theoretically” to check and recheck data with developing ideas; and “theory development” to interconnect data.
A final contemporary example is found in the work of Creswell (2007), who espoused a pragmatic approach to research that focuses on “what works” rather than being committed to a particular method or philosophy. In reference to all types of qualitative research, Creswell advocated for “validation strategies” in a methodologically driven set of evaluation criteria designed to assess the “accuracy” of studies. These strategies, or techniques, share many commonalities with the criteria discussed by Elliot et al. (1999) and Collingridge and Gantt (2008). In particular, Creswell recommended strategies such as triangulation, peer review, the revising of hypotheses to fit negative cases, clarification of researcher bias, member checking, thick description to determine transferability, and external audits. He also discussed ways to enhance the reliability of qualitative research, including blind coding of research transcripts and checking intercoder agreement. Although he identified as pragmatic, the focus on enhancing accuracy and ensuring reliability through objectivist techniques seems to ally closely with post-positivism.
Overall, historical and contemporary work that employ the mode of quality assessment labelled as qualitative criteria as quantitative criteria tend to focus on the use of specific techniques aligned with objectivist epistemology and assume that qualitative research should aim to understand the truth as it exists apart from the researcher. As such, while this approach may work well for qualitative work informed by the positivist and post-positivist paradigms, it does not provide an appropriate mode to assess the quality of work located outside these paradigms.
Second Approach: Paradigm-Specific Criteria
The blurred genres, crisis of representation, and triple crisis historical periods discussed by Denzin and Lincoln (2005) provide the context in which quality criteria specific to paradigms were generated. These moments occurred roughly from the early 1970s through to the mid-1990s. During these moments, the number of paradigms, theories, and methods taken up in qualitative research increased, promoting epistemological diversity that called into question “issues such as validity, reliability, and objectivity, previously believed settled” (p. 18). The legitimization crisis involved a “serious rethinking of such terms as validity, generalizability, and reliability” (p. 19), which as pointed out by Denzin and Lincoln had been reformulated, or rejected, within the evolving alternative paradigms. Table 2 provides a summary of the concepts and terms that will be discussed, which are associated with conceptions of quality in this approach:
Concepts and Terms Associated with Paradigm-Specific Quality Criteria Mode of Assessment
Perhaps one of the best known examples of paradigm-specific criteria comes from Lincoln and Guba (Guba & Lincoln, 1994; Lincoln & Guba, 2003, 2005). For positivism and post-positivism, they listed the conventional benchmarks held out for qualitative research in the traditional and modernist moments, including external validity, reliability, and objectivity. Their criteria for research positioned within the critical theory paradigm include “historical situatedness,” or the degree to which socio-historical antecedents have been considered, as well as more transformative criteria such how the work has made “ignorance and misapprehensions” more transparent and has stimulated “action” for change (1994, p. 114). These criteria were expanded to include attention to social transformation, equity, and social justice (Lincoln & Guba, 2003). In their later work, Lincoln and Guba (2003) took up participatory paradigm criteria advanced by Heron and Reason (1997). These criteria attend to how a participatory project addresses the congruence of different ways of knowing, such as experiential, presentational, propositional, and practical, and the extent to which it “leads to action to transform the world” (Heron & Reason, 1997, p. 293).
Lincoln and Guba perhaps have had the greatest difficulty generating quality criteria for the paradigm in which they positioned their own work, that is, constructivism. Initially, they proposed using their earlier generated forms of trustworthiness (i.e., credibility, transferability, dependability, and confirmability), but they have questioned whether these criteria are the best way to evaluate constructivist work because of their “parallelism to positivist [and post-positivist] criteria” (1994, p. 114). Lincoln and Guba later expanded on conceptions of validity in constructivist work to include their own authenticity criteria (specifically, fairness and ontological, educative, catalytic, and tactical validity) and Lincoln's (1995) ethics-centred criteria, as well as ideas from others such as Richardson (1994) and Lather (1993).
More specifically, Richardson (1994) proposed a postmodern form of validity, termed crystalline validity, to replace traditional conceptions of validity, such as triangulation. Using the crystal metaphor, Richardson described how crystals have “an infinite variety of shapes, substances, transmutations, multidimensionalities, and angles of approach … provid[ing] us with a deepened, complex, thoroughly partial, understanding of the topic” (p. 92). Lather (1993) proposed other transgressive forms of validity, such as rhizomatic and voluptuous, which similarly conceived of validity as being complex and partial and therefore consistent with a relativist and subjectivist position.
In relation to ethics-centred criteria, Lincoln (1995), building on the work of Lather (1993), Palmer (1987), and others, put forward a number of “emerging” criteria that were relational in nature and located in the nexus between epistemology and ethics. These criteria included: “positionality, or standpoint judgements,” where authors and texts “come clean” about their epistemological position (Lincoln, 1995, p. 280); “community as arbiter of quality,” reflecting the idea that “research is first and foremost a community project, not a project of the academic disciplines alone” (p. 282); “voice,” evaluating the extent to which alternative voices are heard in the text; “critical subjectivity,” evaluating the self-awareness of the researcher; “reciprocity,” reflecting the idea that the relationship between researcher and participants should be reciprocal and not hierarchical; “sharing the perquisites of privileges,” addressing how privileges gained from the research are shared with participants; and finally “sacredness,” or the extent to which the work shows a “profound concern for human dignity, justice, and interpersonal respect” (p. 284) through the equalling of power and collaboration with participants.
The criteria presented in this section contrast sharply with conceptions of quality deemed important in the traditional period and modernist phase moments, a contrast that stems from attempts to create criteria and considerations that are commensurate with ontological and epistemological stances of the alternative paradigms. During this period, it was argued by many authors that the swelling of alternative paradigms, theories, and methods of conducting research meant that work could no longer be evaluated the same way, which led to the proliferation of various paradigm-specific criteria. Although such criteria created space for various forms of qualitative research to develop, they were also critiqued as difficult to operationalize and as leaving researchers and evaluators with multiple, sometimes contradictory, frameworks for assessing quality (Lincoln et al., 2011). This growth, in combination with mixing between the social sciences and humanities, led some authors to argue for an individualized approach to quality assessment.
Third Approach: Individualized Assessment
The idea that each piece of qualitative research should be assessed individually can be seen as arising partly as a response to the growing diversity of qualitative research, and also as part of blurring between the social sciences and humanities discussed by Denzin and Lincoln (2005) in the moments of the postmodern period of experimental ethnography and postexperimental inquiry. In light of the crises of the previous moments, some fields looked at new ways of writing and representing their work, such as through poetry and autobiography, as well as performative and visual modes. The methodologically contested present, Denzin and Lincoln's seventh moment, was then characterized by conflict, tension, and even retrenchment. Collectively, these three moments occurred from approximately 1996 to 2004. In these moments, new and creative forms of representing research contributed to the re-thinking of quality criteria by some authors. Table 3 provides a summary of the concepts and terms that will be discussed in this approach:
Concepts and Terms Associated with Individual Assessment Quality Criteria
Perhaps the most overt call for the individualized assessment of qualitative research is Schwandt's (1996) article entitled, “Farewell to Criteriology.” In this article, he insisted that social inquiry should be defined in terms of a practical philosophy and evaluated based on three different considerations. The first consideration is whether or not the inquiry “generate[d] knowledge that complements or supplements rather than displaces lay probing of social problems” (p. 69). Second, social inquiry should cultivate “critical intelligence.” This form of intelligence is differentiated from “operational intelligence” focussed on strategy and procedure, and it involves enhanced moral judgement about the value of research findings. The third consideration discussed by Schwandt is the extent to which the social inquiry “enable[s] the training or calibration of human judgement” (p. 69). In essence, Schwandt is calling for qualitative research to be judged based on its usefulness for enhancing our ability to understand social problems and apply research findings.
Seale (1999a, 1999b, 2002) called for a more pragmatic view of quality criteria, arguing for an emphasis on methodological awareness rather than paradigm-specific criteria and adherence to “the obligation to fulfill philosophical schemes through research practice” (1999a, p. 466). This type of awareness “involves an enhanced capacity to anticipate a broad range of potential criticisms that may be made of a final research report” (2002, p. 108). Methodological awareness can be developed through experience in a particular field, including various apprenticeship experiences where researchers have the opportunity to appraise the strengths and weaknesses of literature published in their field. Therefore, Seale asserts that quality arises from the implementation of research by methodologically aware researchers and that this, in turn, can be appraised by the reader.
Sandelowski and Barroso (2002) proposed a shift from conceptualizing quality as specific standards, criteria, or techniques to seeing quality in terms of attributes of the research report. More specifically, they “hope[d] to shift the debate … from a preoccupation with epistemic criteria toward consideration of aesthetic and rhetorical concerns” (p. 75). Drawing from reader-response theory, they discussed the importance of aesthetic criteria, such as coherence and economy, and the need to render the text convincing to a range of audiences. This type of evaluation proposal is certainly in line with the merging of the social sciences and humanities in areas of qualitative research; however, how to appraise such criteria becomes challenging. For example, Sandelowski and Barroso themselves provided quality checklists as an appendix to their article, addressing topics such as the research question, method, sampling, data collection and management, validity, and ethics, among others, which seem incongruent with the call to focus on aesthetic and rhetorical concerns.
Rolfe (2006) explicitly stated that it is necessary to assess qualitative work individually, questioning any predetermined criteria for quality. More specifically, he said that “each research methodology (and perhaps each individual study) must be appraised on its own merits” (p. 310) because of the diversity of qualitative research. In line with Sandelowski and Barroso (2002), Rolfe argued that quality is both “revealed” and “resides” in the research report, placing responsibility for judging quality on the reader, in addition to the researcher (p. 309). Similar to Seale (1999a, 1999b, 2002), Rolfe believed that the ability to evaluate qualitative research is something that comes with experience and that “appraisals cannot be made by novice researchers merely by following a set of critical guidelines or criteria” (p. 309).
Authors arguing for an individualized approach to quality assessment shifted their gaze away from methods-based or technique-focused appraisals and paradigmatic considerations, towards consideration of features of the research text and the interpretations presented in relation to the research approach used. Although this allows for diverse approaches to qualitative research, greater demands are placed on the reader of qualitative research to be experienced and knowledgeable as a researcher. Thus, this approach is particularly challenging for those readers who are not “experts” in qualitative research, and it also provides no means for readers to come to a consensual agreement on or perhaps even dialogue regarding the quality of research.
Fourth Approach: Bridging Criteria
In the present moment, Denzin and Lincoln (2005) discuss the backlash associated with qualitative methods within “‘Bush Science’ and the evidence-based social movement” (p. 20), a moment that started roughly in 2005. Looking forward, these authors predict a fractured future where methodologists will divide themselves; quantitative researchers on one side where randomized trials stand as the gold standard and qualitative researchers on the other side where “socially and culturally responsive, communitarian, justice-oriented” (Lincoln & Denzin, 2005, p. 1123) work is completed. In the latest Sage Handbook of Qualitative Research (2011), Denzin and Lincoln continued to voice concern regarding the contemporary “politics of evidence” (p. 2), pointing to the positivist resistance to qualitative research in the alternative paradigms within an “increasingly conservative, neoliberal global environment” (p. 13). With regards to legitimization and quality assessment, if qualitative research is to stand united in this fractured future, we, the authors of this article, contend that conceptions of quality are needed that build points of connection, or bridges, across paradigms. More specifically, building on the previous approaches, we propose that flexible criteria are needed that are unique to, and unify, qualitative work and, at the same time, are sensitive to diversity within and between paradigms and methodological approaches. Such criteria would point to considerations of relevance to all qualitative researchers, while at the same time acknowledging that how the considerations are addressed will vary in relation to the paradigmatic, theoretical, and methodological locations of a particular qualitative project. This bridging approach can be seen in several recently published pieces, including the work of Ballinger (2006), Morrow (2005), Tracy (2010), and Whittemore, Chase, and Mandle (2001). As discussed below, many similarities exist across the criteria, considerations, or standards proposed by such authors, signalling that a movement towards a general consensus or agreement has already begun on what criteria are important for, and can connect various forms of, qualitative research.
Ballinger (2006) provided a number of “pointers” or “considerations,” consistently indicating that the application of these considerations needs to be done in relation to the paradigmatic and methodological approach taken in a particular study. Namely, she described how there should be “coherence,” or a matching between the epistemology of the researcher with how the study is carried out and the knowledge claims forwarded. There should be evidence of “systematic research conduct,” which would consider the use of appropriate research techniques based on the specific approach used. There should also be “convincing and relevant interpretation,” which involves “recognition by the reader of the research account that the research has something significant to contribute to knowledge within the domain under investigation” (p. 241). Finally, the study should account for the role of the researcher, which is where reflexivity becomes an important part of the research process. These criteria do not position one paradigm over another, nor are they specific to any one paradigm or approach. They are relevant to the evaluation of a variety of types of qualitative research, which is why they meet the definition of a bridging approach to quality assessment.
Morrow (2005) takes a slightly different approach by advocating for “paradigm-specific criteria” that are supplemented by “transcendent criteria.” She discussed paradigmatic criteria for the post-positivist, interpretivist/constructivist, and critical/ideological paradigms. In one sense, these paradigm-specific criteria parallel Ballinger's (2006) emphasis on epistemological coherence given that these criteria are largely epistemologically-based. At the same time, Morrow's transcendent criteria provide another example of a bridging approach, with several of these criteria also having similarities to some of Ballinger's (2006) considerations. For example, Morrow discussed “social validity” and “adequacy of interpretation,” referring to the need for all qualitative research to be assessed based on its social importance and the completeness and clarity of its findings and interpretations. These criteria resemble Ballinger's call for qualitative research having convincing and relevant interpretation. A third transcendent criterion discussed by Morrow is “adequacy of data,” where research is to be judged based on the depth, quality, and variety of data gathered, including discrepant data, to ensure thoroughness. The way in which this data is gathered will depend on the specific approach taken and methods employed, and therefore it relates to Ballinger's criterion of systematic research conduct. The final transcendent criterion discussed by Morrow, “subjectivity and reflexivity,” is also discussed by Ballinger. Both authors emphasized that addressing researcher reflexivity is important to allow the reader to assess the ways in which the researcher's values and experiences have influenced the shaping of the research and the resulting interpretations, again acknowledging that how and why this is done will vary in relation to the paradigmatic positioning of a study.
Whittemore et al. (2001) distinguished between “primary” and “secondary” quality criteria in qualitative research. They stated that primary criteria, parallel to Morrow's (2005) notion of transcendent criteria, “are necessary to all qualitative inquiry; however, they are insufficient in and of themselves” (Whittemore et al., 2001, p. 529). Primary criteria include “credibility,” “authenticity,” “criticality,” and “integrity.” In their view, credibility relates to ensuring that interpretations are accurate, which is directly tied to their authenticity criterion in that research should remain true to the meanings ascribed by individuals to phenomenon and situations.
Whittemore et al.'s criticality criterion refers to the researcher's own self-criticality in the form of reflexivity, as well as being critical of the data collected to help ensure completeness. Criticality is also related to their final primary criterion, integrity, where there should be evidence that interpretations were checked and findings presented modestly. Parallels between these criteria and those presented by Morrow (2005) and Ballinger (2006) can be drawn. For example, reflexivity is deemed an essential quality criterion by all of these authors, and credibility, authenticity, and integrity relate to Morrow's adequacy of data and adequacy of interpretation criteria as well as Ballinger's systematic research conduct criterion.
Whittemore et al. (2001) presented their secondary quality criteria as additional indicators of quality to be applied as relevant to a particular study. Similar to their primary criteria, parallels can be drawn between these secondary criteria and the criteria proposed by Morrow (2005) and Ballinger (2006). For example, both Whittemore et al. and Ballinger discuss “congruence” as the fit between different parts of the project, including its research questions, methods, findings, and philosophical underpinnings. Other secondary criteria discussed by Whittemore et al. include “explicitness,” referring to the presentation of the research and the provision of an appropriate record of how the research was conducted; “vividness,” referring to the clarity of descriptions provided, as well as their depth and faithfulness; and “creativity,” referring to the use of novel methodological designs to effectively address research questions. These criteria resemble Morrow's call for social validity and adequacy of interpretation, and Ballinger's call for systematic research conduct. Finally, “thoroughness” refers to the use of comprehensive sampling and data collection and analysis, and “sensitivity” refers to the inclusion of multiple and varied perspectives as well as appropriate ethical considerations throughout the project. These criteria relate to Morrow's adequacy of data and adequacy of interpretation criteria, and Ballinger's systematic research conduct.
Tracy's (2010) eight “big-tent” criteria for excellent qualitative research provide a final example of bridging criteria. These cross-paradigmatic criteria were designed to be used for pedagogical reasons and to help in “garnering respect for qualitative methods from power holders who know little about our work” (p. 837). The naming of these criteria as “big-tent” criteria is in reference to the call by Denzin (2008, 2009) for greater unification within qualitative research in the face of a fractured future. Tracy positions her criteria as ones that can be flexibly applied. That is, her criteria are flexible to accommodate the diversity of approaches to qualitative research, allowing for more appropriate and fair evaluation.
Again, the move towards consensus is seen when Tracy's (2010) criteria are compared to those of the other frameworks for bridging criteria. Three criteria proposed by Tracy include having a “worthy topic,” the research having “resonance,” as well as offering a “significant contribution.” To expand on these criteria, the topic should be relevant, timely, and interesting; the research should have the capacity to influence or move readers; and it should contribute significantly in at least one of a number of ways, such as conceptually, practically, morally, or methodologically. These three criteria align well with Morrow's (2005) social validity criterion, Ballinger's (2006) convincing and relevant interpretation criterion, as well as Whittemore et al.'s (2001) explicitness, vividness, and creativity criteria. Tracy (2010) also discussed “rich rigor,” referring to the need for sufficient and appropriate data collection and thorough and transparent data analysis. A related criterion is “credibility,” which Tracy described as being developed through the collection of thick descriptions, multiple perspectives, and member reflections. These two criteria are similar in principle to Morrow's (2005) adequacy of data and adequacy of interpretation, Ballinger's (2006) systematic research conduct, and Whittemore et al.'s (2001) credibility, authenticity, integrity, thoroughness, and sensitivity. Another big-tent criterion that bears resemblance to criteria proposed by Morrow, Ballinger, and Whittemore et al. is “sincerity.” With this criterion, Tracy described the importance of researcher reflexivity where he/she is transparent with his/her own values and biases, as well as transparent with any problems or issues that arose during the process of the project. Sincerity, then, is similar to Morrow's subjectivity and reflexivity criterion, Ballinger's account of researcher role criterion, and Whittemore et al.'s criticality criterion.
The final two big-tent criteria proposed by Tracy (2010) are “meaningful coherence” and “ethics.” Meaningfully coherent studies, as defined by Tracy, involve four main features. These studies: “(a) achieve their stated purpose; (b) accomplish what they espouse to be about; (c) use methods and representation practices that partner well with espoused theories and paradigms; and (d) attentively interconnect literature reviewed with research foci, methods, and findings” (p. 848). Meaningful coherence is therefore espousing an appropriate fit between the different components of the study and is then related to Ballinger's (2006) coherence criterion and Whittemore et al.'s (2001) congruence criterion. Finally, Tracy addressed the incorporation of ethics, including procedural, situational, relational, and exiting ethics. Whittemore et al. also discuss ethics in assessing the quality of qualitative research within their sensitivity criterion.
The bridging criteria discussed in this section share many common emphases (see Table 4), highlighting an emerging consensus on criteria to be used to assess diverse forms of qualitative research. As such, bridging criteria may provide a means for qualitative work to survive and thrive within the fractured future and to take control of the standards by which it is judged, while creating space for valuing diverse forms of inquiry. A problem with these criteria, however, is that it appears authors are using many different terms to essentially describe the same markers of quality. A more common and accessible language to describe these criteria is needed to promote their adoption and acceptance by qualitative researchers, the research community more generally, and research stakeholders.
Comparable Concepts and Terms Associated with Bridging Criteria of Quality Assessment
We agree with Tracy (2010) that a common language framework for quality is needed amongst qualitative researchers if we are to make qualitative research more attractive to a variety of stakeholders, including funding agencies, policy makers, and society in general. We reiterate that this framework needs to be developed in ways that enable points of connection across various approaches to qualitative research, while simultaneously enabling and fostering diverse ways of thinking about and doing such research. There is still much room for discussion on which terms should be used in describing a bridging approach. In an attempt to synthesize the bridging criteria presented, and move the discussion forward, we have created five categories to describe the common emphases alluded to throughout this section. These categories are presented in Table 5. Accompanying the categories are examples of questions that might be asked when assessing a piece of qualitative research.
Categories of Bridging Criteria and Associated Questions
This is not meant to be an exhaustive list of questions, but should help to guide readers in understanding the types of questions that might be asked and to spark dialogue regarding the meaning of the categories. In maintaining the flexible nature of these bridging criteria, the questions asked and responses will be dependent not only on the area of research under investigation but the specific approach (paradigm, methodology, and methods) used. For example, thoroughness and transparency will mean different things in a constructivist grounded theory compared to a critical ethnography or participatory research. We, therefore, see it as the responsibility of the authors to describe how quality was achieved in terms that can be understood on a broader level, that is, in a bridging manner.
Conclusion
Many misconceptions about qualitative research exist, for example that it is not “good” science because it does not test hypotheses and is too subjective (Kvale & Brinkmann, 2009). These misconceptions may often be grounded in inadequate knowledge, on the part of groups or individuals, about qualitative research and how to assess it. In searching for an approach to assess the quality of qualitative work, individuals may be inundated with an overabundance of terms and varying conceptions of quality. Given these conditions, individuals may assess qualitative research using inappropriate criteria, thus devaluing the research and its contribution. It is, therefore, essential that one has an adequate understanding of qualitative research in order to ensure that appropriate quality criteria are chosen and applied correctly. As such, education will remain an important tool for qualitative researchers to use, including the education of policy makers given the political nature of quality criteria for scientific research (Denzin, 2009; Seale, 1999b; Smith & Hodkinson, 2005).
Although the four different approaches to quality presented in this article appear to have clearly defined boundaries, they do not. At times, it was difficult to position the criteria proposed by certain authors into these categories. For example, the primary criteria of Whittemore et al. (2001) may certainly be critiqued as swaying towards post-positivism in their focus on accuracy and agreement, but because of the flexibility of their secondary criteria we believe that they fit best within the bridging criteria. Similarly, the considerations proposed by Schwandt (1996) may be viewed as meeting the flexibility required of bridging criteria, but they were deemed more appropriate for the individual assessment category because of their focus on how a particular study enhances intelligence and judgement. As well, although examples incorporated into this article appear to fit within these four approaches, the literature on quality and qualitative research is vast and may extend beyond the four approaches outlined.
Within this article, the historical moments in qualitative research discussed by Denzin and Lincoln (2005) were used to provide a means to organize existing conceptions of quality in qualitative research. More specifically, four varying conceptions, that is, qualitative as quantitative criteria, paradigm-specific criteria, individual assessment, and bridging criteria, have been offered as means by which to navigate this quality maze. In addition to helping us organize conceptions of quality, these moments also illustrate how politics can influence definitions of quality. For example, politics have played an important role in our present moment, “‘Bush Science’ and the evidence-based social movement” (Denzin & Lincoln, 2005, p. 20), and will continue to play an important role in the fractured future.
The bridging criteria recently proposed by a number of qualitative researchers can help to address criticisms located in the resurgence of positivist thought or neopositivism (Cheek, 2008), and can provide a solid foundation to articulate the social and scientific contributions of diverse forms of qualitative research. Categories of common emphases of these criteria have been presented, including the social value and significance of the research, the thoroughness of data collection and interpretation, the transparency and reflexivity of the authors, the coherence of the research approach, and due regard for the research participants. The criteria deemed important in these categories, and associated questions asked of the research, will depend on the research area and approach used. It is hoped that this review and synthesis of quality in qualitative research will spark continued discussion on the development of flexible guidelines for assessing this type of research, given its rich diversity, its potential to key contemporary social issues, and the current sociopolitical environment in which we find ourselves.