Abstract
Due to the diversity of research options, as well as the lack of consensus regarding criteria for research quality, an evaluative process is needed that can take into account the different characteristics of diverse research options and their specific contributions. Therefore, it is essential to carry out the evaluation of qualitative research in a pluralistic setting. The author’s observation is that research needs to be understood as a political concern within the domain of justice. This means that it is necessary to move forward with the construction of a more democratic setting so that differences between scientists and scientific communities can be dealt with, and the right of researchers to promote and develop different paradigms and research options can therefore be guaranteed, as well as facilitating dialogue and mutual learning between researchers and different paradigms. Its construction will imply discussions, tensions, and agreements that go beyond the epistemological and theoretical sphere into a political and moral one.
Introduction
The evaluation of qualitative research has traditionally been a source of conflict because it confronts the conventional criteria used to evaluate quality, generally referred to as “validity,” which are sustained by the positivist model of science (Connelly & Clandinin, 1995; Denzin & Giardina, 2008; Lincoln, 2005; Lincoln & Denzin, 2005; Reason & Bradbury, 2001; Smith & Hodkinson, 2005, 2009). However, going beyond the struggle between the quantitative community and the qualitative community, there is a growing controversy among qualitative researchers depending on their level of attachment to or rejection of such criteria, and even more so depending on their position relating to the concept of quality in research per se (Barbour, 2001; Denzin & Giardina, 2008; Sparkes, 2001). This situation is due to the upsurge of multiple research currents and approaches, which themselves are products of disparate epistemological, political, and moral influences.
How qualitative research should be evaluated in a pluralistic scientific setting is a relevant issue, since within the scientific field, which is made up of hierarchies and power struggles for legitimacy (Fernández, 2004), some research perspectives and academic communities have been ignored or marginalized (Lincoln, 2005). Likewise, Lincoln put forward that the imposition of tight criteria for research methods due to overly conservative and rigid theoretical and epistemological approaches has generated discrimination against the innovative and transdisciplinary use of certain qualitative methods and that this has affected their development and even, as sustained by Barbour (2001), the specific contribution that these types of research can make.
This discriminative situation has been reflected in a lack of approval and funding for alternative research projects. In addition, researchers, especially students who are attracted to new types of research, hold back in fear of an evaluative process that does not consider the epistemological characteristics of this type of research. Similarly, academic freedom and the teaching of new types of research are affected by this lack of recognition (Lincoln, 2005). All of these situations are manifestations of injustices that take place in the field of science. They are injustices that affect academic communities and researchers who defend certain theoretical, epistemological, and political perspectives.
Furthermore, the dogmatic and domineering positions regarding the concept of “rigor” and regarding the quality of research in general, which are held within the concept of a certain type of scientific method, have generated disputes that have led to a breakdown in communication between researchers and academic communities. Differences are not resolved constructively in an environment with very little openness, meaning that, in general, possibilities for learning between communities and researchers with disparate concepts and positions are blocked.
This confrontation and poor communication between researchers and academic communities take place at a point in time at which the epistemological, ethical, and political foundations of science are being questioned, which is affecting its legitimacy. Since the conditions described in the above situation constitute injustices, they become further elements that damage the aforementioned legitimacy.
In this article, I examine the debate surrounding the evaluation of qualitative research to put forward some ideas that could promote reflection on the theme. The article has been divided into four sections. In the first section, I defend a polysemic and pluralistic perspective of qualitative research, which is necessary to understand its evaluation. In the second section, I present the evaluation of qualitative research within a broad context to demonstrate the lack of consensus surrounding the topic. In the third section, looking at qualitative research within a pluralistic perspective, I propose that it is necessary to view the scientific field as a public setting that could guarantee the rights of the diverse research options in a more democratic environment. In the final section, Conclusions, I reflect upon the possibilities that are opened up when the issue of the evaluation of research is taken out of the epistemological sphere and placed in the sphere of justice.
Qualitative Research: A Polysemic Issue
Qualitative research takes on a polysemic character for researchers with different theoretical, epistemological, and ethical perspectives, which themselves are also products of academic background, race, ethnicity, social class, gender, nationality, and even age differences (Denzin & Giardina, 2008; Denzin & Lincoln, 2000, 2005; Sparkes, 2001).
For some, qualitative research has become a reformist movement (Schwandt, 2000) that expresses a critical view toward the positivist paradigm, which is hegemonic in social and human sciences. This movement has grown and its critical voice has become stronger under the influence of constructionist, critical, feminist, queer, and ethnic research models, as well as cultural studies (Denzin & Lincoln, 2005). Postpositivism, poststructuralism, and postmodernism have offered alternative ways of understanding research and have become important influences for the movement.
All of these influences, as well as the diverse theoretical traditions originating from the distinct disciplines within social and human sciences, have brought forth multiple paradigms that propose different ways of viewing reality and truth, thus creating tensions and contradictions (Denzin & Lincoln, 2005). Some researchers agree that there ought to be some kind of relationship between research findings and an external reality that is difficult to grasp (Hammersley, 1992, 2009; Strauss & Corbin, 2002). Others, working from a nonfoundational epistemological perspective assume that there is no ultimate external parameter from which we can establish a neutral and objective viewpoint and that no observation can be free from a theoretical context, meaning that subject–object duality is impossible (Feyerabend, 1975; Gadamer, 2006; Habermas, 1987; Smith & Hodkinson, 2005, 2009).
Similarly, some researchers are looking at the construction of substantive theory in the hope of establishing some kind of generalization (Corbin & Strauss, 2002). Others, however, are interested in understanding the singularity of a case within its cultural, social, and historical context. These researchers view research as a cultural process that should be useful for members of particular communities and responds to some specific criteria that are relevant to those communities (Denzin & Lincoln, 2005).
Another point of divergence for qualitative researchers is the purpose of research, which generates political and ethical tensions (Denzin & Giardina, 2008) as can be found between those researchers who focus on the construction of knowledge for the theoretical development of the discipline, those who view research as a means of impacting programs and policies, those who direct their research toward the transformation of people and society, and finally, those who work to shed light on everyday issues and tensions within society (Hammersley, 2005, as cited in Denzin & Giardina, 2008).
Some researchers focus on a struggle with data and instruments in order to create products that can be related to “an” external reality (Corbin & Strauss, 2002). From this point of view, objectivity is supported by data. Others, however, confront this position and understand objectivity from an intersubjective perspective, as a product of dialog or argumentation (Gadamer, 2006; Habermas, 1987). Others, following critical theory (Habermas, 1990) and Foucault’s line of thinking, assert that a neutral science does not exist and that knowledge is always managed by concrete interests who are responding to positions of power (Clandinin & Rosiek, 2007). Although some believe that method is central to rigorous research, others talk about going beyond “methodolatry” to focus on the essence of research: the outcome, the object (Adorno, 2006; Janisek, 1994). Researchers with a humanist commitment to social justice also confront those who seek to carry out research with no political alignment. From this epistemological perspective, some researchers consider that if the goal is to bring an end to the domination that occurs in the relationship between the researcher and the participants, empower people, and work toward social justice, then research ought to be participative (Reason & Bradbury, 2001, 2008). For them, the criteria used to evaluate the quality of research have to include the research process itself and not only its results, that is, whether the procedures carried out promote people’s empowerment and participation and whether the results of the research project will serve to help the participants to improve their personal and social conditions and remedy injustices.
Ethical tensions are thereby generated with regard to the researcher’s role in relation to participants’ situations as well as to interactions and the development of relationships with people within their natural surroundings. Some researchers consider that a researcher should not confuse his or her role with that of a practitioner and that although the relationship with participants may be warm and cordial, from a moral point of view, the researcher must not interfere with the participant’s problems and those of their community (Hammersley, 1992). In contrast, others consider that research must be morally committed to social change and that the researcher must therefore get involved in political and community actions (Kincheole & McLaren, 2005; Lincoln & Guba, 2000). Similarly, some researchers confront the behavior of certain institutional ethical review committees who, though claiming to follow bioethical parameters, are actually more interested in protecting the institutions and the researchers than the participants (Christians, 2005).
This wide variety of epistemological, ethical, and political stances are reflected in the growing diversity of research proposals, which are influenced by the discipline they pertain to, as well as by the personal circumstances of the individual researcher (class, gender, age, ethnicity and nationality, among others). This process of expansion, which emerges from a critical perspective and which recognizes science’s plurality, is confronted by other movements that are going in the opposite direction, toward the establishment of homogeneous visions that require enforcement mechanisms. A movement known as “evidence-based science,” which is based on the conventional canons of science, could be included in these. This constitutes a new attempt to unify science after Comte and has become a new source of conflict, as expressed by Denzin and Lincoln (2005, p. XV), “the open-ended nature of the qualitative research project leads to a perpetual resistance against attempts to impose a single, umbrella-like paradigm over the entire project.”
This situation has become even more complex with the emergence a third way–style movement, known as “mixed methods” (Tashakkori & Teddlie, 2003). This movement, which is based on a pragmatic view of research, views the aforementioned epistemological discussion as superfluous, arguing for the preeminence of the instrumental (Patton, 2002; Teddlie & Tashakkori, 2003), that is, “what works” (Creswell & Plano, 2007, p. 24). Mixed methods can therefore be understood to be a series of methodological techniques that can be combined in response to a particular problem. The nature of the problem defines the method, which can be understood as instruments and techniques that may “be separated from the epistemology out of which they have emerged” (Patton, 2002, p. 136).
The Evaluation of Qualitative Research: A Contested Terrain
This brief description aims to put the current debate regarding the evaluation of qualitative research into context, a debate that can be identified in two closely related spheres. On one side is the great explosion of positions that are divergent to traditional positivist research methods. On the other side is the definition of evaluative criteria for scientific production in qualitative research.
With respect to the first area, it is necessary to recognize the relevance that is given to the evaluation of the method as one of the central criterion of quality within the positivist scientific tradition: rigorous application of the method. This is an increasingly complicated sphere that spans multiple conflicting positions. An example can be observed in ethnography were there are multiple approaches, with differing positions such as those found in the ethnographical works of Geertz (1993), Hammersley and Atkinson (1994), Goetz and LeCompte (1988), Wollcott (2003), Angrosino and Kimberly (2000), Tedlock (2000), Agar (1996), and Denzin (2003), to name just a few. Therefore, we can no longer refer to ethnography in the singular but, instead, to “multiple ethnographies.” It is not in vain that Wolcott (2003) suggests the nonexistence of one or several of ethnography’s indispensable characteristics or components, stating that in fact the essence of ethnography depends on the ethnographer. Agar suggested that it is easier to say what ethnography is not than what it really is, a concern that could be even more complex today.
Concerning the establishment of evaluative criteria for qualitative research, the literature captures the vigorous debate that has taken place over the past two decades (Barbour, 2001; Calderón, 2009; Connelly & Clandinin, 1995; Denzin & Giardina, 2008; Fals-Borda, 2001; Flick, 2006; Hammersley, 1992, 2009; Lincoln, 2005; Lincoln & Denzin, 2005; Mays & Pope, 2000; Morse, 1999, 2006; Reason & Bradbury, 2001; Smith & Hodkinson, 2005, 2009; Sparkes, 2001; Steinke, 2004; Stige, Malterud, & Midtgarden, 2009; Whittemore, Chase, & Mandle, 2001). The debate has been fueled by the rise of “evidence-based” research, which is viewed by some as a fresh assault from the circles of power who are seeking to “adapt” research to the canons of positivist science (Denzin & Giardina, 2008; Lincoln, 2005). Within this discussion are some who defend validity and reliability as criteria for all types of research, even though these need to be adapted to qualitative research (Calderón, 2009; Hammersley, 2009; Mays & Pope, 2000; Whittemore et al., 2001). Indeed, for some, these are the only ways to guarantee research’s rigorousness, without which it would be impossible to talk of science (Morse, 1999; Whittemore et al., 2001). Those who advocate the development of specific criteria for qualitative research (Barbour, 2001; Connelly & Clandinin, 1995; Denzin & Giardina, 2008; Lincoln, 2005), as well as claiming the epistemological incommensurability of positivism and alternative perspectives, argue that the criteria of validity and reliability are an affront to the development of qualitative research because they limit its creative and critical nature and even restrict the freedom to teach qualitative research (Denzin & Giardina, 2008; Lincoln, 2005). These researchers therefore reject the term “validity criteria” and tend toward a more open and inclusive concept: “quality criteria.”
This situation also reflects the contradiction between those who seek the general scientific community’s recognition (Hammersley, 2009; Morse, 1999, 2006; Steinke, 2004) and those who aspire toward the dissolution of a unit that is impossible to maintain intact (Hatch, 2006, cited in Denzin & Giardina, 2008) in order to construct alternative academic communities (Denzin & Giardina, 2008). These latter academic communities and their ideas have a growing presence in some undergraduate and graduate curricula, and there has been an emergence of publications (books and periodicals), conferences, and associations organized by their members (Lincoln, 2005).
At this point it is necessary to turn to Bourdieu’s description of science as a field (Bourdieu, 1998), that is, as a dynamic social space that is structured according to hierarchies and fixed rules (Téllez, 2002). Within this space, the truth constitutes a source of conflict within a framework of positions and power related to the distribution of scientific capital (Fernández, 2004). Likewise, Fernández argues that the scientific field’s autonomy is affected by external forces (political power) and by internal forces (acceptance of the game 1 ). Similarly, for Smith and Hodkinson (2005), the criteria for evaluating qualitative research and decisions about how these should be put into practice are the product of complex interactions within a social context marked by power and politics.
There are many institutions that have established guidelines for the evaluation of qualitative research. Within U.S. research communities, these include: The Institute of Education Science (IES) within the U.S. Department of Education; the What Works Clearinghouse (WWC); the Cochrane and Campbell Collaboration (CCC); the National Research Council—SBR (Science-Based Research) framework; the 2006 standards for reporting used by the American Education Research Association (Denzin & Giardina, 2008, p. 16), among others. In England these types of guidelines can be found in institutions such as The British Sociological Association—Medical Sociology, and the National Institute of Health Office of Behavioral and Social Sciences, for the field of Public Health (Flick, 2006, p. 381). But as Reicher (2000, cited in Flick, 2006, p. 381) asserts, these guidelines do not constitute a consensus on the various forms of qualitative research. For Denzin and Giardina, “Standards for assessing quality research are pedagogies of practice: moral, ethical, and political institutional apparatuses that regulate and produce a particular form of science, a form that may be no longer workable in a transdisciplinary, globalizing, and postcolonial world” (Denzin & Giardina, 2008, p. 13).
We can analyze the definitions of criteria and guidelines used for the evaluation of qualitative research from two confluent perspectives. The first, proposed by Sparkes (2001), takes into account their relationship with conventional criteria. The second, proposed by Denzin and Giardina and based on the ideas of Smith and Deemer (2000, cited in Denzin & Giardina, 2008, p. 43), is structured according to their natures, be it foundational, quasi-foundational, or nonfoundational. An integrated overview of both theories is laid out below.
Sparkes (2001) suggests that four positions exist: replication, parallel, diversification, and “letting go.” He uses the term “replication” to represent the way in which evaluation criteria replicate conventional criteria by presupposing that a set of criteria that are applicable to all types of research exists: objectivity, internal validity, external validity, and reliability. This corresponds to the “foundational” position identified by Denzin and Giardina (2008) that focuses on the rigorous application of methodology because methodology has been claimed to be the defining factor in isolating researcher’s subjectivity.
Sparkes (2001) uses the term “parallel” to describe those positions that do not accept conventional standards of validity and reliability, considering them to be inappropriate for qualitative research, although they themselves end up creating “adaptations” of positivist empirical criteria. As in the first position, these criteria also stem from a perspective that emphasizes the rigorous application of methodology. Sparkes believes that Silverman’s (2005) work could also be included within this standpoint. His criteria are based on the refutability principle, the constant comparative method, comprehensive data treatment, deviant-case analysis, and the use of appropriate tabulation. In a similar vein are Guba and Lincoln’s proposals regarding naturalistic inquiry (Denzin & Giardina, 2008, p. 15; Hammersley, 1992, pp. 63-64): credibility (commitment to true value-internal validity), transferability (applicability-external validity), dependability (consistency-reliability), and confirmability (neutrality-objectivity).
Sparkes (2001) uses the term “diversification” for those positions that reject the traditional concept of validity because their vision of truth is based on notions of coherence and pragmatism rather than correspondence. Because there is no universally accepted notion of objective truth, it is assumed to be socially constructed within particular communities and discourses and within specific historical frameworks. Therefore, for these researchers, the establishment of fixed criteria for evaluating qualitative research is not possible, and instead, they adopt an open perspective toward multiple approaches on validity. Some examples of criteria within this position are given below.
Researchers working from a participatory and critical perspective (Lincoln & Guba, 2000; Reason & Bradbury, 2001; Thorne, 1997, cited in Sparkes, 2001) identify the validity of research in terms of its effectiveness in empowering participants and enabling them to create lasting changes concerning themselves and their reality. They also take into account the extent to which the research process promotes the construction of democratic relationships between participants and the strengthening of cultural identity, participation, and community organization. Lincoln and Guba (2000) advance “ontological and educative authenticity” as the process of awareness that participants undergo as a result of the research project, and “catalytic and tactical authenticities” as the degree to which research promotes action on the part of research participants, as well as researchers’ social and political educative work in response to the participants’ demands.
From a postmodern perspective, Lather (1993, cited in Sparkes, 2001) proposes criteria that transgress the correspondence notion of validity, including ironic validity and paralogical validity. Parologic validity comes from Lyotard’s concept of paralogy: “scientific knowledge . . . aims at discovering differences and at leaving contradictions in their tension” (Flick, 2006, p. 373). Following on from Baudrillard’s work, Lather states that ironic validity is obtained through the construction of simulations, as copies without originals, which replace representations as copies of real objects (Flick, 2006, p. 373).
Therefore, from a “diversification” position, quality criteria aim to give preeminence to research processes and findings. It becomes a personal and interpersonal matter rather than a methodological issue (Reason, 1981, cited in Sparkes, 2001). This position corresponds to what Denzin and Giardina (2008) call the “quasi-foundational” position.
The fourth position, which Sparkes calls “letting go” and Denzin and Giardina term “non-foundational,” suggests that we should abandon the concept of validity. Wollcot (1994, cited in Sparkes, 2001), for example, rejects validity as a criterion for the evaluation of his work because he prefers to focus on identifying critical elements along with plausible interpretations of these rather than seeking out the ultimate truth. For these researchers, the moral, ethical, and political consequences of their actions are of greater interest (Ellis, 1999, cited in Sparkes, 2001). They suggest criteria based on an ethical perspective such as care, love, trust, respect, and kindness (Denzin & Giardina, 2008). Following on from this position, Lincoln and Guba (2000) propose criteria that take into account the viewpoint from which judgments are formed, the level of community participation when evaluating the quality of research, the way in which research gives voice to the different participants, and the manner in which privileges are shared. Convergence between Sparke’s and Denzin and Giardina’s perspectives is presented in Table 1.
Convergence Between Sparkes’ (2001) and Denzin and Giardina’s (2008) Perspectives on Criteria to Evaluate Qualitative Research
Qualitative Research Evaluation in a Pluralistic Setting
We are therefore confronted by science’s plurality, not only due to the differences between quantitative and qualitative research but also due to the differences within the latter (Calderón, 2009; Denzin & Giardina, 2008; Stige et al., 2009). As was set out in the previous paragraphs, this diversity implies great differences in the evaluation of research. Flick (2006) put forward that neither the use of the traditional criteria of positivist science nor the creation of alternative criteria have been satisfactory. This situation has led some researchers such as Stige et al. to believe that it is not possible to reach ontological, epistemological, and methodological consensuses within the framework of a pluralistic scientific setting. They consider that it would be possible to reach agreements regarding which themes warrant discussion. “Researchers might agree about what to discuss even though their positions in specific ontological, epistemological, and methodological debates vary in many respects” (Stige et al., 2009, p. 1506).
But if it is not possible to reach an agreement regarding ontological, epistemological, and methodological matters, how to go about evaluating research? To deal with this matter, it is necessary to broaden the lens we use to analyze what is happening in the field of science. This plurality is also found within the fields of philosophy, morality, and religion. Society’s plurality led Rawls (1995) to formulate the following question (which I have adapted to the subject of this article) within the framework of his ideas about justice: How could it be possible that a stable and just community of free and equal scientists, which is deeply divided by mutually exclusive “scientific doctrines,” 2 could exist?
According to Rawls and his theory of “political constructivism,” in ruling out agreements between all-encompassing doctrines because there is no general consensus between citizens (in our case, scientists), we can say that it becomes necessary to construct some principles of justice to regulate the relationship between researchers and between academic communities. Rawls states that these principles would be based on practical reason 3 and not on a theoretical reason. The aforementioned construction therefore is not an epistemological issue but rather a political one, in this case, the political conception of justice that provides a frame of reference for judgments about science.
Rawls thinks that if it is not possible to determine which is the true doctrine, we must work with the idea of “reasonable doctrines,” which in our case means those positions (doctrines) that agree that science should be maintained as a valuable asset to humanity, even though there are different conceptions of it. This is in contrast to Rawls’ notion of justice as fairness, within which it would be expected a political conception of justice is a product of a basic agreement and which, Rawls hoped, would serve as a starting point (overlapping consensus), I understand this construction as a process that is continuous, incomplete, and in constant change, affected by interests, pressures, forces of power, and emotions that require the development of a democratic scenario in which they can be dealt with. This idea would be closer to Sen’s (2010) political concept of justice, which departs from the idea of a social contract leading to just institutions but instead aims for the construction of a partial order with a limited articulation, orientated toward the resolution of a concrete injustice (in our case, the injustice that can be found in the field of science). Sen’s idea of justice comes out of a recognition of the existence of diverse interests, positions, and meanings, and it makes necessary a process of dialog and negotiation in order to overcome parochialism and dogmatism, which is required for the construction of agreements as partial, open, and changing solutions.
What does it mean, therefore, to say that the evaluation of research takes place within the terrain of justice, when this implies the construction of public agreements regarding the concept of the justice required? Viewing science within the context of justice and as a public setting brings up the necessity for the collective construction of a “scientific world.”
According to Arendt (1998), supporting society implies the construction of a shared world that corresponds to the public sphere, where it is possible to connect what is individual to what is collective without it losing its individuality and without excluding the special features of subjectivity (Ospina & Botero, 2007), and this could also be extended to science. Following Rawls (1995), it is necessary to construct a general overview of the way in which the aforementioned public sphere works, and of the scientist as a member of this public sphere, which will allow for the transformation of the current antagonistic positions into more constructive relationships between scientists and communities with divergent views. 4
As a public sphere, the scientific field, in words of Sánchez, needs to become a democratic arena based on dialogue between varying perspectives, all under the fundamental presupposition of communication for the construction of an intersubjective community (Sánchez, 2003) in which there is freedom of reasoning, debate, and deliberation and in which scientists are able to deal with their interests and rights according to an idea of public reason as a collective project.
Thinking of science as a shared world brings up great difficulties, not only due to the differences and tensions within science but also due to pressures and influences from the outside: the market and multinational companies, and political parties and the government. There is tension between the above situation and science’s condition as an institution and its interest in maintaining its legitimacy (Berger & Luckmann, 1968). It would therefore be necessary to view science as a sociocultural and historical practice that takes place within a social field affected by forces of power and interest that generate conflicts (Bourdieu, 1998; Fernández, 2004; Smith & Hodkinson, 2005). Due to the diversity of epistemological, theoretical, and political currents and approaches, the field of science tends to organize itself into academic communities, which their members identify themselves with to varying degrees. Discrimination and lack of recognition take place due to hierarchies and power games, generating injustice. This feeling of injustice leads researchers to fight for recognition and legitimacy. While researchers are willing to work cooperatively within their academic communities for this end, they would also be willing to interact more widely with other academic communities if they perceived that a more just scientific field existed. It is worth noting that the current unjust situation brings the legitimacy of the entire field of science into question. This adds to criticisms of the hegemonic scientific statute made by alternative movements that call for a science that is more modest, more human (referring to its moral, political, emotional, and even spiritual character), more open, more willing to reflect and self-criticize, and more democratic.
Regarding the second point (the scientist as a member of the public sphere, as a person), it is necessary to accept the scientist’s moral and political condition within his or her research practice. It is precisely because of the scientist’s condition as a human being that she or he is a subject who, as well as possessing concepts of good and of justice, possesses a moral capacity (Pieper, 1991). Every human act, including research practice, constitutes a moral act. It is therefore also necessary to consider the researcher’s dignity: as a human being. It would therefore be necessary to view research as a means for the researcher to express his or her moral and political capacity, as well as a means for personal and academic development, meaning that the researcher would be not just a means to an end but an end in her or himself (Kant, 1989). This is another point of conflict that requires ongoing analysis by science as an institution.
From another point of view, and following Lyotard (1998), a plural conception of science corresponds to a science in paralogic conditions, based in petit narratives, that is, local, historic, and changing temporary contracts. As trust in metanarratives for supporting a legitimate order in the scientific field is lost, a concern about justice arises in Lyotard. Truth and justice are not necessarily associated: Something can be true but not just. Rather than a theoretical language, a prescriptive language is needed, which refers to actions. Therefore, Lyotard also finds in practical reason, which demands a type of knowledge founded in phronesis (prudence), a way to approach legitimacy of science in a parologic situation. Then, at a political level, it is necessary to think how to make institutions more just, that is, how to make scientific field more just. So, if there are no legitimate theoretical criteria to judge all scientific products, judgments should respond to particular cases. Thus, as Gallagher (1993) states, “phronesis is a moral insight, a judgment as to what is to be done here and now in the concrete and ambiguous moral situation.”
In this order of ideas, Schwandt speaks about social inquiry from a practical philosophy perspective, which implies dialogical relationships and a critical reflection on the values of its practice. He speaks about a moral conception for evaluating social inquiry as a “guiding ideal,” instead of a set of formal procedures. For him, a “moral ideal of democracy . . . facilitates the dialogical encounter that is the exercise of social inquiry as practical philosophy” (Schwandt, 1996, p. 66).
Finally, now that fixed guidelines for evaluating research have suffered a loss of legitimacy, if the idea of practical reason and phronesis gains relevance, the figure of the evaluator will necessarily become predominant. As increasingly diverse epistemological and theoretical perspectives develop within qualitative research its evaluation becomes a complex task, in which the evaluator’s responsibility is vital, therefore turning it into a moral and political undertaking. The evaluation of qualitative research requires evaluators who have considerable experience and a profound knowledge of the research options and positions taken by the researcher. General evaluators who are able to evaluate all the options and positions that occur within qualitative research are now less prevalent. The evaluator is therefore expected to take also a moral position when she or he is making a decision about his or her own competence to evaluate a particular product.
Stige et al. talking about the importance of reflexivity and taking power relations into account in the evaluation process, states, “The roles of researchers and reviewers are of course not equal, but an agenda promoting a dialogic process of evaluation would invite reviewers to position themselves and would therefore require reviewer reflexivity” (Stige et al., 2009, p. 1513).
Conclusion
The evaluation of qualitative research is a complex matter that is of the utmost importance for the development and consolidation of qualitative research (Calderón, 2009). As put forward by Gadamer (2006), critical rationality constitutes a central characteristic of science, which has a special role in the evaluation of research. But it is also necessary to recognize that the evaluation of research does not take place in isolation from the conflicts of interest and powers that make up the academic and scientific field. In this setting, and in the struggle for resources, recognition, and positions within the scientific field, evaluation has also become an instrument for imposition and exclusion (Lincoln, 2005; Lyotard, 1998; Schwandt, 1996, 2000). Within a setting of checks and balances, though not without difficulty, a more open perspective is developing in terms of recognition for a pluralistic setting, and this entails the emergence of new vocabulary, new values, and new meanings (Schwandt, 1996). In this order of ideas, a more democratic evaluation can facilitate the development of a type of research committed toward social justice. Then evaluation can be open to accept alternative value scales that take into consideration socially constructed moral and political consequences of research (Denzin, Lincoln, & Giardina, 2006). So outcomes such as how the research contributes to the resolution of concrete situations of oppression and injustice can be accepted as criteria for evaluation.
Taking the debate about the evaluation of research into the field of justice means that it is necessary to acknowledge that science is a political field, which presupposes a public arena where differences and particular interests can be dealt with, within a collective project of science that can acknowledge and support plurality, in a setting that can improve mutual understanding. It therefore requires an approach that transcends qualitative research alone. Likewise, it deepens the acknowledgment of the researcher as a rational subject, both in its theoretical and practical (moral and political) dimensions.
Moving forward in the way this articles proposes requires a progressive consolidation of a more democratic and dialogic public setting in the scientific field, which might be achieved, as local and global alternative academic communities become stronger, with the power to push for political agreements 5 with regard to the evaluation of research. These will be agreements in constant transformation, under a sustained process of discussion, both within and between individual academic communities, in response to constantly changing historical, socioeconomic, and political circumstances. Each academic community will be working to achieve a scientific scenario in which there could be equal rights for divergent ways of understanding, carrying out and evaluating research, according to specific principles and procedures suited to each community.
The consolidation of a pluralistic concept of science that moves toward the construction of new scenarios of greater mutual cooperation will also be determined by factors such as the establishment of institutional review boards (IRB), which can respond to the issue of diversity. Another important factor is the teaching of research within a framework of respect and the recognition of different research stances, which requires a communicative and productive environment developed through constructive criticism, controversy, and an understanding of the power relationships that support them (Hernández, 2001).
The above situation raises the need for scientists suited to a pluralistic science as a requisite for the development of a political conception of justice within the scientific field. This would allow progress to be made toward the development of settings that would encourage a greater willingness in researchers to learn from each other and to reflect on their own actions, essential for strengthening the more open position that is required to overcome dogmatism.
However, this opening-up will require a greater effort from researchers to be explicit in clarifying and identifying their epistemological positions, their theoretical backgrounds, and political and ethical standpoints. Epistemological, political, and ethical reflection is therefore no longer a subsidiary issue but instead takes on a central role in research.
Footnotes
Acknowledgements
I want to express my gratitude to María Consuelo Chapela, Marta Gaviria, and Andrés Saldarriaga who read an earlier version of the manuscript and made important contributions.
Declaration of Conflicting Interests
The author declared no conflicts of interest with respect to the authorship and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research and/or authorship of this article: The National Faculty of Public Health, at the University of Antioquia, Medellín, Colombia.