Developing an ethical rationale for collaborative approaches to evaluation

Abstract

As a deeply relational, dialogic, engaged and political approach, the collaborative research context is fairly unique in the world of research, and as such opens up an entirely new set of ethical considerations that serve to differentiate it from other approaches, repositioning ethics as a fundamental rationale for collaborative inquiry. In this paper, we revisit the justifications for collaborative approaches to evaluation—the three Ps—which have become integral to our discourse about the genre. We then elaborate on our rationale for exploring ethics as a legitimate interest in collaborative approaches to evaluation, with special consideration given to why ethics should become an essential consideration moving forward, specifically in terms of the moral obligations of collaborative approaches to evaluation practitioners. We then re-envision the inclusion of an “ethic of engagement” along seven interconnected dimensions, what we refer to as the Seven Rs of collaborative practice: reflexivity, relationality, responsibility, recognition, representation, reciprocity, and rights.

Keywords

collaborative approaches to evaluation ethical justifications moral-political purpose

“The truth of our world will elude us until we learn what it means to hear across the social spectrum”

(Frank, 2013: 365).

“Every research activity is an exercise in research ethics, every research question is a moral dilemma, and every research decision is an instantiation of values”

(Clegg and Slife, 2009: 24)

“[The] concern with the primacy of ethical questions in understanding (and reforming) a social practice is . . . an acknowledgement that the assumed neutrality of technique does not allow us to escape difficult questions about the moral and political meaning of our practice for the society in which we work”

(Schwandt, 2002: 155).

Introduction

In a recent keynote address to the Australasian evaluation community, Mathison (2018) posed the question: “Does evaluation contribute to the public good?” In doing so, she argued that dominant political ideologies shape how evaluation is conceptualized, applied and used. She then traced evolving evaluation theory and practice from “early progressivism” through “neoliberalism” to what she sees as a “surging populism” (p. 111). Her conclusions about evaluation’s contribution to the public good were disquieting to say the least. Mainstream evaluation framed as a technocratic service institution has been lame in this respect. Mathison challenges evaluation’s alleged independence and frames it as a mechanism that ultimately conserves and serves the status quo. The catch phrase “speaking truth to power,” she goes on to say, has become a cliché favored by left leaning liberals who have somehow dissuaded themselves of the likelihood that the powerful already know the truth or choose to ignore it to suit their ideological interests.

A proliferation of approaches that promise participation, emancipation and transformation hold some promise, according to Mathison, to break evaluation’s ineptitude in contributing to the public good. She argues that “speaking truth to the powerless,” “ . . . to empower the powerless to speak for themselves” (p. 118) is worthy of the field’s serious attention. That Mathison embraces a socio-political justification for this assertion is evident; some would perhaps argue a moral-political stance, although we sense that her case is ideological in form and intent, and directly aimed at social justice as the lofty cause. But what of the moral shadings? To what extent have considerations about ethics been included in conversations about how to render evaluation capable of serving the public good?

Participatory and collaborative approaches to evaluation have been with us for quite some time, and indeed, are on the rise (Cousins, 2020; Cousins and Chouinard, 2012). Although designed for varying purposes and taking on distinctive if not unique forms, they all belong to a single genre of evaluation where evaluators work in partnership with non-evaluator members of the program community to co-construct evaluative knowledge. We recently labeled that family “collaborative approaches to evaluation” (CAE) and developed and validated a set of evidence-based principles to guide practice (Shulha et al., 2016). For many years, we have framed the rationale for CAE in pragmatic, political, and philosophic terms (Cousins and Whitmore, 1998), acknowledging interplay and overlap among them and imagining the justification of any instance of CAE necessarily to be a blend of the three. While matters of ethics are implied in all three rationales, we have recently taken pause to consider more broadly the ethical justifications for CAE,¹ and to reflect more deeply on the ethical dimensions of collaborative practice itself.

Collaborative research, where evaluators and participants co-create evaluative knowledge, is not a neutral nor uncontested space, but a deeply relational, dialogic, engaged and political place. This collaborative context is fairly unique in the world of research, and as such opens up an entirely new set of ethical considerations that serve to differentiate it from other approaches (Banks and Brydon-Miller, 2019; Tuck and Guishard, 2012), helping to reposition ethics as a fundamental rationale for collaborative inquiry (Heron and Reason, 1997). In this paper, we revisit the justifications for CAE—the Three Ps—which have become integral to our discourse about the genre. We then elaborate on our rationale for exploring ethics as a legitimate interest in CAE, with special consideration given to why ethics should become an essential consideration moving forward, specifically in terms of the moral obligations of CAE practitioners. In the section that follows, we explore and uncover varied and interconnected dimensions of ethics in CAE. Here we explicate an intertwined conceptual configuration of ethical dimensions. We conclude with some thoughts about why and how evaluators ought to think much more deeply and seriously about ethics in collaborative practice.

Interests in CAE: The Three Ps

The Three Ps—pragmatic, political, philosophical—conceptualization of justification for CAE has defined our own understanding about when and why collaborative approaches are appropriate, if not necessary, for more than 20 years (e.g. Cousins and Whitmore, 1998). These categories arise from our analysis of contributions by Mark and Shotland (1985), and Garaway (1995). To our way of thinking, the categories are anything but mutually exclusive; the motivation for any CAE will draw from two or more of them depending on presenting information needs and interests, and perhaps most importantly, whose interests are to be served. For this reason, the importance of understanding the economic, socio-political, historical, and cultural context within which knowledge is collaboratively produced, distributed and used will be highlighted. While we recognize that each of the Three Ps implicate ethics in some way, a more comprehensive rendering of ethics as a rationale will be reserved for the following section.

Pragmatic justifications for CAE

Pragmatic interests driving CAE are all about leveraging change with evaluative evidence, in other words, using evaluation for practical problem solving. Of primary concern would be instrumental (discrete decision making about interventions) and conceptual (learning) uses of evaluation findings, and to a limited extent, symbolic (persuasive, legitimative) outcomes. Program community members working with evaluators learn about how to change programs in order to improve them or make them more effective. As we have observed above, a good deal of methodological sophistication is required in order for evaluation to fulfill the promise of assessing the relative potency of program components thereby providing an evidential basis for program modification (Kogen, 2018). Yet, users stand to learn about program functionality even through close examination of performance descriptions, and CAE thorough the involvement of program community members in the knowledge production function provides an edge in this regard. Much of what we have come to know about evaluation utilization points to the value of relationship building between evaluators and members of the program community, and we have had a sense of this for quite some time. As early as the late 1970s Patton and colleagues singled out the “personal factor” as being essential to the promotion of the use of findings (Patton et al., 1977). Over time, ample empirical evidence has cast the relationship between evaluation utilization and participatory interaction and concomitant relationship building, as being robust (Cousins and Chouinard, 2012), an observation not lost on contemporary contributors to utilization-oriented approaches to evaluation (Patton, 1997, 2008). But the strength of the relationship lies not only with participation leveraging enhanced instrumental and conceptual use of findings, but also with process use, learning by virtue of proximity to the evaluation, its logic and functions, and quite independent of messages brought about by evaluation results (Cousins, 2007; Patton, 1997, 2008). Process use, as we discuss below, is intimately connected to evaluation capacity building, an essential element of political interests in CAE.

Political justifications for CAE

Historically, we have considered political interests driving CAE to be largely socio-political, and focused on empowerment and the amelioration of social inequity. Through participation in the evaluation knowledge production function, intended program beneficiaries (often from marginalized populations) and other program community members learn to see their circumstances differently and to recognize oppressive forces at play. In Freirean terms, it is through such recognition that subsequent ameliorative action may be precipitated (Freire, 1970). Such engagement holds great promise for the development of an ethos of self-determination. Some would argue that the impetus for a transformative paradigm arose from marginalized communities and their advocates; evaluation was not adequately representing their experiences, nor contributing to their improved living conditions (Mertens and Boland, 2018). Evaluation that embraces a transformative stance holds social justice and human rights as quintessential values that must drive inquiry; evaluation must address power inequities, the impact of privilege and the consequences of these for achieving social justice. The goal of transformative approaches is often to create space for the voices of historically marginalized groups—for example, substance abusers; gay, bisexual or transgendered individuals; indigenous peoples; people who live with disabilities; and those who are homeless and/or suffering mental illness—so that their stories and circumstances can be fully heard and appreciated by evaluation audiences. By virtue of engaging intended intervention beneficiaries in the co-construction of evaluative knowledge, CAE is well positioned to help create such space for historically marginalized and hard-to-reach populations.

Clearly, transformative applications of CAE have different ideological and historical roots than those leaning toward a more pragmatic orientation. Such applications are based on a more radical ideology and emerged in the late 1970s, predominantly in the developing world (Fals-Borda, 1980; Tandon, 1981). Conventional research and evaluation approaches were seen as creating cultural dependency and as denying the knowledge-creating abilities of ordinary people (Hall, 1975). Other influences included philosophical ideologies of Marx and Engels, Gramsci, Habermas and other critical theorists (Hall, 1992). Another dimension of the political justification for CAE is less about radical political ideology and more about the democratization of evaluation. Mark and Shotland (1985) identified representation as a rationale for engaging stakeholders in evaluative processes. Those with an interest in deliberative-democratic approaches to evaluation (e.g. Ryan and DeStefano, 2000) have embraced such a rationale, and often it would seem in the interest of social justice.

Philosophical justifications for CAE

Finally, philosophic justifications for CAE are grounded in a quest for deeper understanding of the complexities associated with the program and the context within which it is operating. Through evaluators working hand-in-hand with program community members, the joint production of knowledge is grounded in historical, socio-political, cultural, economic and educational context; thanks to the insider insights of participating program community members, deeper meaning of evaluative evidence and knowledge is achieved. Many have written about the nature and character of evaluative thinking as an essential aspect of the domain of inquiry. The development of “inquiry habits of mind” (Katz et al., 2002) provides one way to frame evaluative thinking and some would link this closely with critical thinking. Schwandt (2018) recently argued that we would benefit from a reframing of evaluative thinking as collaborative social practice. Much of his argument centered on cross-cultural evaluation: “one can employ a ‘both ways’ philosophy that supports the creation of dialogic spaces where new understandings of what counts as valuable knowledge and evidence can be constructed” (Chigrin and Husuijser, 2015, cited in Schwandt, 2018: 129). The implications for CAE as a route to creating deeper understanding about complex psychosocial phenomena is evident.

Interplay among the Three Ps and the case for developing an ethical rationale

As mentioned, these categories are understood not to be mutually exclusive and as such any given CAE will place relative emphasis on one or more depending upon information needs, participants, contextual and cultural exigencies and circumstances. Cousins and Whitmore (1998) identified two principal streams of participatory evaluation as being practical and transformative. The former would emphasize the pragmatic justification, whereas the latter privileges the political justification; both streams, however, draw from all three interests. For example, in practical participatory evaluation program community members may find the experience to be rewarding in terms of their own professional development even though the primary purpose is to generate knowledge supporting program improvement. Such capacity building is an example of process use even though it may be an unintended positive consequence of the evaluation. On the other hand, transformative participatory evaluation where empowerment and capacity building are central may also lead to positive changes to interventions because of evaluation findings. We observe that Fetterman and colleagues (Fetterman et al., 2018; Fetterman and Wandersman, 2005) have followed this lead in describing two streams of empowerment evaluation.

Whether explicit or not, each of the Three Ps, either alone or in combination, concern ethics, as all decisions—from involving stakeholder inclusion and exclusion, evaluation design, methods and approaches, to adjudicating the worth, value and merit of a program’s quality—are value-laden. While values remain foundational to the practice of evaluation, in CAE the practical concerns of evaluation (e.g. design, planning and implementation) intersect with building relationships, co-constructing knowledge, amelioration of inequities, empowerment, transformation, democratization, and the aspiration of social justice, all of which serves to highlight the moral and ethical disposition of the evaluator in collaborative contexts. This positioning reconfigures the social inquiry process, as divisions between evaluator and stakeholder participant shift into a partnership. Methodological rigor and validity are re-cast as reflexive communication rather than technical skills. Experience and action are privileged over other forms of knowledge (including conceptual, propositional, etc.), and the aims of research can turn to concerns with social justice and the amelioration of societal inequities rather than with program outcomes and impact. In the following section, we explore this repositioning further, as we consider what it means to view ethics as a constituent part of the research process, from the beginning of the study through to the writing and dissemination of the final report.

The intersection of ethics in evaluation

Evaluation is saturated with ethics, as evaluators navigate questions of design, inclusion, perspectives, power, approach, positionality, ownership, voice, methods, and so on. As with most other professional associations, the American Evaluation Association has developed Guiding Principles for Evaluators (2004) which, along with the Joint Committee on Standards for Educational Evaluation’s Program Evaluation Standards (Yarbrough, 2011), lays the foundation for sound, ethical practice to guide evaluators in their work. The Guiding Principles deal with issues of competence, integrity/honesty, and respect, with the Standards focused on issues of utility, feasibility, propriety, accuracy and accountability. These sorts of professional standards and guidelines provide a solid base to build awareness among evaluators and to generate discussion and debate within the field about professional practice. Yet, their range is limited to providing guidance on evaluation quality, method use, and on matters of appropriate conduct for evaluators, rather than on the broader, normative questions that orient and shape the practice of evaluation (Schwandt, 2015). As Schwandt (2015) points out, “avoiding misconduct and acting with methodological integrity is not the same as doing good evaluation” (p. 123).

All of this is made even more complex, as evaluations occur in a wide range of cultural, political, and historical contexts and across local, national and international settings. Questions of ethics and morality become inseparable from cultural ones (Fleischacker, 1999), and how we define what it means to do “good” evaluation or how we define “social betterment” (Mark et al., 2000) becomes a normative rather than a descriptive statement or claim. Despite its technical appeal, evaluation is not a value neutral activity that can be extricated from the evaluator’s moral/political stance and commitment or from its context of practice. As Greene et al. (2004) argue, “given that evaluation inevitably advances certain values and interests and not others, it can either maintain and reinforce the existing system, or it can serve to challenge, disrupt, and strive to change the existing social order” (p. 102). Even early on, anthropologist Johannes Fabian (1971) observed that research ethics was essentially defined by “conformity with the norms of the society which sponsors the scientific enterprise” (p. 230). Thus despite the lingering language of neutrality in evaluation, its “mask of objectivity” (Fals-Borda, 2001), evaluators need to critically reflect on the norms that guide their own practice and positionality, and on those influences that likely shape and inform their methodological choices, their understanding of context and approach, and their appreciation for principles of inclusion and the privileging of voice.

Schwandt (2002) argues that there are three different value/normative orientations involved in evaluation practice: (1) as a value neutral, technical craft, (2) as an emancipatory, value-committed perspective linked to emancipatory politics, and (3) as a value-critical orientation that redefines social inquiry and the role of the inquirer as dialogical and reflective, with an orientation to thinking about the moral and political dimensions of practice. This latter stance, what for Schwandt is considered the least understood and least examined value dimension of evaluation practice, brings focus to the question of what purpose(s) and whose interest evaluation ultimately serves. For Schwandt (2002), the ethical questions are framed around “the way society ought to be and the ways professionals ought to serve that society” (p. 144). These questions are essential in evaluation given the role of the evaluator in constructing knowledge about public programs and policies and in making judgments about their value to society. Recognizing evaluation as a normative, social practice rather than a purely descriptive, technical and scientific one, leads to ethical questions such as, what is the purpose of conducting evaluation? Why do we prioritize certain methodologies or criteria? What are the potential consequences of those prioritizations? To whom are evaluators in service?

Some years ago, Heron and Reason (1997) made a convincing case for extending Lincoln and Guba’s original discussion of paradigms to also include the axiological dimension (along with the participatory paradigm). From their perspective (see Heron and Reason, 1997; Reason, 1996, 1998), axiology poses the essential question “what is intrinsically valuable in human life, in particular, what sort of knowledge, if any, is intrinsically valuable?” (p. 2). As Reason (1996) states, “a central purpose of human inquiry is to develop knowledge that informs and guides the way we and others live our lives” (p. 17). These sentiments are very much in keeping with Schwandt’s (2002) critical stance, where ethics in evaluation is seen as a way to ground the evaluator in fundamental questions about the purpose of social inquiry and the responsibility of the evaluator to questions of aim, purpose and role. Ethics, and in particular ethics as a dimension of social inquiry, thus transforms the inquiry process from the transactional, guided by “extractive logic” (Kuntz, 2015), to an aspirational practice intrinsically related to questions of worth, purpose, and value. In their addition of axiology as one of the four foundational dimensions of paradigms, Lincoln and Guba (2000) came to believe that its inclusion helps inquirers see the “embeddedness of ethics within, not external to, paradigms” (p. 169). It thus seems quite clear to us that collaborative inquiry has always had deep ethical roots, what Park (1992) has referred to as an “ethical imperative.” Rather than operating simply as a code of conduct for how individuals ought to act with stakeholders or how to ensure confidentiality and consent among participants, ethical questions overlap and become deeply intertwined with matters of design and method, and deeply implicated in the relationship between researcher and participant (Schweigert, 2007).

Developing an ethical rationale for collaborative practice

While we acknowledge that the oft-cited rationales for collaboration (e.g. political, philosophic, pragmatic) do offer glimpses of ethics at multiple levels, a more focused and comprehensive normative justification for CAE has yet to be fully developed. Evaluation is a profoundly ethical practice, and as such we foreground evaluation as an “ethic of engagement” (Schwandt, 1996: 34) and advocacy (Greene, 1997), shifting rationales from developing appropriate outcome-focused quantitative measures of program success, to the inclusion of voices and perspectives historically left out of the evaluation conversation. Conversely, this shift also includes questions about the purpose of inquiry and of what it means to be an evaluator. Hence, in addition to the Three Ps justifications of CAE, we envision the inclusion of an “ethic of engagement,” which unfolds along seven interconnected dimensions, what we refer to as the Seven Rs of collaborative practice: reflexivity, relationality, responsibility, recognition, representation, reciprocity, and rights. Reflexivity situates the evaluator as an active, conscious being in the evaluation process, cultivating what Symonette (2004) calls “self-as-responsive-instrument” (p. 284). Relationality brings focus to the interdependent connection evaluators have with participants, as they focus on building a learning community together. Responsibility speaks directly to the purpose of human inquiry (Heron and Reason, 1997) and to our obligations to others in ameliorating historical and ongoing injustices and inequities. Recognition refers to the inclusion of diverse others and to the creation of a “difference-friendly” society (Fraser, 2001). Representation is about questions of voice and perspective and who has a right to speak for others, and reciprocity brings attention to the relational and dialogic process of inclusion. Finally, rights refers to the protection of legal, civil, moral or conventional obligations. While the ethical justification for collaboration dynamically overlaps with the Three Ps rationales, and perhaps especially with the political rationale in terms of its social justice and democratic orientations, what differentiates the ethical justification is its focus on the stance or role of the evaluator in acknowledging diverse others and in creating a favorable space for collaboration to flourish. In what follows, we briefly describe each dimension of the ethical justification.

Reflexivity

As an ethical concept, reflexivity calls on evaluators to cultivate “ethical mindfulness” (Bond, 2000) in their collaborations with participants, and to bring an awareness of community (its background, history and culture), as well as a critical self-awareness of one’s own values and beliefs, what Lincoln and Cannella (2009) call a “reflexive ethics” (p. 279). Etherington (2007) describes reflexivity as a tool for ensuring ethical practice and for conveying a measure of transparency to the research process. As Symonette (2004) explains,

Researchers are not empty vessels, blank slates, or inanimate “tools”—notable robots. . . . Ethical praxis summons us to start with the presumption that default predispositions exist so one needs to commit to discovering, excavating, and taking account of the ways in which they may hinder or enhance one’s work in a given context. (p. 282)

From this perspective, reflexivity is foundational to an ethical research praxis, as it situates and centers the researcher as an active, engaged co-participant in the research process. Heron and Reason (1997) refer to this awareness-of-self-in-context as “critical subjectivity,” a self-awareness that is grounded in the evaluator’s values and personal biases, what they describe as “a self-reflexive attention to the ground on which one is standing” (p. 7). Acknowledgment and awareness of one’s social-cultural location, one’s positionality as a researcher thus becomes essential to navigating what are oftentimes complex ethical moments (e.g. relationships, power dynamics, subjectivity, responsibility) in the research encounter. How we are situated as social and political beings affects the research process (Harding, 1986). Pillow (2015) positions reflexivity as interpretation, drawing on feminist literature she calls attention to voice, representation, awareness, transformation, power and privilege, and researcher subjectivity. A critical unveiling of who we are as researchers thus becomes integral to our ethical responsibilities, as together with participants, evaluators interpret data and co-construct evaluative knowledge.

Relationality

Collaborative research is a deeply relational practice defined by the values of interdependence, connection and engagement, with evaluators and participants intricately linked together in actively co-constructing evaluative knowledge. Relationality thus implies a particular ontological, epistemological and ethical orientation and alignment with the world, an interconnectivity that is shared among participants, cultures, knowledge and values. Poole (1972) characterizes this inter-relationship as occurring across an “ethical space,” a space infused with history, politics, and culture, and composed of people and places that shape the dynamics of the collaborative process itself. From this perspective, relationality describes the interconnections among people interpersonally and epistemologically, where the knowledge that is co-created is the result of the interaction between evaluators and participants (Lincoln and Guba, 1985). Relationality is also understood as describing the connection between ways of knowing (what we know, how we come to know, and the potential for action) and the broader, more macro sociopolitical and institutional contexts of practice, what Lyotard (1979) would call metanarratives, that shape local collaborative possibilities and potential (Cousins and Chouinard, 2012; Kuntz, 2015; Smith, 1987). Relationality, as Kuntz (2015) has observed, requires an ethical engagement with multiple and varied narratives, grounding us in local and global issues and concerns. As Kuntz (2015) states, “we never exist outside of relation to anything or anyone else” (p. 74). Relationality thus opens up an “ethical space” for methodological work connecting local and distant narratives, voices and perspectives, traditions, and values, creating an active ethical engagement in what is often considered a highly contested space.

Responsibility

The notion of responsibility is understood from a personal, societal, global and relational perspective, very much grounded in an “ethic of care” (Gilligan, 1982; Noddings, 1984), where virtues of care, empathy, compassion and benevolence speak to (and inspire) us as evaluators and methodologists. This ethic provides a “shared responsibility” (Morton et al., 2013, cited in Cram and Mertens, 2016: 169) what Simons and Greene (2018) refer to as an “interrelational responsibility” (p. 94) between one human being and another. For Heron and Reason (1997), responsibility extends from ourselves to the planet and to our role (or part) in promoting self-determination and to addressing ongoing and systemic issues of racism, sexism, gender and class inequality, and so forth that define our society. As Mertens (2009) has stated,

Ethical choices in research and evaluation need to include a realization that discrimination and oppression are pervasive, and that researchers and evaluators have a moral responsibility to understand the communities in which they work in order to challenge societal processes that allow the status quo to continue. (p. 48, emphasis added)

Responsibility thus extends to the evaluator’s role in creating space for a more equitable distribution of perspectives (Park, 1992), and for ensuring that the voices of those most marginalized are included in the collaborative process. Responsibility thus can be conceptualized as a relational ethic. Acknowledging local knowledge and the diversity of perspectives is thus given priority (Chouinard and Cram, 2020), as are considerations of ownership and community safety, especially given what are often historical differences in power and privilege between evaluators and community. From this perspective, responsibility refers to a deep responsibility in terms of identifying: who will be included and excluded in the evaluation and whose needs and interests will be given priority. It refers to how we will come to know and address the histories of racism, discrimination and oppression that define so many of the communities in which we work, both locally and around the globe. There is an obligation that we must unlearn our privilege (Landry and Maclean, 1996), as it can blind us from gaining other knowledge. In an important sense, this unlearning begins the ethical relationship we have with participants, as it connects our relational responsibility with our need to be responsible for our own positionality (Madison, 2006).

Recognition

Historically, the discourse of social justice centered on two related claims, a social claim resting on the redistribution of goods and resources and a cultural claim focused on the recognition of the distinct perspectives of ethnic, racial and sexual minorities and gender differences (Fraser, 2001). From our perspective, the political justification, as previously defined, is more closely aligned with a redistributive claim, as it speaks to the emancipation and amelioration of social inequities and to the redistribution of resources. Recognition, on the other hand, refers to the cultural dimension of social justice, whereby people are denied participation or an equitable process because of institutionalized hierarchies (essentially patterns of systemic racism) that prevent them from participation (Fraser, 2005). Fricker (2007) calls this type of obstacle to inclusion or silencing a form of “testimonial injustice,” where people are denied voice and credibility simply based on who they are and where they come from. This silencing or exclusion highlights one of the key epistemological challenges in collaborative work, as it draws attention to the ongoing struggle between what is essentially “subjugated knowledge,” knowledge that is considered local, regional and popular, against knowledge that is held to be more erudite, scientific, and hence more legitimate (Foucault, 1980). Recognition and representation are thus closely aligned, as the questions shift to who is left out of the conversation, and for what reasons? Recognition thus involves acknowledging difference that comes from the appreciation of diverse others and the distinct perspectives that they bring, whether through ethnic, “racial” and sexual minority status or gender differences (Fraser, 2001; Fraser and Honneth, 2003). Ethics in cultural competence means learning about culture, embracing pluralism and accommodation (Paasche-Orlow, 2004).

Representation

Whereas recognition was related to the cultural dimension of social justice, representation points to the political characteristics that frame the parameters and boundaries of inclusion and exclusion, based on terms, procedures and decision rules that do not provide for equal voice or fair representation (Fraser, 2005). Representation also points to questions of community ownership over decisions about which stories to include and whose perspectives to exclude. How we represent others is always contentious and always has consequences (Madison, 2006). As a dimension, representation acknowledges that there are political obstacles to participation that shape and frame social belonging in our society, often based on implicit or explicit rules that preclude people from fair representation. The questions then become: Who is representing the community? Who is speaking? For whom? And whose interests do they represent? Who decides as what point communities speak for themselves? As a rationale for engagement, representation refers to ensuring a context and process that is conducive to engaging stakeholders in collaborative practice, through attention to stakeholder selection and inclusion criteria, establishing rules of who can speak (and for whom), and creating safe spaces that encourage principles of democratic dialogue. How people are represented points to how people are treated; representation has consequences (Madison, 2006). Dimensions of recognition and representation are thus interwoven, as socio-cultural inequities and institutionalized patterns of injustice define the political landscape and local boundaries of inclusion and exclusion in (and from) democratic and civic participation. Thus Spivak (1990) warns that “this question of representation, self-representation, representing others is a problem” (p. 63) that requires “persistent critique” if we are to ensure the inclusion of “other” perspectives and voices.

Reciprocity

Collaborative practice is a profoundly relational practice that involves “give-and-take” between evaluators and participants, what Cushman (1996) refers to as “an open and conscious negotiation” (p. 16), as participants enter into collaborative relationship and co-construct knowledge together. The notion of reciprocity emphasizes the relational, dialogic and political aspects of collaborative practice, the ethical implications of collaboration, the give-and-take aspects, and draws attention to who benefits from the partnership (and who might not). It brings focus to how we are situated in the evaluative relationship and what impact our positions may have on our relationships, our interpretations and our work. Reciprocity also acknowledges power and its connection to meaning among participants who are often situated in unequal positions in the research relationship. For Lather (1986), a maximal approach to reciprocity involves “a mutual negotiation of meaning and power” (p. 263), as participants and social inquirers together engage in reflective practice, seek mutual understanding and negotiate meaning throughout the inquiry process. The concept of reciprocity thus positions ethics as “self-critical, conscious navigation” (Cushman, 1996: 16) and as a fundamental component of our relationships, directing our attention to how we enter the community, under which conditions, and for whose benefit. More than that, reciprocity also speaks to the inclusion of research participants in the negotiation of meaning, bringing focus to how we may impose meaning versus how it is negotiated (Lather, 1986).

Rights

While we acknowledge the contestability and lack of consensus around the concept of rights (Tebbit, 2017; Zivi, 2012), as a rationale for engagement our focus is on rights conceptualized as a set of normative guidelines and practices related to overarching legal, moral, individual, civil or conventional rights and obligations. As Tebbit (2017) pointed out

… rights language remains of value for political movements committed to advancing democratic values and practices because it is through the making of rights claims that we contest and constitute the meaning of individual identity, the contours of community . . . [and it] allows us to question and reconstitute the very meaning of what is common or sensible and what is not. (p. 8)

While the language of rights is often aspirational, it can advance dialogue about democratic principles and participation in civic life, and advance certain codified legal, political and social obligations to include others, and advance issues of equity and claims to democratic rights. As Tebbit (2017) explains, “rights claiming is a democratic practice because it opens up rather than shuts down the possibility for political engagement and thus demands our ongoing political participation” (p. 8). Importantly, codified rights can provide a level of ethical guidance or even guidelines for action in the absence of any other articulated or established standards of practice. As such, the Universal Declaration of Human Rights (1948) and other such universal codes help ensure that rights are protected for all citizens, regardless of status, gender, and the like.

Concluding thoughts

Above, we revisited the Three Ps justifications for CAE (pragmatics, political, philosophical), all of which have been considered integral to discussions about participatory practice. We acknowledged that all three rationales, either alone or in combination, each possess ethical qualities. However, we argued that as a deeply relational, dialogic and contextual approach, CAE opens up an entirely new set of ethical considerations, which we felt warranted a closer look. Our exploration led to the identification and discussion of what we termed the Seven Rs of arguments for collaborative practice: reflexivity, relationality, responsibility, recognition, representation, reciprocity, and rights. It is worth noting, as diagrammatically represented in Figure 1, that these seven dimensions do not stand in isolation. Collectively, they are intertwined and interconnected, whether as supporting mechanisms or in opposition to one another (e.g. responsibility is focused on a shared or collective ethos, whereas with recognition the focus is on what makes us distinct).

Figure 1.

Ethical justifications for collaborative approaches to evaluation.

It is also worthy of note in Figure 1, that the Three P’s as justifications for CAE practice remain intact; any given CAE project will draw (prospective analysis), or will have drawn (retrospective analysis), from pragmatic, political, or philosophical considerations. Depending on the contextual complexities within which the target intervention is situated, the relative emphasis on any of the 3 P’s will be differentiated. Yet now, given our explication of attendant ethical considerations, the Seven R’s undergird choices about whether to, and more importantly why to, engage in collaborative evaluation practice. As discussed above, long ago we identified two principal strands of participatory evaluation (Cousins and Whitmore, 1998), practical and transformative. The strands draw from all of the Three P justifications with differential emphasis, favoring pragmatic for practical participatory evaluation and political for transformative. Missing from, or only implicit in, discussions about whether or why to engage with a collaborative or participatory approach were ethical considerations. Our explication of the Seven R’s now provides evaluators with a means of reflecting more deeply about the motivations for CAE. In that sense, ethical considerations are backgrounded to the principal reasons for doing CAE. Whether that interest is practical or transformative, explicit attention to the complexities of ethical concerns will serve to buttress and inspire CAE’s potential to leverage genuine, positive social change.

In practice, we might also consider foregrounding ethical deliberations as justifications for collaboration. In some contexts, ones that are particularly encumbered by moral/political concerns, we might think of ethics as the principal justification for CAE. Consider, for example, implications from the Canadian case of the Truth and Reconciliation Commission’s (TRC) Call to Action² in the context of federal government reparations to indigenous peoples.

The TRC was an official body tasked with discovering and revealing past wrongdoing by the Canadian government (and other non-state actors, particularly Catholic and Protestant churches across Canada), in the hope of resolving lingering conflict. Of central concern was the federal government residential schools policy, intended to assimilate Indigenous people into the mainstream dominant Canadian culture. The first residential school opened in Ontario in 1849 and the last closed in Saskatchewan in 1996. Affected through mandatory, enforced attendance were an estimated 150,000 Indian, Métis, and Inuit students. Many suffered poor living conditions and sexual and physical abuse. In 2006 the Indian Residential School Settlement Agreement among former students, churches, indigenous peoples organizations and the Canadian federal government was approved by all parties.

Out of that agreement, the TRC was established in 2008 with a mandate for truth-seeking and reconciliation functions. Over a 8-year period at a cost of $72M CAD it conducted a wide range of national events, regional “town halls” and local community hearings, which were attended by tens of thousands of individuals, a good many being residential school survivors and other indigenous parties.³ The TRC released its final report in 2015,⁴ which comprised 94 Calls to Action focusing on child welfare, education, health and justice. The report also called for regular monitoring and evaluation, which subsequently became the mandate of the National Counsel for Reconciliation.

Our purpose here is not to critique the National Counsel’s role in monitoring and evaluation (although that might be a worthy topic for another day). Rather, we wish to highlight the justifications for participatory and collaborative approaches to evaluating interventions arising from the 94 Calls to Action. Here, in our view, is a compelling case for foregrounding ethical justifications for CAE. In the words of Prime Minister Trudeau, it is time for a renewed, “nation-to-nation relationship, based on recognition of rights, respect, co-operation, and partnership.”⁵ Deliberations about whether and why CAE ought to be the principal approach to assessing the implementation and impact of Call to Action interventions would benefit strongly from serious debate of the Seven Rs. For example, considerations about “representation” and “recognition” might speak to the enormous diversity within indigenous cultures in Canada, and the need to avoid oversimplifying the complexities brought by wide-ranging perspectives. Inclusivity and fairness are at the heart of such issues. Consistent with the conclusions of the TRC itself, upholding and protecting legal, civil, moral and conventional obligations, that is, the “rights” of indigenous peoples ought to be seen as a benefit of ethical justifications for CAE in the context of the Calls to Action. While we could probe deeper in this vein, suffice to say, foregrounding ethical justifications for CAE in this context is imperative. Of course, backgrounding the Three Ps will be part of the mix perhaps with particular emphasis on indigenous people’s ways of knowing (philosophical) and the amelioration of the social inequities they face (political) being most salient. In any case, as Bremner (2019) asserts, when it comes to evaluating the impact of the TRC Calls to Action, we need to ask “What stories need to be told and who will tell them? How will these stories speak to truth and reconciliation? How can we ensure that these stories will be heard?” (p. 340). Ethically, a strong role of CAE is undeniable.

Our focus on ethics in collaborative practice helps to reframe CAE as more than a set of techniques, tools or methods, as it draws attention to the embeddedness and intrinsic quality of ethics throughout the process of evaluative inquiry, from the beginning through to the dissemination of final results. The seven Rs are thus not intended as tools or instruments of ethical navigation, as instrumentalities, but rather as a form of praxis or informed, reflexive action, guided by a disposition to act morally, wisely and justly (Carr and Kemmis, 1986), to act reflexively as evaluators shift between context, theory and action. Although the notion of values and ethics in evaluation is not a radical idea (see Karier, 1974; Stake, 1975), well into the 21st century the specter of the “evaluation machine” (Dahler-Larsen, 2012; Kemmis, 1993) continues to cast a long and intellectually menacing shadow over the field of evaluation.

We conceptualize ethics, not as a simple concern with professional conduct and decisions about correct action, but as a normative construct shaping the attitudes, values and purposes about how we are (and how want to be) in the world (Heron and Reason, 1997; Schwandt, 2015). The ethical question thus shifts from how we construct a collaborative evaluation context to why and for what purposes (Heron and Reason, 1997), a shift that enables us to broaden the conversation to engage more meaningfully in questions about evaluation purpose and to ask whose interests are being served. Our discussion of the Seven Rs of collaborative practice, though descriptive at this point, highlight issues of relationality, representation, authorship, ownership, positionality, power, politics, epistemology, ontology, social justice, voice and values, issues which we believe are quite distinctive to CAE. While these issues do not occur in all collaborative relationships, CAE can give rise to the sorts of concerns that more traditional approaches to evaluation, where boundaries are clearer between researchers and participants and research goals are more firmly articulated, are less likely to occur (Banks and Brydon-Miller, 2019). We are far more likely to encounter major ethical concerns in contexts involving the co-construction of knowledge, unequal power and positionality, local or tribal knowledge versus academic knowledge, and decisions around the inclusion or exclusion of participants and voices, and in a world where structures of inequity persist in shaping and maintaining the status quo. These issues relate as much to the process of collaboration (e.g. who do we involve? how we involve them? whose knowledge defines the parameters of practice?), as to the outcome of our collaborative efforts (e.g. who owns the data and findings? who benefits, and how?). We believe that these ethical considerations have the potential to force evaluators to explore their own positionality and participation in collaborative evaluation contexts, and to force them to look beyond institutional review boards and professional codes of conduct, to their own complicity in accepting colonial practice (Tuck and Guishard, 2012).

This perspective of ethics broadens the conversation beyond merely how we use our methods and approaches to generate new knowledge or how we create processes that will bring a plurality of perspectives to the table, to questions such as: What constitutes social betterment? Who decides? What (and whose) cultural, political and institutional values inform and shape our methodological approaches? As Schwandt (2015) suggests, ethics in evaluation is not simply a matter of individual professional conduct but also of “the normative orientations of institutional practices in society” (p. 136). If we truly wish to pursue a more socially just, democratic and humane practice in evaluation, essentially a more ethical aspiration, then we need to encourage critical dialogue about the practice of evaluation as an applied social science, and what it means to co-create knowledge in a society that remains so deeply, profoundly and increasingly divided. We believe that in the end, it becomes an “ethical imperative” (Reason, 1998: 15) to reflect on ourselves, on the choices we make about the inclusion or exclusion of perspectives and voices, and about how we come to know. “To whom we are in service?” thus becomes the ultimate ethical question.

Footnotes

Acknowledgements

A previous version of this paper was presented at the annual meeting of the Canadian Evaluation Society, Calgary, June 2018 and some of the ideas were integrated into a plenary presentation by Cousins at the biennial meeting of the European Evaluation Society, Thessaloniki, Greece, October 2018. Chouinard was located at the University of North Carolina, Greensboro, as previous versions of this paper were developed.

Declaration of conflicting interest

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jill Anne Chouinard

Notes

Jill Anne Chouinard is Associate Professor in the School of Public Administration at the University of Victoria. Her research interests are in culturally responsive and collaborative approaches to evaluation. She positions evaluation as a catalyst for learning, social justice and community change.

J. Bradley Cousins is Professor Emeritus of Evaluation at the University of Ottawa and former Director of the Center for Research on Educational and Community Services. His research interests in evaluation include utilization, collaborative approaches and capacity building.

References

American Evaluation Association (2004) American Evaluation Association guiding principles for evaluators. Available at: https://www.eval.org/p/cm/ld/fid=51 (accessed 12 October 2019).

Banks

Brydon-Miller

(2019) Ethics in Participatory Research for Health and Well-Being: Cases and Commentaries. New York: Routledge.

Bond

(2000) Standards and Ethics for Counselling in Action, 2nd edn. London: SAGE.

Bremner

(2019) Creating new stories: The role of evaluation in truth and reconciliation. Canadian Journal of Program Evaluation 34(2): 331–42.

Carr

Kemmis

(1986) Becoming Critical: Education, Knowledge and Action Research. London: Falmer Press.

Chouinard

Cram

(2020) Culturally Responsive Approaches to Evaluation: Empirical Implications for Theory and Practice. Thousand Oaks, CA: SAGE.

Clegg

Slife

(2009) Research ethics in the postmodern context. In: Mertens

Ginsberg

(eds) The Handbook of Social Research Ethics. Thousand Oaks, CA: SAGE, 23–38.

Cousins

(2007) Process use in theory, research and practice. New directions for evaluation No. 116. San Francisco, CA: Jossey-Bass.

Cousins

Chouinard

(2012) Participatory Evaluation Up Close: An Integration of Research-Based Knowledge. Charlotte, NC: Information Age Publishing, Inc.

10.

Cousins

Whitmore

(1998) Framing participatory evaluation. In: Whitmore

(ed.) Understanding and Practicing Participatory Evaluation. New Directions in Evaluation, No. 80. San Francisco, CA: Jossey-Bass, 3–23.

11.

Cousins

(ed.) (2020) Collaborative Approaches to Evaluation: Principles in Use. Thousand Oaks, CA: SAGE.

12.

Cram

Mertens

(2016) Negotiating solidarity between indigenous and transformative paradigms in evaluation. Evaluation Matters—He Take Tō Te Aromatawai 2: 161–89.

13.

Cushman

(1996) The rhetorician as an agent of social change. College Composition and Communication 47(1): 7–28.

14.

Dahler-Larsen

(2012) The Evaluation Society. Stanford, CA: Stanford University Press.

15.

Etherington

(2007) Ethical research in reflexive relationships. Qualitative Inquiry 13: 598–616.

16.

Fabian

(1971) Language, history and anthropology. Philosophy of the Social Sciences 1: 19–47.

17.

Fals-Borda

(1980) Science and common people. Paper presented at the International Forum on Participatory Research, Ljubljana, 13–22 April.

18.

Fals-Borda

(2001) Participatory (action) research in social theory: Origins and challenges. In: Reason

Bradbury

(eds) Handbook of Action Research: Participative Inquiry and Practice. London: SAGE, 27–37.

19.

Fetterman

Wandersman

(eds) (2005) Empowerment Evaluation Principles in Practice. New York: Guildford, 183–208.

20.

Fetterman

Rodriguez-Campos

Zukoski

, et al. (2018) Collaborative, Participatory and Empowerment Evaluation: Stakeholder Involvement Approaches. New York: Guilford.

21.

Fleischacker

(1999) From cultural diversity to universal ethics: Three models. Cultural Dynamics 11(1): 105–28.

22.

Foucault

(1980) The History of Sexuality, Vol. 1. New York: Vintage.

23.

Frank

(2013) Mitigating against epistemic injustice in educational research. Educational Researcher 42(7): 363–70.

24.

Fraser

(2001) Recognition without ethics? Theory, Culture & Society 18(2–3): 21–42.

25.

Fraser

(2005) Reframing justice in a globalizing world. New Left Review 36: 69–88.

26.

Fraser

Honneth

(2003) Redistribution or Recognition? A Political-Philosophical Exchange. London: Verso.

27.

Freire

(1970) Pedagogy of the Oppressed. New York: Seabury Press.

28.

Fricker

(2007) Epistemic Injustice: Power and the Ethics of Knowing. New York: Oxford University Press.

29.

Garaway

(1995) Participatory evaluation. Studies in Educational Evaluation 21: 85–102.

30.

Gilligan

(1982) In a Different Voice: Psychological Theory and Women’s Development. Cambridge, MA: Harvard University Press.

31.

Greene

(1997) Evaluation as advocacy. American Journal of Evaluation 18: 25–35.

32.

Greene

Millett

Hopson

(2004) Evaluation as a democratizing practice. In: Braverman

Constantine

Slater

(eds) Foundations and Evaluation: Contexts and Practices for Effective Philanthropy. San Francisco, CA: Jossey-Bass, 96–118.

33.

Hall

(1975) Participatory research: An approach for change. Convergence 8(2): 24–32.

34.

Hall

(1992) The west and the rest: Discourse and power. In: Hall

Gieben

(eds) Formations of Modernity. Cambridge: Polity Press; Open University, 276–320.

35.

Harding

(1986) The Science Question in Feminism. Ithaca, NY: Cornell University Press.

36.

Heron

Reason

(1997) A participatory inquiry paradigm. Qualitative Inquiry 3(3): 274–94.

37.

Karier

(1974) Ideology and evaluation in quest of meritocracy. In: Apple

(ed.) Educational Evaluation: Analysis and Responsibility. Berkeley, CA: McCutchan, 237–68.

38.

Katz

Sutherland

Earl

(2002) Developing an evaluation habit of mind. Canadian Journal of Program Evaluation 17(2): 103–19.

39.

Kemmis

(1993) Foucault, Habermas and evaluation. Curriculum Studies 1(1): 35–54.

40.

Kogen

(2018) What have we learned here? Questioning accountability in aid policy and practice. Evaluation 24(1): 98–112.

41.

Kuntz

(2015) The Responsible Methodologist: Inquiry, Truth-Telling, and Social Justice. Walnut Creek, CA: Left Coast Press; London: Routledge.

42.

Landry

Maclean

(1996) Introduction. In: Landry

Maclean

(eds) The Spivak Reader: Selected Works of Gayatri Chakravorty Spivak. New York: Routledge, 29–51.

43.

Lather

(1986) Research as praxis. Harvard Educational Review 56(3): 257–77.

44.

Lincoln

Cannella

(2009) Ethics and the broader rethinking/reconceptualizatioj of research as construct. Cultural Studies-Critical Methodologies 9(2): 273–85.

45.

Lincoln

Guba

(1985) Naturalistic Inquiry. Newbury Park, CA: Sage.

46.

Lincoln

Guba

(2000) Paradigmatic controversies, contradictions, and emerging confluences. In: Denzin

Lincoln

(eds) The Handbook of Qualitative Research, 2nd edn. Thousand Oaks, CA: SAGE, 1065–122.

47.

Lyotard

(1979) The Postmodern Condition: A Report on Knowledge. Minneapolis, MN: The University of Minnesota Press.

48.

Madison

(2006) Critical Ethnography: Method, Ethics, and Performance. Thousand Oaks, CA: SAGE.

49.

Mark

Shotland

(1985) Stakeholder-based evaluation and value judgments. Evaluation Review 9: 605–26.

50.

Mark

Henry

Julnes

(2000) Evaluation: An Integrated Framework for Understanding, Guiding and Improving Policies and Programs. San Francisco, CA: Jossey-Bass.

51.

Mathison

(2018) Does evaluation contribute to the public good? Evaluation 24(1): 113–9.

52.

Mertens

(2009) Transformative Research and Evaluation. New York: The Guilford Press.

53.

Mertens

Boland

(2018) Complex ecology in international development evaluation, focusing on women and people with disabilities. In: Hopson

Cram

(eds) Tacking Wicked Problems in Complex Ecologies: The Role of Evaluation. Stanford, CA: Stanford University Press.

54.

Noddings

(1984) Caring, a Feminine Approach to Ethics & Moral Education. Berkeley, CA: University of California Press.

55.

Paasche-Orlow

(2004) The ethics of cultural competence. Academic Medicine 79(4): 347–50.

56.

Park

(1992) The discovery of participatory research as a new scientific paradigm: Personal and intellectual accounts. The American Sociologist 23: 29–42.

57.

Patton

(1997) Utilization-Focused Evaluation: A New Century Text, 3rd edn. Thousand Oaks, CA: SAGE.

58.

Patton

(2008) Utilization-Focused Evaluation, 4th edn. Thousand Oaks, CA: SAGE.

59.

Patton

Grimes

Guthrie

, et al. (1977) In search of impact: An analysis of the utilization of federal health evaluation research. In: Weiss

(ed.) Using Social Research in Public Policy Making. New York: D. C. Heath and Company, 141–63.

60.

Pillow

(2015) Reflexivity as interpretation and genealogy in research. Cultural Studies-Critical Methodologies 15(6): 419–34.

61.

Poole

(1972) Towards Deep Subjectivity. London: Allen Lane, The Penguin Press.

62.

Reason

(1996) Reflections on the purposes of human inquiry. Qualitative Inquiry 2(1): 15–28.

63.

Reason

(1998) Political, epistemological, ecological and spiritual dimensions of participation. Studies in Cultures, Organizations and Societies 4: 147–67.

64.

Ryan

DeStefano

(eds) (2000) Evaluation as a Democratic Process: Promoting Inclusion, Dialogue, and Deliberation, Vol. 85. San Francisco, CA: Jossey-Bass.

65.

Schwandt

(1996) Farewell to criteriology. Qualitative Inquiry 2(1): 58–72.

66.

Schwandt

(2002) Evaluation Practice Reconsidered. New York: Peter Lang.

67.

Schwandt

(2015) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Stanford, CA: Stanford University Press.

68.

Schwandt

(2018) Evaluative thinking as collaborative social practice: The case of boundary judgement making. Evaluative Thinking: New Directions for Evaluation, Vol. 158. San Francisco, CA: Jossey-Bass, 125–37.

69.

Schweigert

(2007) The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning 30: 394–9.

70.

Shulha

Whitmore

Cousins

, et al. (2016) Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation 37(2): 193–215.

71.

Simons

Greene

(2018) Democratic evaluation and care ethics. In: Visse

Abma

(eds) Evaluation for a Caring Society. Charlotte, NC: Information Age Publishing, Inc, 83–104.

72.

Smith

(1987) The Everyday World as Problematic: A Sociology for People. Toronto, ON, Canada: University of Toronto Press.

73.

Spivak

(1990) Post-Colonial Critic: Interviews, Strategies, Dialogues. In: Harasym

(ed). New York, NY: Routledge.

74.

Stake

(1975) Evaluating the Arts in Education: A Responsive Approach. Columbus, OH: Charles E. Merrill.

75.

Symonette

(2004) Walking pathways toward becoming a culturally competent evaluator: Boundaries, borderlands, and border crossings. New Directions for Evaluation 102: 95–109.

76.

Tandon

(1981) Participatory research in the empowerment of people. Convergence 14(3): 20–29.

77.

Tebbit

(2017) Philosophy of Law: An Introduction, 3rd ed. New York: Routledge.

78.

Tuck

Guishard

(2012) Uncollapsing ethics: Racialized sciencism, settler coloniality, and an ethical framework of decolonial participatory action research. In: Kress

Malott

Porfilio

(eds) Challenging the Status Quo: New Directions in Critical Research. Charolotte, NC: Information Age Publishing, Inc, 3–27.

79.

Yarbrough

(2011) The program evaluation standards: A guide for evaluators and evaluation users. Available at: https://us.sagepub.com/sites/default/files/upm-binaries/35465_Intro.pdf (accessed 12 2019).

80.

Zivi

(2012) Making Rights Claims: A Practice of Democratic Citizenship. Oxford: Oxford University Press.