Abstract
There has been an increasing call for social justice, not just in American society but also within the evaluation profession. Scholars and practitioners like Donna Mertens, Stafford Hood, Karen Kirkhart, Veronica Thomas, and many others have led the movement within the evaluation profession to view evaluation practice as a vehicle for social justice. However, descriptions of social justice-oriented evaluation in the literature tend to lack sufficient detail, leaving many evaluators to wonder exactly what social justice-oriented evaluation is. This article presents a summary of the literature on social justice-oriented evaluation, providing a much-needed description of the approach. This paper also introduces a taxonomy of social justice-oriented evaluation (T-SJOE) practices, developed from the results of the literature review, and presents a framework of social justice-oriented evaluation practices as they are currently discussed in the evaluation literature. T-SJOE is not prescriptive or presumed complete but is presented as a tool for self-reflection and as a resource when planning evaluations. Further development of the T-SJOE is expected and encouraged.
Evaluators are often involved in evaluations of programs or policies contending with the “wicked problems” of society, including poverty and homelessness, inadequate access to healthcare, and differential access to and participation in our democratic, economic, and education systems (Boyce, Reid, et al., 2022; Boyce, Tovey, et al., 2022; Collins et al., 2014; McBride et al., 2020; McKegg, 2013, 2019; Symonette et al., 2020). Evaluators and social researchers see that our societies are fraught with systemic inequities, cultural tensions, and general unrest (Hay, 2017) and call on the profession to use evaluation as a vehicle in the pursuit of social betterment (i.e., attending to these “wicked problems” to improve social conditions). Specifically, many in the field see that interrogating issues of equity, equality, inclusion, culture, power, and privilege can help us address these “wicked problems” and, therefore, call for their colleagues to attend to issues of social justice in their evaluations (Archibald et al., 2018; Caldwell & Bledsoe, 2019; Collins et al., 2014; Desivilya Syna et al., 2015; House, 2019). The American Evaluation Association (AEA) echoes this call and emphasizes the importance of considering issues of social justice during evaluations. Evaluators are called to use evaluation as a vehicle for social justice efforts in the governing and philosophical documents of the field (American Evaluation Association, 2018a; 2018b).
Social justice-oriented evaluation is an approach that evaluators can use to support social justice aims in their evaluation work. The purpose of this paper is to provide a summary of the current understanding of social justice-oriented evaluation approaches and methods by reviewing the evaluation literature. This paper aims to answer the following questions: What is social justice? What is social justice within the context of evaluation practice? Why should social justice matter to evaluators? What does justice work in evaluation currently look like? What are the critiques and challenges to social justice-oriented evaluation?
This paper will also introduce a taxonomy of social justice-oriented evaluation (T-SJOE) practices as they are currently discussed in the evaluation literature. The T-SJOE was developed from the results and findings of the literature review and presented as a framework of practices for professional evaluators and students of evaluation. The T-SJOE is not prescriptive, nor is it presumed to be complete, as some practices might only be discussed in grey literature (i.e., outside of academic journals, such as reports, white papers, and website publications). The T-SJOE is presented as a tool for self-reflection and to help evaluators think about how social justice-oriented evaluation might bring about positive social change. The taxonomy is intended to be mutable—changing and adapting to new techniques developed within the field. The author has endeavored to present a current exhaustive list based on the literature but recognizes that evaluation as a field continues to evolve; additions and changes to the framework are expected and encouraged.
Review Method
Journal Selection
Articles from the last 10 years prior to this review (2012–2022) were selected for review from peer-reviewed journals in the field of Evaluation, including the American Journal of Evaluation, New Directions for Evaluation, Evaluation and Program Planning, Evaluation Review, Evaluation, Journal of MultiDisciplinary Evaluation, and the Canadian Journal of Program Evaluation. Each journal was chosen for its reputation within the Evaluation field as a publisher of high-quality scholarly work. The American Journal of Evaluation and New Directions for Evaluation are both published by the American Evaluation Association and publish works on the theory, practice, and ethics of evaluation, though each issue of New Directions for Evaluation is devoted to a particular topic. For example, the topic of the Summer/Fall 2023 issue of New Directions for Evaluation is “Evaluation and Artificial Intelligence” (Mason & Montrosse-Moorhead, 2023). Evaluation and Program Planning publish multidisciplinary articles relevant to both evaluation and programs (e.g., public health, education). Evaluation Review and Evaluation are international journals that publish articles on evaluation methods, practice, research, and the results of evaluation projects. The Journal of MultiDisciplinary Evaluation is published by The Evaluation Center at Western Michigan University and publishes articles on the theory and practice of evaluation as well as the intersection of evaluation and other disciplines. The Canadian Journal of Program Evaluation is published by the Canadian Evaluation Society and focuses on theory and practice.
Grey Literature
Grey literature refers to resources that are not published in academic journals or books (i.e., white literature) and often include web articles, white papers, reports, policy documents, etc. Grey literature was not included in this review for two primary reasons related to reproducibility and efficiency. First, the quality of grey literature is greatly varied because these resources do not usually go through a peer review process before being published. They are also edited, changed, and deleted from their online locations after their initial publication far more often than white literature; consistency and longevity are questionable with grey literature. Second, these resources are not often indexed by databases, making the work of locating them far more intensive. Some grey literature types may be more difficult to find because of the limited ways that are made available, if at all (such as internal reports or some government documents) or, as is the case with online grey literature, the number of resources available becomes exponentially larger with time, making the task of finding them quite Sisyphean.
Article Selection
The following keywords were used to search for articles relating to social justice and evaluation practice: social justice, collective action, social justice advocacy, social justice orientation, social movement, social movement participation, and social justice-oriented evaluation. These keywords were chosen because one of the primary purposes of the review was to understand how the literature connected evaluation work with social justice work. Specific evaluation theories or specific evaluation practices were not selected as search terms because the author did not want to impose a preconceived operationalization of social justice-oriented evaluation onto the literature. Two databases were used for the initial search of peer-reviewed articles, using the above-listed journals and keywords: The Web of Science and New Directions for Evaluation’s online archive of published articles. Excel files with article meta-data (i.e., publication information and abstracts) were downloaded from both databases.
Web of Science did not return any articles from New Directions for Evaluation. Therefore, the same keywords were entered into New Directions for Evaluation’s online archive of articles, and Excel files with article meta-data were downloaded and reviewed. Web of Science also did not return articles from the Journal of MultiDisciplinary Evaluation, and a search for relevant articles was done manually on the journal’s online database. Evaluation Review did not return articles relevant to the topic of social justice and evaluation. Evaluation and Program Planning returned 10 candidate articles; however, none were found eligible to be included in the review.
Inclusion Criteria
Abstracts were scanned to determine if the articles met the following inclusion criteria, chosen by the author: (1) relevant to the interests of the review; (2) published within the past 10 years prior to the review (2012–2022); (3) articles described the connection between evaluation work and social justice work; (4) articles described the need for reevaluating how evaluation is done in order to promote social justice within the field, within projects and evaluands, and in the greater society; and (5) articles discussing evaluation practice in the United States, primarily, though it is acknowledged that some may cover evaluation practices in multiple countries, including the United States, in a more generalized context (e.g., articles from the Canadian Journal of Program Evaluation). Since this review is not an overview of the evolution of social justice-oriented evaluation practice, these criteria were chosen to ensure the review included articles describing social justice-oriented evaluation as it is currently discussed and practiced in the field. Articles included in the review are indicated with a single asterisk (*) in the “References” section.
During the identification phase of the literature search, 210 articles were identified as candidate articles, and 55 articles were selected for screening. After screening and a full article review, 39 articles were included in the review. Figure S.1 in the online supplemental materials illustrates the identification, screening, and inclusion process, and Table S.1 details the number of articles from each journal source.
Limitations
There are a few notable limitations of this literature review method. First, as already mentioned, this review did not include grey literature or other non-mainstream publications. It is anticipated that more details regarding specific social justice-oriented evaluation practices would be found in these sources. Second, this review is limited to the geographic area of the United States. This choice was deliberate, as the review aims to describe the practice of social justice-oriented evaluation within the U.S. culture and context. Third, while the Web of Science offers scholars a fairly robust database of literature, it is recognized that the use of the Web of Science databases may have limited the list of candidate articles for the review. The author amended the Web of Science search results with a search of the New Directions for Evaluation’s online archive due to the journal’s exclusion from the Web of Science databases. Lastly, there is a distinct difference in the appearance of literature written by evaluators and scholars from majority and minority populations. This perhaps speaks to disparities inherent in academic publishing and, as the antecedent, higher education (Hopkins et al., 2013; Willis et al., 2021). Regardless, it is a limitation that needs mentioning.
Literature Review
What Is Social Justice? What Is Social Justice Within the Context of Evaluation Practice?
Evaluation scholar Ernest House is credited with building the foundation for the conceptualization of social justice within evaluation (Bledsoe, 2014; Griffith & Montrosse‐Moorhead, 2014). House has written much on the idea of justice and the use of justice theories in evaluation (House, 1980a, 1980b, 1990, 2019). In his seminal work, House argued that evaluators should not only look for what is true but should also strive for justice in their evaluations (House, 1980a). He concluded that the Rawlsian “justice as fairness” theory of justice (Rawls, 1999) should guide evaluators in their pursuit of social justice in their evaluation practice (House, 1980a). Rawls’ “justice as fairness” theory of justice, where everyone holds the same basic rights and freedoms, has an equal opportunity to fulfill their own potential, and any inequalities within society are minimized.
Evaluation scholars have followed House’s example and have adopted the Rawlsian take on social justice, defining social justice similarly to how Thomas and Campbell define social justice: “equitable distribution of wealth, opportunities, and privileges within a society; maintains that all individuals deserve and should have access to the same rights and resources” (2020). The phrasing of each evaluation scholar’s definition of social justice varies somewhat, but similar themes arise: themes of equal access to civil/human rights, opportunities, and resources (Bitar, 2015; Bledsoe, 2014); appropriate distribution of burdens (i.e., one group is not disadvantaged to the benefit of another group) (Collins et al., 2014); focused considerations given to marginalized groups in order to more equitably distribute resources and opportunities (Desivilya Syna et al., 2015); and the weighing of social exclusion and inclusion when considering issues of social justice within the evaluation context (Lapidot-Lefler et al., 2015).
Principles of Social Justice as Discussed in Evaluation Literature
There are several principles, or foundational concepts, of social justice discussed in the evaluation literature: diversity, equity and equality, power and privilege, culture and cultural competence, inclusion and participation, and advocacy and disruption.
Diversity
Diversity refers to all of the ways that people can differ from one another. These include, but are not limited to, demographic differences such as gender, race, sex, ethnicity, disability, etc.; culture and identities; values and beliefs (Boyce, Tovey, et al., 2022; Garibay & Teasdale, 2019). Differences in treatment or access to resources are typically associated with a measure of diversity; people are treated unfairly because of their race or receive differential educational opportunities because of their socioeconomic status, as examples. Social justice is primarily concerned with how inequality and discrimination show up and align along the various lines of diversity. For this reason, diversity is the enduring backdrop for social justice, the ever-present reality, and for the remainder of this review paper, Diversity will not be named as a separate social justice principle.
Equity and Equality
Equity and equality are often used interchangeably, but these terms are not direct synonyms. As typically used in the literature, equality is concerned with equal rights, opportunities, and resources. Equity is concerned with a fair and just distribution of resources or opportunities so that everyone can achieve the same outcome. For example, suppose a school is putting together a science competition team, and any student who is interested can join the team. Then, the school will tutor different students on different science topics depending on where they need additional help. Here, every student was afforded the same opportunity to join the science competition team (equality) but then also supported in ways that might have differed for each student based on their own strengths and needs so that everyone was successful on the team (equity).
Social justice is concerned with ensuring that everyone has the same rights and opportunities (equality) and that everyone has the resources and support they need to take advantage of those equal rights and opportunities and can thrive wherever they are (equity). Within social justice-oriented evaluation, evaluators consider issues of equitable access to a program, services, and/or opportunities and examine sources of inequality, both within the evaluand and as part of understanding context and culture (Boyce, Tovey, et al., 2022; Garibay & Teasdale, 2019; Gruskin et al., 2015; Hay, 2017).
Power and Privilege
“Power” can take on a myriad of definitions and conceptualizations, but definitions can be simplified to “the ability to influence.” According to scholars Fox and Post, power can be expressed either explicitly or indirectly and can be held by individual people, groups of people, or institutions and social structures (e.g., a government, male-female dynamics) (2021). Related to power is the concept of privilege. Thomas and Campbell define privilege as “unearned access to resources that are only readily available to some people because of their group membership” (Thomas & Campbell, 2020). Those in privileged positions hold power, and this power could be used to maintain or disrupt the status quo. Evaluators who examine power and privilege are often attempting to identify who holds power and who does not, who experiences inequality and who does not (Caldwell & Bledsoe, 2019; Garibay & Teasdale, 2019; Ginsberg, 2021; Reid et al., 2020; Silver, 2021; Symonette et al., 2020; Wolfe et al., 2020).
Culture and Cultural Competence
Social justice work cannot be done without regard for culture (Bledsoe, 2014). Culture is “the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating” (American Evaluation Association, 2011, p. 3). Often, mistreatment and discrimination occur in the interactions between two cultures, most typically the White dominant culture and a minoritized culture. Understanding the cultures at play (and, therefore, the power dynamics) is often the first step in working toward social justice. Working within different cultures takes cultural competence, a skill that requires iterative reflection and learning, respect as the default, and empathy (Brewington & Hall, 2018; Caldwell & Bledsoe, 2019; Garibay & Teasdale, 2019; Symonette et al., 2020).
Inclusion and Participation
Who is welcomed and allowed to participate in systems and experiences is of concern to social justice advocates. Inclusion is examined, and questions are raised about who is not participating and why (Bitar, 2015; Boyce, Tovey, et al., 2022; Gruskin et al., 2015. Genuine participation by all relevant groups—particularly by those historically left out—in decision-making (broadly, in society) is considered a cornerstone of social justice work (Brewington & Hall, 2018; Collins et al., 2014; Desivilya Syna et al., 2015; Garibay & Teasdale, 2019; Gruskin et al., 2015; J. N. Hall, 2020; M. E. Hall, 2020; Lustig et al., 2015; Meléndez, 2012; Woelders & Abma, 2015).
Advocacy and Disruption
A core part of social justice work is advocacy and disruption. Advocacy means actively working toward a cause or goal and disrupting (or attempting to disrupt) existing conditions (Raynor et al., 2021). In the evaluation field, there is debate about the role of the evaluator and whether advocacy has a place in the field. There are those who feel that advocacy is a component of evaluation or is naturally integrated into evaluation (opponents to the “value-free” orientation of evaluation) (Caldwell & Bledsoe, 2019; Ginsberg, 2021; McBride et al., 2020), and there are those who feel that evaluation, and therefore evaluators, should maintain a distanced position so as to introduce as little bias as possible (Archibald et al., 2018).
Defining Social Justice-Oriented Evaluation
As with definitions of social justice, social justice-oriented evaluation fails to have a simple, direct definition. In their textbook, Thomas and Campbell (2020) offer this definition of social justice-oriented evaluation: We define a social justice-oriented model of evaluation as one that depends on a culturally competent framework and a social justice aim, but goes further. It includes a strength-based perspective that focuses on uncovering assets and strengths of communities, particularly marginalized communities, rather than simply uncovering problems. A social justice-oriented evaluation includes basic assumptions and methods that help to ensure that the evaluation will have social justice implications regardless of the topic. It plans for a process of learning and not simply judging and infuses the cultures of the target individuals, program implementers, groups, and communities throughout the evaluation inquiry process. Social justice evaluations situate the evaluand within its sociocultural, historical, political, and organizational context to better understand the antecedents and links between interventions and outcomes. (p. 153)
This definition illustrates part of the trouble with defining social justice-oriented evaluation: the concept is too large for any one definition to capture all aspects. It eschews conciseness, refusing easy description. Ask an evaluator to define it, and they’ll provide an essay. This is perhaps partly due to the dynamic nature of social justice, where what is crucial, salient, and in need of immediate attention ebbs and flows. One of the most powerful examples of this came with the death of George Floyd at the hands of police officers. Where racial inequities and discrimination were of concern to many, that concern swelled to almost unprecedented levels after his death (Pew Research Center, 2020a, 2020b).
The other issue contributing to its lack of a concise definition could be the relative novelty of the term “social justice-oriented evaluation.” The first mention of the term “social justice-oriented evaluation” in the evaluation literature was used by Thomas and Madison (2010). Other scholars have used phrases such as “social justice evaluation” (Caldwell & Bledsoe, 2019; Guajardo et al., 2020) or “transformative evaluation” (Bledsoe, 2014; Cook, 2015; Guajardo et al., 2020; Neubauer & Hall, 2020) to mean evaluations with a focus on social justice as a primary goal.
Regardless of the exact phrase used, evaluators have been speaking about social justice-oriented evaluation for decades in their conversations about evaluation practice. However, they usually discuss social justice-oriented evaluation in terms of specific evaluation approaches or methods. They speak of deliberative democratic evaluation or participatory evaluation (Bledsoe, 2014; Brewington & Hall, 2018; House, 2019; Lustig et al., 2015; Wolfe et al., 2020). They speak of attending to sources of inequality, positions of power, and context and culture (Bledsoe, 2014; Cook, 2015; Dhaliwal et al., 2020; Garibay & Teasdale, 2019; Ginsberg, 2021; Gruskin et al., 2015; House, 2019; Neubauer & Hall, 2020; Woelders & Abma, 2015; Zoabi & Awad, 2015). For example, when discussing the use of hip-hop as an evaluation approach, Brewington and Hall describe social justice-oriented evaluation approaches as “participatory, culturally and contextually responsive, and self-empowering” (2018). As many evaluation scholars have done, they have identified social justice-oriented evaluation practices as a proxy for defining social justice-oriented evaluation.
Why Should Social Justice Matter to Evaluators?
Wicked Problems
Evaluation scholars agree that we are currently experiencing some of the toughest social problems around the world today, including poverty and economic inequity, violence, lack of adequate healthcare, and disparate access and participation in our democratic, economic, and education systems (Boyce, Reid, et al., 2022; Boyce, Tovey, et al., 2022; Collins et al., 2014; McBride et al., 2020; McKegg, 2013, 2019; Symonette et al., 2020). These problems are often referred to as “wicked” problems, a term first coined by Rittel and Webber (1973) to refer to the complexity of these issues, which often require multiple solutions.
These “wicked” problems are our domain. We conduct evaluations of programs and policies to determine if the solutions program planners and policy makers have come up with are making the impact they are intended to. These “wicked” problems persist due to systemic inequality, exclusion, discrimination, etc., which are issues of concern to social justice. Therefore, to adequately examine the proposed solutions to these “wicked” problems, evaluators should attend to social justice and its principles (outlined in the previous section) in the course of conducting evaluations (Hopson & Cram, 2018).
Professional and Ethical Imperative
According to the AEA, evaluators have a professional and ethical imperative to consider issues of social justice in their evaluation practice. This responsibility is spelled out, in part, in the core philosophical and guiding documents of the evaluation profession in the United States: The American Evaluation Association Guiding Principles, The American Evaluation Association Evaluator Competencies, and the Program Evaluation Standards (American Evaluation Association, 2018a, 2018b; Yarbrough et al., 2011). The American Evaluation Association explicitly names social justice as a goal of evaluation and requires evaluators as one of the core competencies to be able to articulate how evaluation can promote social justice aims (American Evaluation Association, 2018b).
The Guiding Principles on respect for people and common good and equity dictate that evaluators honor the agency and perspectives of all individuals and work to address issues of social justice and equity in evaluation work (American Evaluation Association, 2018a). All five of the evaluator competency domains touch on considerations of social justice, but the domains of professional practice, context, and interpersonal specifically address attending to issues of social justice and attending to culture, power, and privilege (American Evaluation Association, 2018b). The Program Evaluation Standards of responsive and inclusive orientation, contextual viability, and human rights and respect underscore the importance of attending to social justice issues in evaluations (Yarbrough et al., 2011).
Evaluation’s Privileged Position
Evaluators hold an incredibly privileged position in any evaluation. Evaluators decide who has access to the evaluation, who can participate, and whose viewpoint matters; we decide which data to collect and what they tell us about what is real; our judgments and recommendations can have significant impacts on what comes after an evaluation (e.g., program continuance or termination, distribution of services) (Aponte-Soto et al., 2014; Boyce, Tovey, et al., 2022; McBride et al., 2020; Symonette et al., 2020).
Because of our position of power, some evaluation scholars argue that evaluations must consider issues of social justice or else risk “condoning and perpetuating oppressive systems and practices” (Boyce, Tovey, et al., 2022, p. 20). If the status quo is not questioned, then what can be changed? If nothing is changed, then what was done before will continue. If what was done before was oppressive or unfair, and we have not brought attention to it or have not worked to change it, then we’re complicit in its continuation. Inaction, regrettably, does not absolve evaluators.
What Does Justice Work in Evaluation Currently Look Like?
Social Justice-Oriented Evaluation Philosophical Foundations
Social justice-oriented evaluation philosophy is largely influenced by the transformational paradigm of research and evaluation (Mertens & Wilson, 2019). The axiological stance is that ethical research and evaluation should promote human rights and social justice, be respectful and responsive to cultural norms, acknowledge the strength of the community, and give back to the community wherever possible. The ontological stance is that there are multiple realities that are shaped by our social positionings. Evaluators should understand the consequences of privileging certain realities over others, often using this understanding to promote social justice by highlighting underrepresented or oppressed voices. The epistemological stance is that knowledge is socially and historically situated and constructed primarily by those with power and privilege. Evaluators should work collaboratively with the community to create a shared understanding of what is known and understand the context within which the community lives, challenging what has historically been understood as legitimate knowledge, if necessary. The methodological stance is that decisions regarding methods should be informed by what will best facilitate the use of findings to promote social justice and identify sources and reinforcers of existing (undesired) conditions as well as catalysts for desired change. This typically begins with qualitative information gathered through dialogue with the community and evolving into a mixed methods data collection plan. The philosophical orientations of social justice-oriented evaluation are summarized in Figure 1. Philosophical Positioning of Social Justice-Oriented Evaluation. Note. Axiological position = values and ethics and their role; ontological position = view of reality; epistemological position = how reality is known/what is knowledge; methodological stance = procedure for gaining knowledge
Evaluation Theories and Models
Social Justice-Oriented Evaluation Theories and Models’ Frequency of Appearance in Review
It should be noted that the following theories and models are not mutually exclusive. Often, each model incorporates the principles of other theories and models. For example, while Culturally Responsive Evaluation (CRE), a transformational model, might prioritize social justice aims, stakeholder participation is critical to the successful implementation of CRE.
Theories and Models from the Transformational Paradigm
Theories and models from the transformational paradigm are characterized by the prioritization of social justice aims. They lie along the Social Justice branch of the evaluation theory tree and pay particular attention to power, privilege, equity, and the incorporation of marginalized voices and perspectives (Mertens & Wilson, 2019). Culture is often central to the evaluation because traditional evaluation and research methods may not be appropriate for all communities; traditional methods center the perspectives of the White majority. Approaches like Culturally Responsive Evaluation and Culturally Responsive and Equitable Evaluation frame the evaluation in terms of the culture and context of the evaluand, resulting in an evaluation plan that honors the culture and norms of the community and addresses issues of social justice in an organic way (Bledsoe, 2014; Brewington & Hall, 2018; Garibay & Teasdale, 2019). Transformational theories and models like Equity-Focused Evaluation and Human Rights-Based evaluation have equity as the central goal of the evaluation. These models set out to interrogate power and structural inequities and gather evidence in the service of dismantling the systems that continue to unfairly advantage some over others (Fox & Post, 2021; Guendel, 2012). Because of their focus on social justice, these theories support the aims of all social justice principles but emphasize the principles of Equity and Equality, Power and Privilege, and Culture and Cultural Competence.
Critical Theory-Influenced Theories and Models
Theories and models that take their foundation from various critical theories prioritize the inclusion of perspectives from specific marginalized communities, such as Black/African American communities (critical race theory), women (feminist theory), Hispanic communities (LatCrit theory), and LGBTQA+ communities (queer theory). These theories require the interrogation of power structures and an examination of equity and the sources of inequity, particularly along identity lines (e.g., Black vs. White, heteronormative vs. queer) (Archibald et al., 2018; Ginsberg, 2021; Guajardo et al., 2020; M. E. Hall, 2020; Harnar, 2014). Consideration of context and sociopolitical climates are integral to these theories and models, for an understanding of the sources of inequity and the locations of power cannot be formed without understanding context (Archibald et al., 2018; Ginsberg, 2021). Like the transformational paradigm theories and models, Critical Theory-Influenced Theories support the aims of all social justice principles, but they emphasize the principles of Power and Privilege, Cultural and Cultural Competence, and Advocacy and Disruption.
Engagement Theories and Models
Engagement theories and models guide the participation of stakeholders in the evaluation process. These theories and models discuss who should be included and how to include them, with historically disadvantaged and marginalized communities given new priority. For example, Deliberative Democratic Evaluation provides guidance on conducting an evaluation through democratic processes, incorporating the interests and values of all stakeholder groups involved (Bledsoe, 2014; House, 2019). When additional stakeholder groups are involved in the evaluation decision-making, current power structures within the evaluand and community contexts are examined and often disrupted. This stakeholder participation necessitates the consideration of culture and context throughout the evaluation, and, when done well, the evaluation becomes infused with the cultural norms of the community. Because of the priority given to stakeholder participation, these evaluation theories and models are particularly suited to address the social justice principles of Power and Privilege and Culture and Cultural Competence, in addition to the primary principle of Inclusion and Participation.
Taxonomy of Social Justice-Oriented Evaluation (T-SJOE) Practices
This section will discuss a proposed taxonomy of social justice-oriented evaluation (T-SJOE) practices. The T-SJOE is meant to be a tool for evaluators; it is not meant to be prescriptive but to serve as a guide for self-reflection and evaluative thinking and as a reference for practices to incorporate into a social justice-oriented evaluation. The taxonomy was developed from the current academic literature on social justice-oriented evaluation practices and, therefore, limited in its comprehensiveness. It is not presumed to be complete, as some practices may only be found in grey literature. The T-SJOE is considered mutable and subject to updating, changing, and adaptation as new methods, techniques, and practices are developed and shared with the field.
This taxonomy was developed through the constant comparative method, an iterative coding process of the literature included in this review (Glaser, 1965). Coding was completed using the MAXQDA qualitative analysis software. Social justice-oriented evaluation practices were inductively coded in a first-level, open coding process. These codes were then reviewed, refined, and recoded as necessary. While the coding process was conducted solely by the author, initial and refined codes were shared in multiple sessions with two colleagues to gather feedback and identify areas for refinement or expansion of definitions, themes, and exemplars.
The first-level codes are the individual social justice-oriented evaluation practices reported in the literature. There were 58 practices identified, such as “critical reflection” and “establishing social justice norms” (see the online supplemental materials for the full list of practices). Next, the first-level codes were analyzed for emergent themes and grouped into similar practices, and new second-level codes (areas) were developed. These themes were reviewed and refined. There were 10 areas identified. These second-level codes were analyzed and grouped into higher-level dimensions of practice. There were three dimensions identified. Figure S.2 in the online supplemental materials illustrates the coding process.
Social Justice-Oriented Evaluation Practice Frequency of Appearance in Review, Context = Personal Professional Growth
Note. The total column reflects the number of unique articles in each sub-theme. This total may not be equal to the sum of the number of articles in each context because an article may be counted in multiple contexts.
Social Justice-Oriented Evaluation Practice Frequency of Appearance in Review, Context = During Evaluation
Note. The total column reflects the number of unique articles in each sub-theme. This total may not be equal to the sum of the number of articles in each context because an article may be counted in multiple contexts.
Social Justice-Oriented Evaluation Practice Frequency of Appearance in Review, Context = Evolving the Field
Note. The total column reflects the number of unique articles in each sub-theme. This total may not be equal to the sum of the number of articles in each context because an article may be counted in multiple contexts.
Social Justice-Oriented Evaluation Practice Frequency of Appearance in Review, Context = Beyond the Evaluation
Note. The total column reflects the number of unique articles in each sub-theme. This total may not be equal to the sum of the number of articles in each context because an article may be counted in multiple contexts.
The second domain, Social Justice Principle, contains practices related to one of the social justice principles discussed above. The areas of Advocacy and Disruption, Culture and Cultural Competence, Equity/Equality, Inclusion and Participation, and Power and Privilege all contain practices that center those particular social justice principles. These practices are more external than those in the Deconstruction domain and are often done during the planning and implementation stages of an evaluation. Some of the practices in this domain are “identifying levers of change,” “consideration of culture,” “attention to gender equality,” “stakeholder participation,” and “power-building.”
The last domain, Skills/Competencies, discusses the particular skills that are most important to social justice-oriented evaluation. These skills are grouped by the areas of Interpersonal, relating to the “people skills” required of evaluators such as “on-going dialogue/communication,” and Technical, relating to the set of specialized tasks that can be carried out in an evaluation such as “multicultural validity.”
Tables 2–5 summarize the social justice-oriented evaluation practices’ frequency of appearance in the articles in the review. The practices are organized by domain and area as well as by context (i.e., Personal Professional Growth, During Evaluation, Evolving the Field, and Beyond the Evaluation) and actor (i.e., Evaluator, Client/Funder, Professional Associations/Field at Large). While reviewing the practices discussed in the literature, it was found that each practice was discussed in terms of when it was most appropriate to be done and by whom. For example, Deep Examination—an area of practice that involves critical reflection and examination of deeply held beliefs and opinions—is cross-cutting in nature, applicable to all contexts and all actors. On the other hand, technical skills such as “ensuring the multicultural validity of data collection instruments” are primarily relegated to evaluators and the evaluation project context rather than practices used to evolve the field or promote social justice beyond the evaluation. The social justice-oriented evaluation practices are presented as four separate tables, one for each context, to facilitate discussion of the practices.
The actors discussed in this taxonomy are Evaluator, Client/Funder, and Professional Associations/Field at Large. Evaluator refers to the individual evaluator conducting social justice-oriented evaluation. Client/Funder refers to the client, program participants, program staff, policymakers, evaluation commissioner, or funder; essentially, stakeholders of the evaluation that do not formally include members of the evaluation team. Professional Associations/Field at Large refers to any and all professional associations that lead the field of evaluation, such as the American Evaluation Association or the Canadian Evaluation Association, and the totality of the membership of the field.
The contexts are conceptualized in terms of the sphere of influence of each individual evaluator, beginning with the evaluators themselves and ending with influence beyond the evaluation and out into society. Figure 2 illustrates this conceptualization. Personal Professional Growth refers to practices that are conducted for the development of evaluators’ skills, competencies, beliefs/opinions, philosophical orientations, etc. During Evaluation refers to the practices that are conducted during an active evaluation project. Evolving the Field refers to practices done to move the field forward in terms of social justice orientations and standards of practice. Beyond the Evaluation refers to practices that are done to positively impact society beyond the scope of the evaluation project, usually in the form of advocacy/mobilization or in the service of advocacy. Conceptualization of Contexts
Context: Personal Professional Growth
The Personal Professional Growth context refers to practices that are done for the development of evaluators’ skills, competencies, beliefs/opinions, philosophical orientations, etc. As can be seen in Table 2, the evaluation literature is rife with practices the individual evaluator can do for personal professional growth. Deep Examination practices such as critical reflection were, by far, the most referenced practices for evaluators (Archibald et al., 2018; Boyce, Reid, et al., 2022, Boyce, Tovey, et al., 2022; Caldwell & Bledsoe, 2019; Collins et al., 2014; Dhaliwal et al., 2020; Garibay & Teasdale, 2019; Ghanbarpour et al., 2020; Guajardo et al., 2020; J. N. Hall, 2020; M. E. Hall, 2020; McBride et al., 2020; Neubauer & Hall, 2020; Symonette et al., 2020). For example, evaluators are encouraged to critically reflect on their role in society and consider what the ultimate goals of evaluation are and should be (Archibald et al., 2018).
Clients and funders had no practices attributed to them in this context, suggesting, perhaps falsely, that clients and funders have no hand in the personal growth of individual evaluators. Professional Associations/Field at Large have a few practices in this context, and these are primarily in the service of helping individual evaluators grow and develop into social justice-oriented evaluation practitioners. For example, the literature discusses how associations like the AEA can promote embracing diverse epistemologies by offering professional development training to evaluators (Caldwell & Bledsoe, 2019).
Context: During Evaluation
The During Evaluation context refers to the practices that are done during an active evaluation project (i.e., from planning to implementation to dissemination of the results of an evaluation). For example, the AEA’s Digital Knowledge Hub is a source of on-demand training for evaluators and includes a “Social Justice Series” of trainings (American Evaluation Association, no date).
Table 3 summarizes the social justice-oriented evaluation practices in this context. Again, evaluators have the most practices assigned to them during this context, then Client/Funder, and then Professional Associations/Field at Large. For evaluators, practices that support the sub-theme of Participation are the most frequently discussed in the evaluation literature. These practices primarily support the involvement of various stakeholders in the evaluation project, centering their perspectives and voices in the evaluation process, and ensuring the evaluation is inclusive and accessible (Bledsoe, 2014; Brewington & Hall, 2018; Cook, 2015; Meléndez, 2012; Woelders & Abma, 2015; Zoabi & Awad, 2015). Other practices discussed frequently were Deep Examination, Advocacy and Disruption, Equity/Equality, Power and Privilege, and Technical, all with nine articles discussing practices in each of these sub-themes. For example, an important part of Advocacy and Disruption is ensuring that the community has the capacity to facilitate change. The evaluator may engage in capacity building with the community to ensure that (1) the community is able to make needed changes and (2) the evaluation is used and has influence after the evaluation is completed (Cook, 2015; Dhaliwal et al., 2020).
Client/funder practices primarily involve working with evaluators to address social justice principles in the evaluation. The most frequently discussed practices involve attention to power and inequities within the evaluand (Cook, 2015; Woelders & Abma, 2015). For example, clients can work with evaluators to present findings in ways that are understandable and actionable by those with the most power to create change, such as policymakers (Cook, 2015). Also discussed were practices to help build power within the communities served by the evaluand (Fox & Post, 2021). Power-building refers to redistributing power to and building the capacities of communities affected by inequities to bring about positive social change. Critical reflection and building connections were also discussed as practices that clients and funders can do to support social justice-oriented evaluation efforts (Caldwell & Bledsoe, 2019; Cook, 2015; Garibay & Teasdale, 2019; Lustig et al., 2015). For example, clients can call together community groups with similar goals and then facilitate working sessions to discuss and plan actions, events, and programs to bring about community-led change at the local level (Cook, 2015).
Little is reported in the evaluation literature on what professional associations and the field at large could do to support social justice-oriented evaluation aims during evaluations. Practices that professional associations and the field at large could do in this context involve providing support to evaluators as they deconstruct and implement their practice. This support primarily looks like making available resources and readings and providing just-in-time (i.e., asynchronous) education on new ways of thinking (Garibay & Teasdale, 2019; J. N. Hall, 2020). For example, the AEA’s Digital Knowledge Hub is a source of on-demand training for evaluators and includes a “Social Justice Series” of trainings (American Evaluation Association, n.d.).
Context: Evolving the Field
This context refers to practices done to move the field forward in terms of social justice orientations and standards of practice. Table 4 summarizes these practices. In this context, individual evaluators and Professional Associations/Field at Large have approximately equal obligations. Practices to evolve the field lie primarily within the Deconstruction domain. Practices not generally discussed in other contexts are those associated with establishing new norms for evaluation practice. Scholars argue that if the field wants social justice to be an explicit value of the field, then professional associations must provide recognition for social justice-focused work, support social justice-oriented evaluation initiatives, and accredit social justice-oriented training programs (Caldwell & Bledsoe, 2019; J. N. Hall, 2020). Evaluators must do the hard work of deconstructing and critically examining their practice (Archibald et al., 2018; Caldwell & Bledsoe, 2019; Garibay & Teasdale, 2019; J. N. Hall, 2020; Neubauer & Hall, 2020). Clients and funders can help evolve the field by prioritizing the funding of social justice-focused contracts (Caldwell & Bledsoe, 2019).
Context: Beyond the Evaluation
Practices in this context are those done to positively impact society beyond the scope of the evaluation project. This could look like using evaluation findings to inform new policies or lobbying local, state, and federal governments to support social justice laws and initiatives. Table 5 summarizes the practices discussed in the literature. Surprisingly, the ways in which clients and funders can positively impact society were discussed less frequently than for individual evaluators. It would be logical to assume that once an evaluation project is complete, the next steps would include the client/funder acting on the findings and recommendations to continue to build momentum toward positive social change. It would then make sense for the literature to discuss at length the practices that clients and funders engage in after the evaluation, perhaps as articles co-written by the evaluation and client teams discussing the efforts made at the local level after the conclusion of an evaluation. However, these sorts of articles are currently absent from the literature. The practices of evaluators are discussed far more frequently, creating an exaggerated sense of responsibility on the part of evaluators to take evaluation findings and act on them themselves to create positive social change.
Advocacy and disruption practices are the most discussed practices for evaluators; however, scholars primarily talk about providing findings that set up others to do the work. For example, Cook argues that part of the evaluator’s job is to ensure that communities have the capacity to effect change on their own after the evaluation (2015). This is a noble and appropriate goal for the evaluation, but one that feels like evaluators prepare others to affect change but ultimately have “no skin in the game,” so to speak. Very few scholars talk about the evaluator joining the efforts personally. This is not necessarily a criticism, but it leaves unanswered the question of “what is our responsibility once the evaluation is over?” This question is something that the field is discussing and has always discussed. And perhaps the answer is a personal one: What is my responsibility after the evaluation is over? What do I feel that I need to do now? For some contexts, it is appropriate for the client to “take it from here” and move forward with findings and action steps. In other contexts, perhaps the evaluator feels compelled to join the efforts of the client in a more engaged role or as a community member.
For professional associations and the field at large, practices center on deconstruction (e.g., critical reflection, embracing diverse epistemologies). Discussions on the ways professional associations and the field at large can create a positive impact beyond the evaluation center on deconstructing and evolving research, particularly social science research, methods to include new ways of thinking and conducting social inquiries (Caldwell & Bledsoe, 2019; J. N. Hall, 2020).
What Are the Critiques and Challenges to Social Justice-Oriented Evaluation
While the field of evaluation generally professes a commitment to addressing social justice issues, some practitioners are less inclined to add social justice-oriented evaluation practices to their professional toolkit. Many express concerns about the role of evaluators and the appropriateness of addressing social justice, or fear that conducting evaluations through a social justice lens may undermine their credibility. The most common critiques of social justice-oriented evaluation often center on methodology, with many critics voicing apprehension about approaches that diverge from conventional Western research methods. They question the rigor, objectivity, and potential biases of social justice-oriented evaluation methods.
Rigor
Some evaluators contend that engaging in social justice-oriented evaluation approaches means engaging in less rigorous inquiry. They advocate for careful control of as many variables as possible in an evaluation study, such as in randomized controlled trials (RCTs), to more accurately attribute cause (Levin-Rozalis, 2015; Thomas & Campbell, 2020). However, given that evaluations are conducted “in the wild” within complex systems that involve numerous external factors, this approach is rarely truly possible in evaluations. As Thomas and Campbell explain, “an overemphasis on [traditional notions of] rigor can cause some voices not to be heard equally, or at all, and can allow important differences to cultural context to be left out, making an evaluation less strong” (2020, p. 149). Evaluation scholars have argued that what results from evaluations that place a strict adherence to traditional notions of rigor is an evaluation that is unable to answer complex questions, has likely harmed participants in some way, and is ultimately less useful (Silver, 2021; Thomas & Campbell, 2020).
Current definitions of rigor typically prescribe a narrow approach to inquiry. Questions are framed in a particular way, data collection is often restricted to quantitative sources and specific collection methods, and even data interpretation can be highly prescribed. These prescriptions value one particular way of knowing and preclude all others, specifically ontologies and epistemologies of marginalized communities. “Rigor … is socially constituted and reflective of mainstream presumptions and social hierarchies in a specific time and place” (Riley, 2017, as cited in Thomas & Campbell, 2020).
Objectivity
Closely related to rigor is the notion of scientific objectivity. Some argue that a lack of scientific detachment threatens the scientific validity of an evaluation. Inquiry into issues of social justice undermines the evaluator’s objectivity; therefore, the evaluation lacks validity (M. E. Hall, 2020). Personal notions of social justice compromise the objectivity of results (Levin-Rozalis, 2015; Symonette et al., 2020).
Critics argue that the focus should be on whether the evaluand achieves its intended outcomes, viewing other considerations, such as social justice issues, as irrelevant to their role. However, the evaluand exists within a broader context, often as a response to societal disparities or problems. Ignoring these social issues risks undermining both the evaluation’s design, such as appropriately tailoring it for the community, and the interpretation of findings, which must reflect the community’s realities (Bledsoe, 2014).
Silver suggests that evaluators adopt a “passionate detachment,” meaning they should critically engage with sources of inequity while remaining open to what the data reveals (Silver, 2021). In the words of the esteemed mathematician and statistician J. P. Benzécri, “The model must follow the data, not the other way around” (Benzécri, 1973). He is speaking explicitly about building new mathematical models, but his point also applies more broadly to interpreting data and findings from research and evaluation: let the data speak for itself. Evaluators should allow their convictions to motivate critical inquiry into systems and established norms while allowing the evidence to inform their conclusions, even if that evidence is damning.
Bias
Concerns about bias in evaluations are legitimate and widespread. Evaluations that are social justice-oriented imply that the evaluator holds a certain set of beliefs, often perceived to be personal and emotionally charged (Levin-Rozalis, 2015). However, all evaluators hold ontological, epistemological, and axiological beliefs that inevitably influence their evaluation work, shaping the methods they choose and how they interpret the data collected. Evaluation is never “value free”; beliefs about appropriate methods, data, analysis, and interpretations are deeply entrenched in all evaluation processes (Archibald et al., 2018; Garibay & Teasdale, 2019; J. N. Hall, 2020; Silver, 2021; Zoabi & Awad, 2015).
For example, the evaluator who chooses to use experimental and quasi-experimental designs in their evaluations and gather quantitative data upon which they can conduct statistical hypothesis testing does so because they believe these methods produce the most valid and credible evidence. They are likely positivist or post-positivist in their stance. This isn’t wrong, but it is a stance and a belief that is imposed on their evaluation practice. And because it is a belief, there is bias. The evaluator likely believes that she has done all she can to reduce the bias in her evaluation by choosing this specific methodology. However, we know that bias is unavoidable; the trick is to use methods and approaches that make the level of bias tolerable.
Other concerns regarding the introduction of bias stem from participatory evaluation methods. Critics of participatory research methods argue that research participants or program participants involved as partners in the evaluation study cannot remain objective and will impose their viewpoints and judgments to the detriment of the validity of the study. Including program participants can, however, provide an “epistemic advantage to enrich an evaluation study” (Silver, 2021). Multiple perspectives provide richer data, adding color and texture to a story that may only have been done in greyscale without research participant involvement. Silver used the example of the inclusion of incarcerated women and women in college in the evaluation of a prison college program to illustrate how participant involvement can strengthen the validity of an evaluation. Research participants co-constructed knowledge and explanations with the evaluation team, providing insights that brought value and legitimacy to the study (2021). Multiple and new perspectives can widen the scope of the evaluation beyond traditional lines of inquiry to answer new and bigger questions that ultimately allow for a deeper understanding of a program’s impact.
Scholars explain that bias is an unavoidable artifact of being human and interacting with other people. We bring our biases everywhere we go, including into the work of evaluation. Sometimes we know what our biases are (explicit), and sometimes we are unaware of them until they are pointed out to us (implicit) (Brownstein, 2015; Schultz & Woodcock, 2023). Advocates of social justice-oriented evaluation approaches argue that biases cannot be eliminated entirely, but they can be balanced by incorporating participatory approaches into evaluations (Thomas & Campbell, 2020). The incorporation of multiple perspectives not only honors social justice principles but also acts to mitigate the impact of the evaluator’s biases on the evaluation.
Closing Thoughts on Critiques to Social Justice-Oriented Evaluation
Social justice-oriented evaluation calls for evaluators to reconsider what they regard as valid evidence and inquiry methods. It prompts them to critically reflect on their own practice, a process that can be quite uncomfortable for some. Many might even shy away from this self-examination, as questioning their training and what they have been taught is rigorous and valid inquiry can challenge the foundations of their beliefs and identities as evaluators or raise concerns about their perceived credibility. However, the literature on social justice-oriented evaluation argues that valuing multiple perspectives, approaches, and ways of knowing expands the evaluators’ toolkit and practice. Incorporating social justice-oriented evaluation methods can enhance evaluation practice while still maintaining a high level of rigor and validity.
Future Work
The introduction of a newly developed model, framework, or tool, such as the T-SJOE, is not the final step of a scholarly project; the work has only just begun. The T-SJOE is intended to be a living document, a tool to help advance social justice-oriented evaluation aims within the field. Thus, the author offers the following suggestions for further development of the T-SJOE.
The taxonomy was developed using only the practices identified in this review; it is not assumed to be complete, given the amount of grey literature in the field that might offer more social justice-oriented evaluation practices. Additions and modifications to the practices listed in the taxonomy informed by review of grey literature are welcome. Specifically, practices from Indigenous and other non-Western frameworks are needed. Details from these perspectives are likely more abundant in the grey literature.
When reviewing the frequency of social justice-oriented evaluation practices discussed in the evaluation literature, most practices are “assigned” to the individual evaluator. Much of the onus of justice work in evaluation lies on the individual evaluator; clients, funders, and the field at large have significantly fewer obligations—or so it appears. The development of social justice practices and agendas for clients, funders, and professional associations is an identified gap within the evaluation field. Discussions on how evaluators can collaborate with clients and funders to advance social justice after the evaluation are also needed.
As a general critique of the way practices are discussed in the literature, scholars tend to keep descriptions of specific practices vague (unless publishing a case study). For example, several articles discuss the practice of attending to inequalities but fail to describe what this really looks like in practice (Gruskin et al., 2015; House, 2019; Neubauer & Hall, 2020). This might be a reflection of who gets published in evaluation journals (academics, primarily), which is a bit of a paradox for a field dominated by practitioners. There are more concrete examples to be found elsewhere (e.g., grey literature such as web articles, white papers, reports), but no centralized database or warehouse of practices and case examples, which makes it difficult to research social justice-oriented evaluation and difficult to get into the hands of practitioners.
The literature review was based on evaluation practice as conducted in the United States. The T-SJOE, therefore, is initially conceived using social justice-oriented evaluation practices as operationalized within the United States. An exploration of the T-SJOE’s applicability to non-U.S. contexts, informed by non-U.S. literature and traditions, would be illuminating. For example, is the T-SJOE relevant and/or useful to practicing evaluators in the Global South? How would evaluators in the Middle East modify the T-SJOE for use in their geopolitical contexts? Is the T-SJOE of any value to evaluators practicing in Central or East Asia? What use do evaluators in international development get from the T-SJOE, if any?
In Closing
Scholars and practitioners have called for re-framing evaluation practice as a means of promoting social justice. However, the literature often provides insufficient detail about social justice-oriented evaluation, leaving many evaluators uncertain about what it entails. This article summarizes the existing literature on social justice-oriented evaluation, offering a much-needed clarification of the approach. Additionally, it introduces a taxonomy of social justice-oriented evaluation (T-SJOE) practices, derived from the literature review findings, and outlines a framework for these practices as currently discussed in the evaluation field. The T-SJOE is not intended to be prescriptive or exhaustive; rather, it serves as a tool for self-reflection and a resource for evaluation planning. The hope is that it serves as a helpful tool for individual evaluators, clients, funders, and the field at large.
Supplemental Material
Supplemental Material - The Pursuit of Social Betterment Through Evaluation Practice: A Review of the Literature on Social Justice-Oriented Evaluation and a Taxonomy of Practices
Supplemental Material for The Pursuit of Social Betterment Through Evaluation Practice: A Review of the Literature on Social Justice-Oriented Evaluation and a Taxonomy of Practices by Sarah Zlatkovic in Evaluation Review
Footnotes
Ethical Considerations
Not applicable. This study was not human subjects research.
Consent to Participate
Not applicable. This study was not human subjects research.
Consent for Publication
Not applicable. This study was not human subjects research.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Data are not available as this study was primarily a review of existing literature.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
