“A Lot of It Really Does Come Down to Values”: An Empirical Study of the Values Advanced by Seasoned Evaluators

Abstract

This study challenges persistent misrepresentations of evaluation as a value-neutral inquiry process by presenting an empirical study that deepens understanding of evaluators’ values and how they “show up” in evaluation practice. Through semistructured interviews and inductive analysis, we examined the values advanced by a sample of eight experienced evaluators. We surfaced and examined 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies these evaluators conducted and guided how the evaluators positioned their work. Our findings provide empirical evidence about the role of values in evaluation practice and can support evaluators in reflecting on their own values and enacting their professional and ethical responsibilities to identify and articulate their values in the context of evaluation practice.

Keywords

values research on evaluation evaluation practice values engagement

In many societies, social and educational policies beget programs that are evaluated for quality and impact. While policymaking is commonly understood as a political process, the activities of program development, implementation, and evaluation are often perceived as logical, rational, and neutral (Campbell, 1969; Schwandt, 1997). From this standpoint, programs are developed from evidence-based research, and evaluations of those programs are understood to be objective and values-neutral.

Yet within the field of evaluation, scholars have long argued that values are omnipresent in social programming (Shadish et al., 1991) and evaluation rests, fundamentally, on the intertwining of facts and values (Davidson, 2005; Fournier, 2005; House & Howe, 1999). Specifically, values permeate decisions about which evaluation purposes and audiences to address, questions to investigate, criteria to use for judging quality, and designs and methods to employ—and values guide choices about the social relational aspects of the evaluator's role (Greene, 2012; Greene et al., 2011; Gullickson & Hannum, 2019). Given that evaluations are inevitably partial and incomplete, evaluators are urged to make explicit the values that shape their decisions and, ultimately, their findings and conclusions. For example, the Program Evaluation Standards and American Evaluation Association (AEA) Guiding Principles for Evaluators call on evaluators to assess, clarify, and articulate the key values that shape the program and evaluation (AEA, 2018; Yarbrough et al., 2011).

Evaluators might, like Shadish (1994), ponder the implications of not explicitly engaging values:

So what harm would be done if evaluators ignored values? In one sense, no harm would be done, because evaluations would still have values implicit in them […] But in another sense, real harm is done if evaluators deal with values naively or poorly through their implicit choices. (p. 355)

These harms stem from the risk of prioritizing the values of some stakeholders—typically those who exercise the most power—while overlooking or excluding others’ values (Greene, 1997; House & Howe, 1999), as well as failing to recognize how evaluator's values shape the study (House, 2015; Smith, 1980). Doing so could produce a slanted or limited view of the evaluand; lessen credibility, relevance, and meaning for key stakeholders; and reproduce unjust or undemocratic power relationships (Greene, 1997; House & Howe, 1999; Thomas & Campbell, 2021). In addition, reporting an evaluation without attention to the values that shaped it denies readers information about who was and was not included in the framing, how the study unfolded and why, and what was not addressed in the evaluation and why.

Despite their central importance, there has been limited empirical research that explicitly addresses values in evaluation practice (Coryn et al., 2017). This has led to gaps in the knowledge base about how values shape the evaluation process, the nature and source of those values, and how evaluators negotiate competing values—and prompted calls for research that examines values in evaluation practice (Coryn et al., 2017; LaVelle et al., 2022; Roorda & Gullickson, 2019). In response, this article reports findings from an empirical study focused on the nature and character of values held by a sample of seasoned evaluators and how those values “show up” in their evaluation practice.

We conducted semistructured interviews in 2016 and inductively analyzed the transcripts to surface and examine 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies they conducted and guided how the evaluators positioned their work. Our findings challenge persistent misrepresentations of evaluation as a values-neutral endeavor and add to the body of scholarship that illuminates the ways in which values constitute the very heart of evaluation as an empirical and sociopolitical process of judgment.

Values and Evaluation Practice

In evaluation, the word “values” is used to refer to a range of related ideas, including attitudes, opinions, priorities, wants, needs, and desires (Hassell et al., 2020; House & Howe, 1999). In this article, we consider values to be beliefs about what is important, desirable, or preferable that transcend specific situations and motivate or guide decision making and behavior (Hassell et al., 2020; House, 2015; Sagiv et al., 2017).

Individuals hold multiple types of values, including personal, professional, social, cultural, political, and epistemic values (Hassell et al., 2020; Schwandt, 2015). For example, personal values are beliefs about what constitutes a good life that motivate action and serve as guiding principles (Sagiv et al., 2017; Schwartz et al., 2012); cultural values are shared ideas about what a group considers good and desirable (Sagiv et al., 2017; Schwartz, 1999); and political values are beliefs about what should constitute the core responsibilities of government, citizenship, and society (Hassell et al., 2020; Schwartz et al., 2010). The category of professional values includes at least two related types: individuals’ beliefs about what is important in work contexts (Ros et al., 1999) and shared ideas within a professional community about what constitutes desirable or ethical conduct in a specific profession (Chadwick, 1998; Schwandt, 1997). For example, AEA (n.d.) lists “excellence in evaluation practice, utilization of evaluation findings, and inclusion and diversity in the evaluation community” among the organization's values. The AEA Guiding Principles “reflect the core values of the AEA and are intended as a guide to the professional ethical conduct of evaluators” (AEA, 2018, para. 1). These Guiding Principles focus on values of systematic inquiry, competence, integrity, respect for people, and common good and equity.

Apart from attention to professional ethics, discussion of values in evaluation practice was long marginalized due to a dominant view that endorsed a fact-value dichotomy in social science (House & Howe, 1999; Shadish et al., 1991). This viewpoint held that facts were observable and objective, while values were subjective expressions of personal taste. Further, facts and values were to be kept separate, and social science—including evaluation—was to focus on objective facts. This view has been challenged in recent years as social scientists and philosophers have grappled with the profound entanglement of facts and values in the social world (Hassell et al., 2020; Putnam, 2002).

Values are central to many facets of evaluation. These include evaluative judgments about program merit or worth (Scriven, 1991), as well as “value judgments by funders, clients, and evaluators as to what is the desirable or proper form of an evaluation study” (Smith, 1980, p. 104). That is, the evaluative process applies values to empirical data to reach conclusions (Fournier, 2005), and evaluation methodologies themselves reflect and advance values (Greene, 1997, 2002; Gullickson & Hannum, 2019). Currently, discussions of values in evaluation often focus on valuing—the activity of reaching a judgment by specifying criteria, establishing standards, gathering data, and interpreting data to reach a conclusion (Fournier, 1995; Scriven, 2007). This includes recent scholarship addressing the valuing process (Ozeki et al., 2019) and, more specifically, evaluative criteria and standards (Harman & Azzam, 2018; Roorda & Gullickson, 2019; Teasdale, 2021, 2022). However, LaVelle et al. (2022) argue that this focus on valuing neglects “the importance of understanding the individual values held by evaluators and the ways in which those values might indeed influence evaluation practice” (p. 2).

Evaluators’ values shape how they conduct evaluations (House, 2015; Smith, 1980); therefore, the AEA (2018) Guiding Principles urge evaluators to examine and openly articulate their own values, and the Program Evaluation Standards specifically warn against assumptions that “evaluators are objective and able to make value-free judgments” (Yarbrough et al., 2011, p. 40). To understand evaluators’ values, LaVelle et al. (2022) drew on inventories of personal and workplace values and a political values scale. Surveyed evaluators were found to value self-direction (pursuing independent thought and action), universalism (understanding, appreciation, tolerance, and protection for the welfare of all people and nature), achievement (finding personal success by demonstrating competence according to social standards), and benevolence (preservation and enhancement of the welfare of people with whom one is in frequent personal contact). Respondents also valued “competence and opportunities for growth,” a values cluster focused on work that is intellectually stimulating, makes a social contribution, and encourages knowledge and skill development. Respondents’ political values were “very liberal” or “liberal.”

Gullickson and Hannum (2019) considered how evaluators’ personal values might “show up” in evaluation practice. For example, they hypothesize that an evaluator who values self-direction might seek to engage multiple perspectives by employing participatory evaluation approaches and including stakeholders in data interpretation. They posit that an evaluator who values universalism might use transformative evaluation approaches, be open to multiple kinds of information, and prioritize evaluative criteria that address equality and justice. Gullickson and Hannum (2019) note that the values evaluators espouse may not align with the values they enact. That is, explicitly stated values may or may not be “demonstrated through the choices and actions” that constitute evaluation practice (Gullickson & Hannum, 2019, p. 165).

The current article seeks to deepen understanding of evaluators’ values and how those values shape their practice. We report results from an empirical study focused on the values held by a sample of seasoned evaluators. We asked: Which values show up in evaluators’ practices? How do those values show up?

Methodology

We designed an interpretive, qualitative study to answer our research questions because (a) our inquiry was exploratory, (b) we aimed to elicit open-ended reflection about values, (c) we sought to surface both espoused and enacted values, and (d) this methodology was consistent with the values guiding the study. We were aware that values related to evaluation and social programs likely span multiple domains. Thus, rather than limiting respondents to a set of predetermined values, we aimed to invite reflection about their evaluation practice and explore values in that context. Our open-ended, reflective approach was consistent with exploratory values research in other disciplines, including pharmacy (Benson et al., 2009), nursing (Jiménez-López et al., 2016; Ravari et al, 2012), and medicine (Lee et al., 2013).

Research, like evaluation, is not value-free (Putnam, 2002; Schwandt & Gates, 2021). Our inquiry was motivated by our belief in the importance of engaging explicitly with values and grounded in Greene's (1997; Greene et al., 2011) conceptualization of values engagement. We framed the study in the democratic tradition, which goes beyond merely engaging with a plurality of values to center values of inclusion, equality, and social justice (Greene, 2006; Greene et al., 2004; Howe, 2003) and prioritizes understanding of participants’ varying experiences, perspectives, and contexts (Greene et al., 2004; Kushner, 2000).

Data Collection

We¹ conducted interviews with eight experienced North American evaluators to surface the values evident in their evaluation practice and understand how those values shaped that practice. We developed a semistructured interview guide, which we refined through pilot interviews. Questions asked respondents to describe an evaluation they conducted that they considered to be well done, describe a challenging evaluation, and identify aspects of practice that were most important to them. Finally, we asked respondents to discuss the values they endeavored to advance in their work. These open-ended, reflective questions were consistent with exploratory values research in other disciplines. For example, Benson et al. (2009) asked pharmacists to discuss good and challenging situations in their practice. We concluded by asking directly about respondents’ values because we anticipated they would be able to identify and reflect on their own values.

After receiving Institutional Review Board approval, we purposively selected a sample of eight experienced North American evaluators using two criteria (Patton, 2015): (a) each respondent had strong experience as an evaluator and (b) collectively, the group represented a diversity of demographic characteristics and places of employment. We emailed each individual to invite participation in a study of experienced evaluators’ practices. We did not inform them that the study focused on values. Each provided informed consent and was interviewed in March 2016. The sample included five women and three men (Table 1). Six respondents identify as white and two as people of color (race and ethnicity are not reported to protect anonymity). Respondents’ evaluation experience ranged from 5 to 9 years to more than 30 years and spanned a variety of professional contexts. One respondent lived outside of North America. Two researchers jointly conducted each interview, which ranged from 33 to 71 minutes with a mean and median of 49 minutes. Seven interviews were conducted by phone or Skype, and one interview was conducted in-person. Interviews were recorded and transcribed.

Table 1.

Respondent Characteristics.

Respondent	Place of Employment	Years of Experience	Gender
1	Academia	25–29	Female
2	Evaluation firm	25–29	Female
3	Foundation	5–9	Male
4	Foundation	30+	Female
5	Government	10–14	Male
6	Government	30+	Male
7	Independent evaluation consultant	25–29	Female
8	Independent evaluation consultant	30+	Female

Data Analysis

Transcripts were analyzed to identify patterns of values commitments respondents deemed important or desirable. This interpretive analytic process reflected principles of grounded theory (Corbin & Strauss, 2015), as we identified segments of text that communicated values, labeled each segment with a code that captured the essence of the value it communicated, and then grouped codes into larger categories and clusters. The iterative process included (a) coding and recoding data into emerging or existing categories, while noting revisions in the codes and (b) reviewing the categories themselves for consistency and coherence.

Analysis proceeded in six stages. First, one of the interviewers closely read the transcript to inductively identify values and how they were enacted through core evaluation commonplaces: purpose and audience, key questions, evaluative criteria and standards, design and methods, communication and reporting, and social relational aspects of the evaluator's role (Table 2). The researcher marked and coded segments of text that pointed to values. The full research team then reflected together on each excerpt, asking, “What value(s) does this point to?”

Table 2.

Key Evaluation Commonplaces.

Commonplace	Description
Evaluation purposes	The evaluation purpose frames its primary ambitions, intended products, and intended uses. Values shape which purposes and agendas are pursued.
Evaluation audiences	The audience of an evaluation is the stakeholder group(s) for whom the evaluation is primarily intended. That is, evaluation audiences are those whose interest and needs for information are addressed by the evaluation. Values inform decisions about whose interests are addressed.
Evaluation questions	Key evaluation questions are the primary inquiry questions in an evaluation. These questions reflect the purpose of the evaluation and the concerns and interests of identified evaluation audiences. Values guide decisions about the key questions that are addressed.
Criteria for judging program quality	Criteria used for judging program quality are the heart of the evaluative enterprise; they distinguish evaluation from other forms of inquiry. Criteria represent values about what constitutes a “good” program.
Evaluation designs and methods	Designs and methods are chosen to provide useful and meaningful information to address the evaluation questions. Values shape decisions about which evaluation design and methods are selected.
Social relational aspects of the evaluator's role	Social relationships help to constitute the information and knowledge generated in an evaluation. Values guide evaluators’ approach to relationships with stakeholders.
Evaluation communication and reporting	Communication and reporting are essential for all evaluation activities. Values guide decisions about what is communicated, how, when, and with whom.

Source: Adapted from Greene (2012) and Greene et al. (2011).

Second, we looked for themes and commonalities across interviews. We searched for patterns in subsets of two or three interviews and prepared summary memos (Miles et al., 2014). We then integrated this analysis across the dataset. At each step, the full team reviewed and discussed our memos. This stage of analysis involved extensive discussion about the codes each researcher had used to label values. Through this process, we generated, defined, and refined a set of preliminary codes (Miles et al., 2014). Initially, we often used labels that reflected respondents’ terminology; for example, first-level codes included Rigor and Controlling for bias. Second-level codes represented higher levels of abstraction (Gioia et al., 2013; Saldaña, 2016) and names that more broadly captured the values expressed. For example, Systematic high-quality empiricism represents the category to which Rigor and Controlling for bias belong.

Third, two researchers coded each transcript using the preliminary coding framework. The full team discussed and further refined our codes and definitions. For example, we recognized some excerpts coded as Inclusion focused on including multiple perspectives and stakeholders, while others focused on seeking out the voices of those least represented or placed most at risk in a context. Thus, we developed two codes to delineate these values: Inclusion and Equity and justice. Four researchers (members of the interview analysis team) then conducted two additional rounds of coding.

Fourth, the interview analysis team compiled the excerpts associated with each code and drafted a summary of what the code represented. We identified and discussed connections between codes while aiming to assign one or, at most, two codes to each segment.

Fifth, two researchers organized the codes into clusters, which the full interview analysis team reviewed and revised (Table 3). We worked through several iterations of the cluster structure, seeking to reflect both the substance and frequency of the codes, and named each cluster to represent the overarching theme of the values it contained. For example, an earlier version included a cluster labeled Values related to relationships and context, which included Communication, Inclusion, Collaboration, and Attention to context. As we discussed and refined the cluster structure, we determined that Attention to context was distinct from the other values and divided the cluster into two: Values related to relationships—relabeled Values related to engaging with stakeholders to better represent stakeholders’ participation in the evaluation process—and Values related to attending to program and evaluation context. This process also led us to revise codes. For example, we reviewed the meaning of the Educating stakeholders code along with a code labeled Knowledge sharing and identified conceptual overlap. We combined them, retaining the name Educating stakeholders. Additionally, we reviewed data excerpts in which respondents discussed the importance of reflective practice, which we originally coded as a value. We decided these excerpts revealed how respondents navigated values, removed Reflective practice as a coded value, and framed these findings as an overarching theme.

Table 3.

Values Evident in Respondents’ Evaluation Practice.

Cluster (Number of Respondents)	Value (Number of Respondents)	Definition	Examples from the Interview Data
Values related to evaluation process and/or results (8)	Systematic high-quality empiricism (8)	Thorough, methodical, data-based inquiry. Involves the timeliness of the study, strong evaluation design and methods, defensible implementation of the design and methods, and clear evidence that the findings reported are directly connected to the data and results of the analysis.	“I believe strongly in rigor. I try to hound it into my students constantly because they are not the least bit rigorous. So, it is not a touchy-feely type of a thing. I really do stringent and rigorous evaluation. That is very important”
	Usefulness (7)	Evaluation process and results are valuable and beneficial for stakeholders in meeting their needs. Includes attention to instrumental, conceptual, and/or process use.	“For me, it is important to be an evaluator that can be beneficial to whomever I work for”
	Credibility (6)	Being perceived as competent, trustworthy, and fair. Refers to specific evaluation studies, individual evaluators, and the endeavor of evaluation generally. Perceptions of these distinct aspects of credibility are intertwined.	“Credibility is extremely important. If we don't have credibility, we have nothing going for us. It takes a long time to establish it and can take one mistake to ruin it”
	Educating stakeholders (6)	Enabling and supporting stakeholders to gain meaningful knowledge about and insights into the evaluation and the program being evaluated.	“Michael Quinn Patton talks about being an educator. That is something I put into my practice with all my clients”
Values related to engaging with stakeholders (8)	Communication (7)	Respectful, two-way discussion, information-sharing, listening, and negotiation. Contributes to and relies on healthy, robust relationships with multiple stakeholders.	“The big distinction is that with the [stakeholder] groups where [the evaluation] has been going great, it's about communication and it's about respect and listening to each other and getting to a place where you both agree on. Where I have had bad situations, it has just been the opposite: no matter what you say, it's going to be wrong”
	Inclusion (7)	Deliberately seeking out the voices and perspectives of all stakeholder groups and incorporating those voices into the evaluation process and results.	“[I value] getting all the different viewpoints and perspectives heard…it is a value that we can design an evaluation where everyone has a voice”
	Collaboration (5)	Making key decisions in an evaluation jointly with stakeholders. Requires and fosters respectful and inclusive relationships.	“I think the stakeholder collaboration was the most important [thing I learned]…learning to work with various stakeholders. Now I have used a collaborative approach ever since”
	Transparency (3)	Evaluation processes, decisions, and results are clear, well-documented, and available to all stakeholders.	“I was presented with boxes of documents [for the meta-evaluation], and many weren't evaluations. [My] first challenge [was]…how do I sort through them in a way that is transparent and acceptable?”
Values related to social betterment (5)	Equity and justice (5)	Promoting fairness, addressing power imbalances, and disparities. Serving individuals and communities that have been historically marginalized.	“Equity is actually a value that I try to advance in my evaluation practice, with a particular attention to racial equity. That is from where I grew up as well as where I work”
	Making things better in the world (3)	Making a positive, consequential impact. Contributing to the constructive or positive difference the program being evaluated makes in the community and/or broader society.	“For me it's about doing something good, or helpful, or useful […] I want to work with programs that are trying to make a difference. I want to help them make a difference. That is an important part of my evaluations practice”
Values related to attending to program and evaluation context (5)	Attention to context (5)	Attending to the setting and social, historical, and political background of the program being evaluated. Includes the specific locations of the program being implemented, the political motivations for or debates about the program and/or the evaluation, and the timeframe of the evaluation.	I feel like I’m very responsive. The aspects [of evaluation practice] that are important to me are the ones that make the most sense in a context. I would describe myself as…that would be what's important to me. What makes the most sense in this context?
Values related to economics of social programs and evaluations (3)	Return on investment (3)	Supporting accountability to funders' and program leaders’ operational and strategic decisions. Includes comparing program results to its costs.	“What you need to think about is the [resources] that you spent on this program, are you actually going to get benefit out of it? Will you get future benefit out of it as an investment? If not, don't keep investing”

Sixth, two researchers retraced our audit trail of transcripts, memos, codebooks, diagrams, and email communication. Revisiting every data excerpt and discussing the decisions made at each step of the research process led us to recode several data segments and rename one cluster.

Influence of Values in Data Analysis

In this section, we seek to make explicit how our values influenced analysis. First, our belief in the importance of attending to how values shape each step of the evaluation process led us to use the evaluation commonplaces to focus our first round of analysis. Second, because we framed this study within the democratic tradition, we sought to evoke and identify values within the unique contexts of respondents’ practices. Therefore, we analyzed the transcripts for evidence of explicit and implicit values and developed codes and labels through reflective discussions. Third, as in all research, our values directed our attention toward certain ideas and themes, requiring us to continually check that our inferences could be traced back to the data.

Data Quality and Trustworthiness

We took steps throughout the research process to maximize data quality and the trustworthiness of our findings (Lincoln & Guba, 1985). The team was led by a senior researcher with expertise in the topic and methodology (Greene) and each member had training in evaluation and interpretive research. The interview guide was collaboratively designed, reviewed, and tested. Interviews focused on topics about which respondents had in-depth knowledge: their own evaluation practice. We established rapport during interviews and included member checks to verify our understanding. The team engaged in peer debriefing to interrogate the quality of data collection and interpretations. We maintained a detailed audit trail that we referenced over the course of this research. Through these multiple and varied processes, we sought to establish findings that are credible, dependable, and confirmable.

Findings

In this section we describe findings about (a) key values held by respondents, (b) relationships among those values, (c) how values showed up in respondents’ evaluation practices, and (d) how respondents navigated values and value conflicts.

Findings: Key Values

Analysis of interviews with eight seasoned evaluators inductively revealed 12 key values, which we organized into five clusters (Figure 1), based both on substance and frequency. Each cluster represents a conceptual grouping of values (Table 3) and is positioned in the diagram to reflect its centrality or prevalence in our sample. The five clusters are: (a) Values related to evaluation process and/or results, (b) Values related to engaging with stakeholders, (c) Values related to social betterment, (d) Values related to attending to program and evaluation context, and (e) Values related to economics of social programs and evaluations. We observed connections among values within different clusters, which informed our decision to represent the values clusters as a set of circles, some concentric and some overlapping.

Figure 1.

Five clusters of values evident in respondents’ evaluation practice.

Values Related to Evaluation Process and/or Results

Each respondent discussed one or more values related to the character of the evaluation process and/or results. These included four primary beliefs about the importance of Systematic, high-quality empiricism; Usefulness; Credibility; and Educating stakeholders. These values represent the core of what most evaluators do in their practice.

Systematic, High-Quality Empiricism

Every respondent described the importance of thorough, methodical, data-based inquiry as the basis of their evaluation practice and spoke about the desirability of rigor, even in challenging contexts. Examples included using data collection methods that are extensive or deep in nature, achieving reliability in data collection and coding, and ensuring informed consent. In addition, respondents noted that data should be collected in the most appropriate language for program participants, and evaluators should use instruments that are valid for the purpose and context. Respondents stated that high-quality empiricism requires sufficient time and funding and emphasized that evaluators should only do work they are well equipped to do. Experience and training are important for high-quality work and seeking out additional training may be necessary.

Usefulness

Seven respondents expressed the belief that the evaluation process and results should be valuable for meeting stakeholders’ needs. One respondent explained, “I honestly don't like to do evaluations where it is not clear to me who is going to use the information.” Another asked, “If it's not going to be used, why do it?” In one instance, a respondent spoke of not necessarily being supportive of the evaluand but still considering the work to be worthwhile if it was useful to stakeholders. Some respondents also described a desire to conduct evaluations that are useful to others beyond the immediate program context.

Credibility

Six respondents discussed the importance of being perceived as credible by stakeholders, which included being seen as competent, trustworthy, fair, and open-minded. Respondents said they established credibility by saying what needs to be said even when it is unpopular or risky, being honest and transparent about study weakness and difficulties, using well-established methods that are acceptable to key stakeholders, and recognizing the political context of evaluations. Some respondents discussed credibility in relation to program beneficiaries, noting that the ability to meaningfully engage constituents who have been systemically marginalized and placed at risk or are difficult to reach is an indicator that the evaluator is credible in the eyes of that population. One respondent discussed how working with trusted community partners can enhance credibility.

Educating Stakeholders

Six respondents described their beliefs in the importance of educating stakeholders—primarily program funders, leaders, and staff—about evaluation results, often to help the program to make improvements and continue operating. They accomplished this by providing clear, nontechnical presentations of data and findings. Some respondents discussed the importance of educating a broader range of stakeholders—such as those working with related programs or issues—about the positive aspects of the evaluand. Respondents also discussed the importance of educating stakeholders about evaluation purpose and design. For example:

When I work with my clients, I always set aside time…whether paid or not…to really talk about evaluation…what it is and what it can do and what it cannot do. […] it helps the client get a better product, but it's also about educating the client so the next time they hire an evaluator and the person is trying to push an approach on them, they will know there are other ways to do this.

Values Related to Engaging With Stakeholders

Each interview participant discussed one or more values related to engaging with stakeholders, which included beliefs about the importance of Communication, Inclusion, Collaboration, and Transparency. These values reflect the nature of evaluation as a stakeholder-engaged form of inquiry.

Communication

Seven respondents addressed the importance of strong communication skills and processes, including the ability to communicate and negotiate respectfully with diverse stakeholders. Respondents indicated that strong communication contributes to and relies on healthy, robust relationships. One respondent prioritized in-person conversations to share evaluation findings. Another described a commitment to open, honest communication with clients, including sharing issues that arise—“I don't try to keep any secrets from the clients”—and emphasized the importance of communication in building relationships and laying the groundwork for delivering evaluation findings. Clear communication was also identified as important for addressing conflicts.

Inclusion

Seven respondents spoke about the importance of inclusion, which involved deliberately seeking out the perspectives of all stakeholder groups and incorporating those into the evaluation. Respondents sought to include policymakers, program administrators, staff, and beneficiaries: “I try to [include] as broad a swath of stakeholders as I can. I try to find contrarians, people who will be on the opposite sides of the issue.” Respondents spoke about evaluators’ responsibility to remove barriers to participation and adopt appropriate communication and language practices. For example, one respondent discussed plans for sharing data with stakeholders:

We should make the conversation as easy as possible for the group we want to hear from. It's as simple as that. Thinking about what is going to make it easy for people to share their feelings and priorities with us.

Some respondents also described specific methodologies they employed to support inclusive participation, such as concept mapping and feminist evaluation approaches.

Collaboration

Five respondents discussed their belief that the process of evaluation should be implemented collaboratively with multiple stakeholders and, as appropriate, other evaluators. Respondents’ aim for collaboration was bringing stakeholders’ voices, perspectives, and priorities into evaluation design and implementation. For example, one respondent described a collaborative process for developing logic models. Another respondent developed a data collection instrument for use by case managers in community-based program sites. After receiving negative feedback from a case manager: “I created [a new] form based on her and the other case managers’ feedback and that community loved it.”

A third respondent described collaboration with members of a historically marginalized population that a program served: “I only agreed to do [the evaluation] if I could hire members of that community to work as co-study investigators […] They made every decision with me.” Respondents created the conditions for collaborative decision making by providing facilitation, scaffolding, or evaluation capacity building. They emphasized that collaboration requires a common goal—and it cannot be assumed that common goals exist, even when official goals have been established.

Transparency

Three respondents discussed the importance of facilitating open access to information about evaluation design, process, and results, believing it is important that information be clear, well-documented, and available to all stakeholders. Respondents described being transparent about evaluation methods and why those methods were chosen, being clear about what a specific study can and cannot do, and ensuring that decision making and reporting were transparent. A respondent working in the government sector discussed the intersection of transparency and laws that govern public access to information: “We have open records in [this state]. I have nothing to hide. And my goal is to make sure everyone can get the information they need.”

Values Related to Social Betterment

Five of the eight respondents expressed values related to social betterment, which included values related to Equity and justice and Making things better in the world. These values address the broader purpose of evaluation in society.

Equity and Justice

Five respondents discussed their beliefs that evaluation should promote equity, address power imbalances, and serve individuals and communities that have been historically marginalized. Respondents expressed this value by focusing their practice on programs that sought to advance equity, including those for migrant families and families living in poverty, and programs that provide access to education, housing, and clean drinking water. Respondents also described a focus on equity during the evaluation process, such as assessing equity of opportunities and resources. One respondent described work with a population that had been criminalized, emphasizing the importance of not causing harm: “We did not want to draw attention to the fact that we were working regularly with people that the mayor was trying to put into jail.” In advancing this value, respondents stated that they work closely with stakeholders, especially program participants, to understand their lived realities.

Making Things Better in the World

Three respondents expressed a commitment to making a difference in the world—at the level of the evaluation client, program participants, or society—as a driving force in their practice. One respondent explained, “What I value is doing something that is going to make a difference, whether that is for the evaluators I’m working with or for my client or for their beneficiaries.” These values were evident in the evaluation approaches used, such as democratic or feminist approaches, and in the programs they evaluated, which focused on addressing community problems or helping to improve conditions. One respondent described their enthusiasm when beginning an evaluation for an organization that sought to reduce hunger and poverty and protect the natural environment: “I thought, ‘We ARE going to save the world!’” Another respondent explained: “I want to work with programs that are trying to make a difference. I want to help them make a difference.”

Values Related to Attending to Program and Evaluation Context

Five of the eight respondents discussed the importance of Attention to context. This included a belief that understanding context is important to understanding the evaluand. Respondents indicated that attention to context addresses the relationship between the program, evaluation, and larger geographic, social, political, and historical setting; enables them to align the evaluation with program timelines and stakeholder needs; and helps respondents attend to and negotiate the political context. One respondent explained, “We recognized the importance of timing of releasing our report. We recognized the political context of this project. And we did everything possible, everything humanly possible, to understand the stakeholders.” Respondents indicated that rich descriptions of context in reporting can illuminate the many facets of an evaluand, positive and negative. For example, one respondent used portraiture methodology in an evaluation of a program for migrant workers to help audiences understand “what was going on in the fields…where the migrant workers came from and the issues that they were encountering.”

Values Related to Economics of Social Programs and Evaluations

Three of the eight respondents discussed values related to Return on investment of program resources. This value addresses the role evaluation plays in supporting accountability to and by program funders. Respondents reported situations in which they purposefully engaged their clients in conversation about a program's predicted benefits as compared to its costs. One evaluator described an evaluation that investigated questions about program and organizational efficiency: “…we were looking at […] how the different sites were running and using their funding. It was a big scope and primarily qualitative…the extent to which the program worked efficiently and how they used their funding.” Another respondent addressed the role evaluation plays in assessing accountability of grantees to a foundation that provided funding and accountability of the foundation staff to their board.

Findings: Relationships Among Values

As we grouped the identified values, we noted relationships across and within clusters. The two innermost clusters, Values related to evaluation process and/or results and Values related to engaging with stakeholders contain the values most frequently present in our data; we consider these to be reflective of the core of what most evaluators do in their practices. However, the cluster of Values related to engaging with stakeholders is distinguished by a conceptual focus on relationships. The three outermost clusters—Values related to social betterment, Values related to attending to program and evaluation context, and Values related to economics of social programs and evaluations—include values identified less frequently (hence the choice to not include them as concentric to the others), yet still related to the other clusters (thus the choice to present them as overlapping).

The Educating stakeholders value reveals a key cross-cluster relationship. This value could have been placed in the Values related to engaging with stakeholders cluster, as stakeholder education involves communication and collaboration. However, many respondents viewed their roles as highly educative and, given the centrality of this value to their practice, we felt that Educating stakeholders fit best in the innermost cluster, Values related to evaluation process and/or results. Similarly, we included Credibility in that innermost cluster, as many respondents expressed that credibility was established via thorough, methodical work. However, respondents noted that Credibility was also related to Transparency and Communication. We also identified relationships between Attention to context, Usefulness, and Equity and justice. Respondents explained that understanding context was necessary to ensuring that an evaluation was beneficial to stakeholders. Similarly, contextual understanding was essential to promoting fairness and addressing power imbalances.

Findings: How Values Showed Up

Respondents’ values informed each of the evaluation commonplaces. As one respondent explained, values are “communicated through the way I do my work.” For example, values of Inclusion prompted respondents to incorporate a broad range of stakeholder voices and attend to historically marginalized communities. A commitment to Collaboration guided one respondent to help stakeholders determine the evaluation questions they wanted to ask and another respondent to redesign a data collection instrument to align with stakeholder priorities. Values related to Systematic, high-quality empiricism motivated respondents to employ multiple methods in their studies and use validated instruments. Respondents’ values around Communication, Collaboration, and Inclusion led them to develop reciprocal, interactive relationships with stakeholders, rather than enacting a stance of detachment and authority.

Values also shaped how respondents positioned their work. For example, respondents who valued stakeholder education enacted evaluation as an educative endeavor, rather than inquiry focused exclusively on accountability. Respondents’ values around Usefulness motivated them to meet stakeholder needs. Among those with the freedom to choose the programs they evaluated, respondents’ values related to Equity and justice and Making things better in the world led respondents to evaluate programs that sought to address critical social issues.

Depending on their workplaces, some respondents had the freedom to choose evaluands that aligned with their values, while others were assigned to evaluations that may or may not match their values. One respondent explained:

I’ve spent almost all of my career doing evaluations that have been identified [by supervisors] and often the constraints on them are external. […] I have ethics and values that I think are almost always communicated through the way I do my work. Having been in a lot of practical situations, I don't think about the kinds of values I’m advancing in those terms. I think about what is good practice. Advancing values does not come first. It's about how the values I hold as a human being manifest themselves in the evaluation work I do.

Some respondents felt their values were implicit in their work. For example, when asked what values they advanced in their work, one respondent noted: “I wouldn't even say ‘advance,’ that's just who I am. That is just what I bring to the table.”

Importantly, only two of the eight respondents explicitly discussed their values with evaluation stakeholders. Professional context influenced, but did not dictate, the extent to which values were explicitly discussed and advanced. One respondent in the government sector stressed the importance of setting aside personal values and conducting all studies that were requested; however, another government evaluator had declined requests due to a clash of values commitments. A third respondent expressed frustration that clients did not inquire about evaluators’ values. When the respondent was asked if they tried to make values explicit, they replied:

Maybe I wouldn't say the word “value” [to a client] but maybe try to approach it a different way […] A lot of it really does come down to values. I just think it is so strange that no one ever really asks me what my values are.

Findings: Navigating Values

We observed a high level of self-awareness and understanding of respondents’ own values and how those values affect evaluation practice. Many respondents discussed the importance of reflection and reflective practice. Some respondents indicated that understanding the origins of and influences on their values deepened self-understanding. These included family and upbringing, mentors, colleagues, reading evaluation literature, and cumulative evaluation experiences. As one respondent said, “it's critical to understand your own values and to understand where those values come from […] so you can choose your behavior. Values don't determine your behavior, but they certainly determine how your behaviors are influenced.” Another respondent noted that this understanding can foster “profound respect for the perceptions and opinions of values of other people.”

Respondents shared examples of values conflicts they had navigated. One respondent described a school-based evaluation in which leaders wanted to focus evaluation questions and criteria on students’ test scores to satisfy their funder. Yet the evaluator learned from school stakeholders that increased test scores was not a focus of the program. “The logic model […] talked about access to computers, opportunities to learn through different modalities, students enjoying school more, increased attendance, fewer disciplinary actions.” The respondent valued stakeholders’ perspectives and aligning the evaluation with the school context; therefore, the respondent educated leaders about the limitations of focusing only on test scores and designed a study that addressed an array of questions and criteria.

A second respondent faced a conflict when assigned an evaluation of a prison-based program focused on accountability to a government funder. The respondent valued capturing rich contextual information even though “…my boss didn't like the fact that I was describing the environment: the guards, cinderblock walls, and the expressions of the prisoners and their issues and problems…” The same evaluator discussed a different situation in which the stated evaluation purpose was for a program administrator to “determine what programs were either good or bad.” The respondent valued the inclusion of multiple stakeholders:

I would only do the evaluation if I could be inclusive and could have people within the program give their ideas, values, and opinions. And when I write [the report] it will be theirs. And it will be for them to validate whether or not that thinking is accurate. And she said, “No, I’m going to make the decisions.” So, I said, “No, I’m not doing the evaluation.”

Finally, a third respondent described a conflict in which a powerful stakeholder wanted certain findings removed from an evaluation report. “[The stakeholder] had a closed-door meeting with me and attacked the methodology of the report. He attacked the team leader on that project. I stood behind what was written in the report.” The respondent valued credibility and held firm, despite the professional risks. “Luckily the [project] leadership had [our] back. But it was a calculated risk. It was worth it if we said the thing that needed to be said.”

Discussion

Our study reveals multiple ways in which evaluators express values in their work. While scholars have argued that values shape evaluation activities (Greene, 2012; Shadish, 1994; Smith, 1980) and speculated about what that may look like (Gullickson & Hannum, 2019; LaVelle et al., 2022), the current study provides empirical evidence about the role of values in evaluation practice, making the often “fuzzy” notion of values in evaluation (House, 2015, p. xiii) more visible and concrete. Our findings complement LaVelle et al.’s (2022) report of overarching values espoused by evaluators by surfacing values expressed in specific evaluation activities that reflect both espoused and enacted values (Gullickson & Hannum, 2019).

Schwandt and Gates (2021) encourage a focus not on whether values should be part of the inquiry process, but where values have an influence. Our findings reveal how values show up in each evaluation commonplace and all stages of the evaluation process, as well as how values shape the larger positioning of evaluation and evaluators. This provides empirical confirmation that evaluation is not values-neutral (House & Howe, 1999) and begins to clarify how values operate across the evaluation process. Our study also provides empirical examples of values conflicts, including those that arise because evaluation commissioners and sponsors frame evaluation to reflect their values (Schwandt & Gates, 2021). In doing so, the study contributes to shifting the field from a conceptual and prescriptive engagement with values to empirical, descriptive understandings (LaVelle et al., 2022).

Convergence and Divergence in Values

We found considerable overlap in the values that guided this group of evaluators’ practices. This finding was somewhat surprising given that values and their enactment are understood to be highly contextual (Gullickson & Hannum, 2019), and we purposely selected evaluators from varied professional contexts. Further, we employed methods that allowed varying values to emerge. The overlap may reflect our selection of seasoned evaluators, normative aspects of evaluation practice, and shared professional values about what is desirable (Schwandt & Gates, 2021). For example, the AEA (n.d.) core values include excellence in evaluation practice and utilization of evaluation findings, which align with the values we labeled as Systematic, high-quality empiricism and Usefulness. This link between individual and shared values reflects the nature of professional values, which includes individuals’ beliefs about what is important in a professional undertaking (Ros et al., 1999) and shared ideas within a profession about desirable or ethical conduct (Chadwick, 1998; Schwandt, 1997).

Divergence within our data did, at times, reflect variations in context (Gullickson & Hannum, 2019). Differences may also reflect varying conceptualizations and evaluation models, pointing to the need for research into the alignment of evaluators’ values and their preferred evaluative approaches (LaVelle et al., 2022). However, respondents also reported influences on values beyond professional identities and experiences, such as family and upbringing. Thus, our findings suggest a reciprocal relationship between what individuals bring to the field, how they are trained and socialized into the evaluation community, and shared values within the field.

Values inventories suggest clear-cut categories of values (e.g., Schwartz et al., 2012). However, we found respondents’ values to be deeply entangled with one another. We observed connections between individual and shared values and intersections between personal and professional values, and represented these complex relationships by depicting the clusters of values with concentric and overlapping circles.

Nature of Values Engagement

Finally, our findings highlight how evaluators enact their values through complex, context-specific decisions (Gullickson & Hannum, 2019; Schwandt & Gates, 2021). Rather than a clear-cut application of fully formed values, respondents described wrestling with values conflicts and trade-offs (Schwandt & Gates, 2021). Respondents navigated these situations by choosing projects that aligned with their values, discussing issues with stakeholders, setting conditions for accepting an evaluation, and declining projects. Although engaging directly with values can bring greater clarity about what is important and where value differences lie (Gullickson & Hannum, 2019), we found that few respondents explicitly discussed their values with stakeholders. This may reflect challenges evaluators face in understanding how to explicitly engage with values (Schwandt & Gates, 2021).

Limitations and Future Directions

The limitations of this study point to areas for future research. The study included a small number of seasoned evaluators. Respondents were not selected to be representative of the broad field of evaluation, and findings are not intended to be statistically generalizable. Future studies should investigate values among expanded groups of evaluators, including novice or early career evaluators. In addition, we designed our inquiry to surface and examine values within the unique context of each individual's life and experiences and did not employ an established values framework. Additional research is needed to understand how evaluators’ values align with or diverge from frameworks such as the AEA Guiding Principles or particular values inventories. This study elicited evaluators’ reflections on their own practices and, as such, did not capture evaluations as they unfolded or from stakeholders’ perspectives. In addition, we did not examine the (mis)match between espoused and enacted values. Future study might offer a direct account of how values are engaged and advanced. We did not explore how evaluators’ values work or clash with different evaluation approaches. Further study is needed to explore potential tensions and how those could be resolved.

In the years since these data were collected, the United States, the world, and the field of evaluation have grappled with the ongoing violence of racism and white supremacy (Caldwell & Bledsoe, 2019; Hall, 2018; McBride, Casillas, & LoPiccolo, 2020); the continued rise of the “post-truth” era in which distinctions among evidence, opinion, and lies are blurred (Patton, 2018; Picciotto, 2019; Thomas & Campbell, 2021); and, relatedly, the devastating impact of the COVID-19 pandemic (Ofir & Rugg, 2021; Patton, 2021). Follow-up research is needed to understand how evaluators’ values have been shaped by recent national and global events and how evaluators express those values in their work.

Finally, this study did not address the resources evaluators need to productively engage with values. Further research is needed to develop tools and supports for identifying and examining values, discussing and educating stakeholders about values, and resolving values conflicts.

Implications for Evaluation Practice

Our findings raise implications for how evaluators might enact their professional and ethical responsibilities to identify and articulate their own values in the context of practice. First, this study can support evaluators in responding to calls to reflect on their own values (Schwandt & Gates, 2021; Yarbrough et al., 2011) by using the values and sources of values we describe as a starting point for self-reflection. Second, findings can help evaluators understand how values motivate or guide decision making and behavior in evaluation practice (Gullickson & Hannum, 2019; Schwandt & Gates, 2021). The empirical examples we present illustrate how values can show up, and even define, evaluators’ understanding and positioning of their work.

Third, our findings can also support evaluators in responding to calls to more explicitly articulate and discuss the values dimensions of evaluation planning and implementation (Gullickson & Hannum, 2019). As one respondent noted above, “A lot of it really does come down to values.” These conversations can help evaluators and stakeholders understand how their values intersect with the values that underpin the evaluand (Gullickson & Hannum, 2019) and commissioners’ values that have informed evaluation framing (Schwandt & Gates, 2021). Fourth, Gullickson and Hannum (2019) point out that making values transparent also makes values differences transparent. Our data offer examples of how evaluators might navigate instances where their own values and stakeholders’ values are at odds.

Finally, evaluator education often focuses on methods, with less attention to values (Gullickson & Hannum, 2019). Our study can serve as a resource for evaluator educators—those who teach in universities, lead professional development courses and workshops, or provide workplace mentoring and coaching—for reflective discussion to help emerging evaluators identify and articulate their values commitments.

Conclusion

This article challenges persistent misrepresentations of evaluation as an objective and value-neutral inquiry process by presenting an empirical study of the evaluation practices of eight seasoned evaluators. We surfaced and examined 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies these evaluators conducted. Values also guided the ways evaluators understood and positioned their work and the types of programs they evaluated. Evaluators can draw on our findings to critically reflect on their own values and beliefs related to evaluation practice and social programs and how those values show up in their practices. By surfacing, examining, and openly articulating their own values, evaluators can take an important step in enacting their professional and ethical responsibilities as evaluation practitioners.

Footnotes

Acknowledgments

We thank Candice Solomon-Strutz, Dominic Combs, and Benjamin Helton for their important contributions to the study reported in this article, including valuable contributions to developing the interview guide, conducting interviews, and the initial analysis of interview transcripts.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rebecca M. Teasdale

Maria Isabel Ramírez Garzón

Notes

References

American Evaluation Association. (2018). Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles

American Evaluation Association. (n.d.). Values of the American Evaluation Association. https://www.eval.org/About/About-AEA

Benson

Cribb

Barber

. (2009). Understanding pharmacists’ values: A qualitative study of ideals and dilemmas in UK pharmacy practice. Social Science and Medicine, 68, 2223–2230. https://doi.org/10.1016/j.socscimed.2009.03.012

Caldwell

L. D.

Bledsoe

K. L

. (2019). Can social justice live in a house of structural racism? A question for the field of evaluation. American Journal of Evaluation, 40(1), 6–18. https://doi.org/10.1177/1098214018815772

Campbell

D. T

. (1969). Reforms as experiments. American Psychologist, 24, 409–429. https://doi.org/10.1037/h0027982

Chadwick

(1998). Professional ethics. In Crane

(Ed.), Routledge encyclopedia of philosophy (pp. 733–737). Abingdon, UK: Routledge.

Corbin

Strauss

(2015). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage.

Coryn

C. L. S.

Wilson

L. N.

Westine

C. D.

Hobson

K. A.

Ozeki

Fiekowsky

E. L.

,. . . Schröter

D. C.

(2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347. https://doi.org/10.1177/1098214016688556

Davidson

E. J.

(2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.

10.

Fournier

D. M

. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 1995(68), 15–32. https://doi.org/10.1002/ev.1017

11.

Fournier

D. M.

(2005). Evaluation. In Mathison

(Ed.), Encyclopedia of evaluation (pp. 139–140). Thousand Oaks, CA: Sage.

12.

Gioia

D. A.

Corley

K. G.

Hamilton

A. L

. (2013). Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational Research Methods, 16(1), 15–31. https://doi.org/10.1177/1094428112452151

13.

Greene

J. C

. (1997). Evaluation as advocacy. Evaluation Practice, 18(1), 25–35. https://doi.org/10.1177/109821409701800103

14.

Greene

J. C

. (2002). With a splash of soda, please: Towards active engagement with difference. Evaluation, 8(2), 259–266. https://doi.org/10.1177/1358902002008002522

15.

Greene

J. C.

(2006). Evaluation, democracy, and social change. In Shaw

Greene

J. C.

Mark

M. M.

(Eds.), The Sage Handbook of Evaluation (pp. 118–140). London: Sage.

16.

Greene

J. C.

(2012). Values-engaged evaluations. In Segone

(Ed.), Evaluation for equitable development results (pp. 192–206). New York: UNICEF.

17.

Greene

J. C.

Boyce

A. S.

Ahn

(2011). A values-engaged, educative approach for evaluating education programs: A guidebook for practice. Urbana, IL: University of Illinois at Urbana-Champaign.

18.

Greene

J. C.

Millett

R. A.

Hopson

R. K.

(2004). Evaluation as a democratizing practice. In Braverman

M. T.

Constantine

N. A.

Slater

J. K.

(Eds.), Foundations and evaluation: Contexts and practices for effective philanthropy (pp. 96–118). San Francisco: Jossey-Bass.

19.

Gullickson

A. M.

Hannum

K. M

. (2019). Making values explicit in evaluation practice. Evaluation Journal of Australasia, 19(4), 162–178. https://doi.org/10.1177/1035719X19893892

20.

Hall

M. E

. (2018). Evaluation’s race problem in the U.S.: A call to action for the profession and the American Journal of Evaluation. American Journal of Evaluation, 39(4), 569–583. https://doi.org/10.1177/1098214018792624

21.

Harman

Azzam

. (2018). Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards. Evaluation and Program Planning, 71, 68–82. https://doi.org/10.1016/j.evalprogplan.2018.08.004

22.

Hassell

Gullickson

A. M.

Boyce

A. S.

Hannum

(2020). Editorial: What do we mean when we talk about values? Evaluation Journal of Australasia, 20(2), 63–67. https://doi.org/10.1177/1035719X20931250

23.

House

E. R.

(2015). Evaluating: Values, biases, and practical wisdom. Charlotte, NC: Information Age.

24.

House

E. R.

Howe

K. R.

(1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.

25.

Howe

K. R.

(2003). Closing methodological divides: Toward democratic educational research. Boston: Kluwer.

26.

Jiménez-López

F. R.

Roales-Nieto

J. G.

Vallejo Seco

Preciado

. (2016). Values in nursing students and professionals: An exploratory comparative study. Nursing Ethics, 23(1), 79–91. https://doi.org/10.1177/0969733014557135

27.

Kushner

(2000). Personalizing evaluation. London: Sage.

28.

LaVelle

J. M.

Stephenson

C. L.

Donaldson

S. I.

Hackett

J. D

. (2022). Findings from an empirical exploration of evaluators’ values. American Journal of Evaluation. Advanced online publication: https://doi.org/10.1177/10982140211046537

29.

Lee

Y. K.

Low

W. Y.

C. J

. (2013). Exploring patient values in medical decision making: A qualitative study. PLoS One, 8(11), e80051. https://doi.org/10.1371/journal.pone.0080051

30.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Newbury Park, CA: Sage.

31.

McBride

Casillas

LoPiccolo

. (2020). Inciting social change through evaluation. New Directions for Evaluation, 166, 119–127. https://doi.org/10.1002/ev.20405

32.

Miles

M. B.

Huberman

A. M.

Saldaña

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.

33.

Ofir

Rugg

. (2021). American Journal of Evaluation section on international developments in evaluation: Transforming evaluation for times of global transformation. American Journal of Evaluation, 42(1), 47–52. https://doi.org/10.1177/1098214020979070

34.

Ozeki

Coryn

C. L. S.

Schröter

D. C

. (2019). Evaluation logic in practice: Findings from two empirical investigations of American Evaluation Association members. Evaluation and Program Planning, 76, 101681. https://doi.org/10.1016/j.evalprogplan.2019.101681

35.

Patton

M. Q.

(2015). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

36.

Patton

M. Q

. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200. https://doi.org/10.1177/1098214018763121

37.

Patton

M. Q

. (2021). Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation, 42(1), 53–89. https://doi.org/10.1177/1098214020933689

38.

Picciotto

. (2019). Is evaluation obsolete in a post-truth world? Evaluation and Program Planning, 73, 88–96. https://doi.org/10.1016/j.evalprogplan.2018.12.006

39.

Putnam

(2002). The collapse of the fact/value dichotomy. Cambridge, MA: Harvard University Press.

40.

Ravari

Bazargan-Hejazi

Ebadi

Mirzaei

Oshvandi

. (2012). Work values and job satisfaction: A qualitative study of Iranian nurses. Nursing Ethics, 20(4), 448–458. https://doi.org/10.1177/0969733012458606

41.

Roorda

Gullickson

A. M

. (2019). Developing evaluation criteria using an ethical lens. Evaluation Journal of Australasia, 19(4), 179–194. https://doi.org/10.1177/1035719X19891991

42.

Ros

Schwartz

S. H.

Surkiss

. (1999). Basic individual values, work values, and the meaning of work. Applied Psychology: An International Review, 48(1), 49–71. https://doi.org/10.1111/j.1464-0597.1999.tb00048.x

43.

Sagiv

Roccas

Cieciuch

Schwartz

S. H

. (2017). Personal values in human life. Nature Human Behavior, 1, 630–639. https://doi.org/10.1038/s41562-017-0185-3

44.

Saldaña

(2016). The coding manual for qualitative researchers. London, UK: Sage.

45.

Schwandt

T. A

. (1997). The landscape of values in evaluation: Charter terrain and unexplored territory. New Directions for Evaluation, 76, 25–39. https://doi.org/10.1002/ev.1085

46.

Schwandt

T. A.

(2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Palo Alto, CA: Stanford University Press.

47.

Schwandt

T. A.

Gates

E. F.

(2021). Evaluating and valuing in social research. New York, NY: Guilford.

48.

Schwartz

S. H

. (1999). A theory of cultural values and some implications for work. Applied Psychology: An International Review, 48(1), 23–47. https://doi.org/10.1111/j.1464-0597.1999.tb00047.x

49.

Schwartz

S. H.

Caprara

G. V.

Vecchione

. (2010). Basic personal values, core political values, and voting: A longitudinal analysis. Political Psychology, 31(3), 421–452. https://doi.org/10.1111/j.1467-9221.2010.00764.x

50.

Schwartz

S. H.

Cieciuch

Vecchione

Davidov

Fischer

Beierlein

,. . . Konty

(2012). Refining the theory of basic individual values. Journal of Personality and Social Psychology, 103(4), 663–688. https://doi.org/10.1037/a0029393

51.

Scriven

. (2007, June 6–7). The logic of evaluation. Paper presented at the Ontario Society for the Study of Argumentation, Windsor, Ontario.

52.

Scriven

(1991). Evaluation thesaurus (4th ed.). Newbury Park CA: Sage.

53.

Shadish

W. R

. (1994). Need-based evaluation theory: What do you need to know to do good evaluation? Evaluation Practice, 15(3), 347–358. https://doi.org/10.1177/109821409401500315

54.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

55.

Smith

N. L

. (1980). Sources of values influencing educational evaluation. Studies in Educational Evaluation, 6(2), 101–118. https://doi.org/10.1016/0191-491X(80)90013-9

56.

Teasdale

R. M

. (2021). Evaluative criteria: An integrated model of domains and sources. American Journal of Evaluation, 42(3), 354–376. https://doi.org/10.1177/1098214020955226

57.

Teasdale

R. M

. (2022). How do you define success? Evaluative criteria for informal STEM education. Visitor Studies 25(2), 163–184. https://doi.org/10.1080/10645578.2022.2056397

58.

Thomas

V. G.

Campbell

P. B.

(2021). Evaluation in today’s world: Respecting diversity, improving quality, and promoting usability. Thousand Oaks, CA: Sage.

59.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.