Abstract
The concept of power is a complex and often intangible aspect of the evaluation process that is frequently a focal topic among the conceptual evaluation literature concerning collaborative or culturally responsive evaluation (CRE) contexts. Unfortunately, there remains a significant theory to practice gap as power is often rarely addressed or specifically identified in empirical practice. In the current article, the authors provide a comprehensive review of the empirical literature on CREs, with an emphasis on the manifestation of power in practice. The analysis of the literature selected for review leads to the development of a conceptual framework, which identifies intersecting power dynamics across the relational, political, discursive, and historical/temporal dimensions. The authors conclude with a research synthesis and implications for practice, including a discussion of practical ways that evaluators can begin with considerations of power rather than reflecting on the concept retrospectively.
Power is everywhere; not because it embraces everything, but because it comes from everywhere. (Foucault, 1978, p. 3) Power relations are relations of autonomy and dependence, but even the most autonomous agent is in some degree dependent, and the most dependent actor or party in a relationship retains some autonomy. (Giddens, 1979, p. 93)
In evaluation, the focus on power often arises in collaborative or culturally responsive evaluation (CRE) contexts, where attention is given to who participates, under which conditions they participate, to what extent they participate, and whose knowledge frames the evaluation and is considered of most worth. Specifically, culturally responsive evaluators seek to engage participants and stakeholders in a meaningful and culturally sensitive way, attending to the cultural context of the evaluation (Conner, 2004). Ironically, despite the intention to engage participants in developing, framing, and conducting the evaluation, power continues to influence and shape the parameters of practice in various technical, conceptual, historical, and relational ways (Chouinard & Cousins, 2012). Power and culture are inextricably linked together (Rosaldo, 1989; Seidman, 2004), and despite the use of culturally responsive and collaborative approaches, issues of power, inequity, and privilege persist throughout the evaluation process (Chouinard, 2014). For some, the emphasis is on identifying techniques and strategies with a focus on the redistribution of unequal power dynamics (e.g., Mason & Boutilier, 1996; Matthews, 2001; Van der Riet & Boettiger, 2009), whereas for others, power is not a commodity that can be possessed by some and not others, with an ability to be redistributed at will (Chouinard & Hopson, 2016; Gallagher, 2008).
This latter view problematizes the idea that power can simply be given or taken away and views power as a dynamic and complex set of relationships and actions that is constantly shifting and changing shape as the evaluation unfolds. In our view, this conceptualization of power is more consistent with the complex and multidimensional nature of evaluations, as power dynamics are complex and can never truly be eliminated or fully redistributed. As a relational and productive concept (Foucault, 1979), power informs and shapes both the relationships between evaluators and stakeholders, as well as the broader sociopolitical level, what Gaventa and Cornwall (2006) refer to respectively as the “micropolitical relationships of power” and the “macro-political relationships of power” (p. 73). From this perspective, power is not conceptualized as something that is possessed by any one individual, state or higher authority, but as something that circulates and structures knowledge and its social constructions (Foucault, 1979). Knowledge generation is thus not a neutral activity, as it is reflective of the values and relations of power and privilege within which it takes place (Greene, 2002). While evaluators who work in culturally responsive program contexts acknowledge the challenges of conducting evaluations and building relationships amid unequal power and status, our research findings suggest that this fact remains underreported in both the literature and empirical findings (Chouinard, 2014; Chouinard & Cousins, 2009), making it difficult to understand how power manifests in practice and how it ultimately impacts both the process and outcome of the evaluation itself.
In this article, we turn to a review of the empirical literature related to CRE in a North American context to explore the ways in which power is conceptualized and tackled in these settings. In total, we located 18 empirical studies written between 2000 and 2016, all of which are reflective narratives based on experiences with CRE practice. In our reading and analysis of the articles, we focused on understanding how (and in which ways) power is manifest in practice, both conceptually and from a methodological perspective. Our goal in this article is 2-fold, to provide a thorough analysis of power as it is described in the literature and to contribute to the development of a conceptual framework on the location of power in CRE contexts. Our hope is that this framework will assist future evaluators in navigating the complexity and multidimensionality of power across culturally responsive settings. We begin by describing our conceptual framework, how it informed our work, and how our reading of the studies informed and shaped the final product. We then provide a description of our sample and our methods used for analysis. In the main section of our article, we provide an analysis of the literature through the prism of our conceptual model, based on four dimensions of power (relational, political, discursive, and historical/temporal). We conclude with a research synthesis and implications for practice. Our hope is that through a focus on power in CRE contexts, we also shed light on the ways in which evaluators address what are, without doubt, some of the more intransigent challenges in culturally responsive practice.
Situating Power: A Conceptual Model for Analysis
Before moving to a discussion of our findings, we begin with a conceptual framework that has informed our thinking about power and that guided our analysis of the literature. Our current model is based on the prior work of Cousins and Chouinard (2012), which we have expanded upon based on our recent review and critique of the CRE literature. The original conception of power was depicted along three dimensions (relational, political, and discursive), to which we have added a fourth dimension based on our synthesis of the scholarly literature, the historical/temporal dimension (see Table 1). Each of these four dimensions is interconnected and dynamic and together helps illustrate the multiple levels or tiers of power within the evaluation context. Our four dimensional conceptualization of power (relational, political, discursive, and historical) can also be understood as hidden, visible, and invisible (Gaventa, 2006; Lukes, 2005; Veneklasen & Miller, 2002), depending upon whether it manifests as open and observable, or more furtively and less obvious, and hence more difficult to locate (see Figure 1). Our model frames power as a dynamic, relational, and productive concept (Foucault, 1979), as something that circulates and structures knowledge and its social construction (Foucault, 1980) at multiple levels within the research and evaluation setting. In what follows, we provide a brief discussion of our four identified dimensions.
A Conceptual Model of Power in Culturally Responsive Evaluation Approaches.
A conceptual model of power in culturally responsive evaluation approaches.
Relational Power
Power is considered a dynamic and relational concept that informs relationships and broader sociopolitical levels, constantly circulating and informing knowledge and meaning on multiple levels (Cousins & Chouinard, 2012). For Foucault (1980), power is not the purview of the state or of a higher authority (it is not a hierarchical notion of power), but rather something that is embedded in daily life, and that circulates and structures knowledge and its social constructions. Power can thus be conceptualized as a “network of social boundaries” (Hayward, 1998, p. 2) that includes laws, norms, institutional structures, and identities and that delimits the sphere of action. This relational view of power draws attention to how relations of power are socially and politically shaped and how they influence relationships between evaluators and stakeholders in the field. The knowledge that emerges out of this collaborative and fundamentally social process involves multiple, complex, and conflicting social, contextual, political, institutional, and cultural factors, as relations of power continue to define the parameters of place and possibility. Given the nature of most evaluation contexts and the relational dynamics that are inherent among human beings, unspoken or invisible power dynamics are often diffused among and across participants and evaluators and can greatly affect the evaluation process.
Political Power
Political power shapes the barriers and biases that prevent people from participation, influencing which voices and perspectives will be included and which will be excluded (Gaventa, 2006; Veneklasen & Miller, 2002). This dimension of power also includes the macrostructures of inequality (Kothari, 2001) around issues of gender, citizenship, sexuality, and class, as these demarcations often influence decisions about who can participate (and who cannot), all of which alters collaborative landscape and the knowledge that is ultimately constructed. At this level, power circumscribes the heterogeneity of inclusion (Mosse, 2013), delimiting who can and cannot participate in the evaluation at either the levels of design, planning, and process, all of which ultimately informs the construction of knowledge itself. Power and politics are considered central to evaluation, shaping both the practice and outcome of the evaluation itself (Parkinson, 2009).
Discursive Power
Beyond political power, some authors have also discussed discursive forms of power, a form which includes the internalized values, beliefs, and norms which govern practice and may prevent meaningful participation by certain groups or individuals (Cousins & Chouinard, 2012). This level of power connects the relations of power to the production of “truths” that govern our lives (Kothari, 2001). As Foucault (1980) describes, discourses do more than simply structure “reality” and what is considered valid knowledge, as they actually legitimate and create it. Discursive power, while perhaps less tangible and obvious (Veneklasen & Miller, 2002), circulates invisibly shaping our beliefs, values, and norms.
Historical/Temporal Power
The dimension of historical power emerged through our reading and synthesis of the empirical literature; it was not part of our original conceptual framework. It became apparent to us that power does not enter the setting only at the moment of contact and representation (Gupta & Ferguson, 1992) but rather it circulates and is continuously mediated by cultural and historical forces that reside outside of the local evaluation context. Here, we are reminded of Dorothy Smith (1987), a Canadian feminist scholar who described people’s everyday world of experience as historically and politically mediated by often invisible social and institutional forces that originate outside of the parameters of the local setting. In other words, the relationships that define our experiences in the field are socially constructed prior to the evaluation and thus must be understood within this broader historical, cultural, political, and economic narrative. As Cardoza Clayson, Castaneda, Sanches, and Brindis (2002) described, “the role of the evaluator occurs within a particular context, at a particular period of history, not independent of it” (p. 34).
Adding further complexity is the idea that we are all composed of numerous identities (such as gender, ethnicity, and class) that comingle, interact, and “intersect” to shape the multiple dimensions of what constitutes ourselves. The notion of “intersectionality” is thus used to refer to the interaction between gender, ethnicity, and other categories of potential identity and to understand how they are mutually constructed and produced, reproduced, and experienced in everyday life (Collins, 1998; Davis, 2008; McCall, 2005). Thus, there is no single, all-defining category that can capture or that can be used to define a community’s identity (or a person’s). Although we may perceive identities simultaneously, particularly if we consider their intersectionality, we must acknowledge that the categories do not necessarily comingle equally, depending upon the context at hand (Collins, 1998). It is therefore important to think about the multiple and intersecting categories at play, which enriches our understanding of the contextual and historical conditions that together create the multiple identities that define individual’s social, cultural, and temporal existence.
Method
Sample Selection
The purpose of the current literature review was to explore the role of power in evaluation practice through rigorously examining the cultural evaluation empirical literature. The goal of our analysis was to enhance our understanding of how the multiple dimensions of power manifest in practice. Unfortunately, although power dynamics have been widely discussed in theory by a myriad of scholars, there remains a significant theory to practice gap. As we identify ways that power manifests in practical empirical research, we can increase our tangible understanding of power dynamics at play and approach evaluation contexts by starting with power considerations, rather than reflecting on it retrospectively. Therefore, our guiding question for the current analysis was: How does power manifest in the cultural evaluation empirical literature?
Our literature search was limited to empirical studies in cultural evaluation contexts. This included any evaluation that specifically worked from a CRE approach or those that were conducted in cross-cultural program contexts and that considered culture a prominent variable for consideration in methodological processes. Culture was considered a key variable in evaluations that highlighted culturally based results, discussions, or lessons learned. In addition, we specifically analyzed empirical literature as opposed to conceptual works in order to explore how power is manifesting in the world of practice. For our purposes, empirical evaluations included not only traditional research methodologies (e.g., mixed-method, case study) but also critical or reflective narratives based on identified empirical evaluations conducted in cross-cultural program contexts. In order to identify our body of literature, search terms for this study included “cross-cultural evaluation,” “culturally responsive evaluation,” “cultural context,” “culturally responsive evaluation,” “cultural context,” and “culturally competent evaluation.” We searched a number of databases to obtain a wide range of scholarly literature such as ERIC and Psych Info. In addition, we closely examined bibliographies of recently published work in order to locate otherwise undetected or unidentified articles or studies that were not uncovered in our initial search. Our inclusion criteria included peer-reviewed articles in the North American context from the years 2000 to 2016, in order to examine more recent evaluation work. Although we cannot claim our sample is completely exhaustive, we made every effort to ensure it was sufficiently extensive in order to fully represent the current state of CRE literature.
Sample Characteristics
Our literature search and inclusion criteria led us to 18 empirical articles written between 2000 and 2016. All evaluations took place in North America including Canada (n = 2) and the United States (n = 16). The program contexts in our sample of articles roughly fell into the following four categories: education (n = 10), community or community initiatives (n = 3), health (n = 2), and social services (n = 3). All programs were designed to improve inequities or provide specific services to enhance the educational, health, or social challenges for the target population such as violence prevention, increasing underrepresentation, transforming distressed neighborhoods, parent education, improving positive educational outcomes for underrepresented students, smoking prevention, HIV prevention, rehabilitation, improving developmental outcomes, and enhancing school–family–community partnerships. All the target populations were people of color and/or underrepresented populations identified as Hispanic or Latino (n = 4), African American or Black low income (n =2), underserved or underrepresented 1 (n = 5), immigrants (n = 2), refugees (n = 1), Hmong (n = 1), low income (n = 1), or multiple participants in multiple contexts (n = 2). We explicitly omitted all evaluations conducted in Indigenous contexts, as we recognize that evaluations in these contexts are replete with power dynamics that are historically specific to these particular contexts (Chouinard & Cousins, 2009). The majority of evaluations utilized a mixed methods approach with other approaches including qualitative, case study, participatory action research, and Delphi method. Theoretical frameworks and approaches included CRE, social justice, social cognitive, transformative, postmodern, anthropological, critical theory, value based, talent development framework, and integrated theory of influence. Most of the evaluations included implications for evaluation practice particularly surrounding issues of culture and working with culturally diverse populations.
Review Strategy and Analysis
After the 18 articles were identified, we closely analyzed and discussed each article. Articles were placed in a comprehensive table that included the following sections to guide the readings and discussion: authors, program context, population context, evaluation approach and theoretical framework, relational power, discursive power, political power, and historical/temporal power. To identify power dynamics, we used a preordinate design based on the emergent conceptual framework in order to guide our analysis and reading of the articles (i.e., relational, discursive, political power). However, it’s important to note that our reading and analysis also informed our conceptual framework and added additional components as we identified aspects of power that emerged in the evaluation context. Specifically, we added the historical/temporal power category to the conceptual framework after analyzing the articles. Finally, we came to consensus as to the identified manifestation of power, which are outlined in the following section.
Of particular note, however, is that power is difficult to operationalize (hence the title of our article). The purpose of our article is to try and develop a conceptual model of power with the hopes that evaluators can better understand what it looks like and how they can locate it in practice. Part of our challenge in writing this, and in reviewing the articles, was determining how to operationalize the concept. We began by relying on a prior conceptual model as a guide, although it wasn’t particularly robust initially. Furthermore, evaluators did not necessarily talk about power explicitly, which made it hard to locate as well. Power is a very implicit issue for many evaluators, and as a result, we had to extensively review our sample of articles and, in some cases, try to read between the lines while remaining true to the literature at hand. As a result, we had to read and reread the articles to look for evidence of power. Through the process of analysis, we really strengthened and identified more operational constructs of power, ultimately adding to the final development of our conceptual model. The following section outlines our findings from this process and the specific ways that we identified power manifesting in CRE contexts.
Research Findings
Our analysis of the literature was guided by the conceptual model described in Table 1. What follows is a description of our findings associated with each of the four dimensions of power (relational, political, discursive, and historical/temporal).
Relational Power
Cultural responsiveness requires active stakeholder engagement and participation (Chouinard & Cousins, 2009), and as such, the majority of the studies included in this review focused on relationships, whether between the evaluator and stakeholders or between or among diverse stakeholder groups. We note that the use of collaborative methodologies raises issues of power and politics, shifting the focus to who participates (and who does not), under what conditions, and how they participate. Our findings point to relational power issues in framing processes of negotiation, alliances among stakeholders, competing interests, rules of inclusion and exclusion, and evaluator roles. In what follows, we identify some of the key issues we have identified in terms of the relational power dimension.
Power relations between and among various groups
As a number of evaluators have pointed out, all aspects of the evaluation process from design to outcomes are shaped by power relationships (Cardoza Clayson, Castaneda, Sanches, & Brindis, 2002; Stokes, Chaplin, Dessouky, Aklilu, & Hopson, 2011). Across our analysis, relations of power were identified between and among different levels of stakeholders including between evaluation staff, between the researcher (often as “outsider”) and participants, between groups, and between participants themselves. These relational power dynamics can be influenced by social class, education level, race, gender, status, and needs (Stokes et al., 2011; Thomas, 2004). To address these issues, a number of evaluators intentionally attempted to shift their focus to elicit the voices of underrepresented groups. For example, Christie and Barela (2005) specifically used the Delphi method in order to ensure that all previously silenced voices were heard since this method allows for all participants to have equal influence, while also maintaining their anonymity. Wilson Cooper and Christie (2005) shifted their planned evaluation design to include a social justice approach and changed their emphases from focusing on all stakeholders to specifically emphasizing the perspectives of the “least powerful stakeholder group,” in this case Latina parents (p. 2249). Mertens and Hopson (2006) described the importance of raising critical guiding questions in order to elevate the experience of marginalized groups such as, Who needs to be included? How should they be included? How can they be invited such that they feel truly welcome and able to represent their own concerns accurately? What kind of support is necessary to establish an appropriate venue for people with less privilege to share their experiences? (p. 43)
Additionally, a number of evaluators noted that stakeholders often have competing agendas and interests (e.g., Cardoza Clayson et al., 2002; Copeland-Carson, 2005; Harklau & Norwood, 2005), which may play a role in power differentials. Some evaluators saw the emergence of resistance given the enmeshed power relations among evaluators, the evaluated, and additional stakeholders. For example, Harklau and Norwood (2005) identified the resistance of students in their educational evaluation as evidenced by students questioning why the evaluators were there and other students trying to control what was documented about them. Additionally, they noticed the conflicts in perspectives and purposes across various stakeholder roles. For example, evaluators were directed by program staff away from activities or meetings where negative aspects of the program may have been discovered. Other evaluators identified power relations through the manifestation of tension or conflict. For example, Prilleltensky, Nelson, and Sanches Valdes (2000) described the tensions that arose among the evaluation committee comprised of Latin American immigrants that were members of the local community. As time came for members to write the final report, conflict arose between “veterans” (members that had been in North America for a longer period of time) and those that had more recently relocated. As Prilleltesky et al. (2000) noted, “this conflict…shows that even within minority groups there are cultural disagreements, something anyone who works with communities, homogeneous or heterogeneous, knows” (pp. 111–112). Thus, power relations were observed not only between evaluators and participants but between and among groups which sometimes manifested in resistance, tension, or conflict given competing agendas or interests.
Competing agendas or interests may also be explained by the concept of framing or how individuals organize and view their experience (Goffman, 1974). Stakeholders who hold different agendas often have a specific framework by which they view their goals, expectations, and previous experiences. As a result, they may identify “right” and “wrong,” “good” or “bad,” “healthy” and “unhealthy,” and so on. This has the potential to greatly influence power dynamics as each individual and/or group views the evaluation and the goals of the program from a particular frame of reference that ultimately influences their decisions and interactions with others. In turn, this may lead to tension, conflict, and competing agendas that can often be seen in CRE contexts.
Evaluator role is entrenched in power
In addition to the power dynamics that were observed between and among stakeholders and groups, the inherent role of the evaluator is entrenched in power and was evident throughout the majority of evaluations in our analysis. A number of evaluators recognized that there are inherent power differentials between evaluators/researchers and participants (e.g., Anderson-Draper, 2006; Cardoza Clayson et al., 2002; Copeland-Carson, 2005; Harklau & Norwood, 2005; LaPoint & Jackson, 2004; Small, Tiwari, & Huser, 2006). As Harklau and Norwood (2005) discussed, the role of the evaluator is inherently relational, with multiple and often fluid roles that include “subject positions” from which power is exercised (p. 281). For example, Small, Tiwari, and Huser (2006) reflected on the inherent privileged status they had as educated academic scholars of a dominant culture, which inevitably led to power differentials throughout the evaluation process. Similarly, Harklau and Norwood (2005) identified the power that was inherently conferred on them as representatives of a university. Beyond the specific reflections or identifications of the power inherent within the evaluator role, our analysis of these studies revealed that despite attempts to be inclusive, the evaluator nonetheless played a large role in the evaluation design and decision-making processes (e.g., Wilson Cooper & Christie, 2005), highlighting the power of that role. Overall, our analysis highlighted the entrenched power that was inherent in the research/evaluator role, which many evaluators observed as they reflected on the evaluation process.
Use of participatory and collaborative approaches
In order to address the inherent power relations that existed not only within the role of the evaluator but among various groups throughout the evaluation process, the majority of evaluators in our analysis attempted to diminish power relations through the use of collaborative strategies (e.g., Alkon, Tschann, Ruane, Wolff, & Hittner, 2001; Anderson-Draper, 2006; Cardoza Clayson et al., 2002; Christie & Barela, 2005; Conner, 2004; Copeland-Carson, 2005; Manswell Butty, Daniel Reid, & LaPoint, 2004; Small et al., 2006; Thomas, 2004; Wilson Cooper & Christie, 2005; Zulli & Frierson, 2004). Specifically, a number of evaluators attempted to involve community members through their inclusion on steering committees or by training community members in evaluation practices (e.g., LaPoint & Jackson, 2004; Prilleltensky, Nelson, & Sanchez Valdes, 2000; Small et al., 2006). Anderson-Draper (2006) reflected on the importance of truly engaging participants in the process as opposed to being consultative or simply asking for agreement with what the evaluator has already proposed. According to Stokes, Chaplin, Dessouky, Aklilu, and Hopson (2011), If the involvement of those who benefit from a program, the local community, is not sought, the evaluation will continue to have a colonial nature. Evaluation results will only inform those who manage the evaluation and already have access to the knowledge. (p. 168) Simply expecting members of another culture to adopt an egalitarian and collaborative approach could be seen as an example of the dominant culture researchers exerting their status on the community partner. (p. 362)
Political Power
Political power includes more microlevel political agendas and the macrosystem of inequality that influence evaluation processes and planning. The majority of evaluations within our review were situated within some type of organizational or societal political context that influenced the evaluation process, thus manifestations of power through the political context were observed or identified. Our findings highlight the need to consider whose agenda is being served throughout the evaluation process. Below, we identify some of the key issues related to political power.
Evaluation is innately political
Our synthesis revealed that evaluation is innately political with certain stakeholder agendas often valued over others. Political manifestations of power were identified in the majority of the evaluations included in our sample (e.g., Anderson-Draper, 2006; Copeland-Carson, 2005; Harklau & Norwood, 2005; LaPointe & Jackson, 2004; Manswell Butty et al., 2004; Ryan, Chandler, & Samuels, 2007; Small et al., 2006; Stokes et al., 2011; Thomas, 2004). For example, Anderson-Draper (2006) took time to reflect on whose agenda they were trying to push forward. They reflected on the specific time lines they had set, style of facilitation they employed (rigid or flexible), and the way they were approaching group discussion and facilitation, which felt inconsistent with their program team members. As a result, they adjusted their approach, concluding that, “concepts like fixed and firm start/end time and structured agendas should be relevant to the context and the ebb and flow of the group” (Anderson-Draper, 2006, p. 71). However, often due to funders restrictions and Western mentalities, many evaluators approached evaluation contexts with specific agendas and structured approaches that are culturally specific to Western research and evaluation techniques. As Kirkhart (2011) noted, when the primary agenda has been determined by external sources, it raises questions about what is valid and truly meaningful engagement throughout the evaluation process (Kirkhart, 2011).
Evaluations are situated within complex political contexts at both program and macro levels
Our review indicated that evaluations are situated within complex program and organizational contexts that can be politically charged (e.g., Christie & Barela, 2005; Wilson Cooper & Christie, 2005). In other words, political power often governs and shapes relational power (Cardoza Clayson et al., 2002). For example, in the educational evaluation conducted by Christie and Barela (2005), authors described how the organizational structure of the program itself was entrenched with power dynamics. Specifically, program staff were often left out of planning processes that were conducted by administrators of the program. Given the complex intersection between political power and organizational structures, evaluators found it important to adjust evaluation approaches accordingly. Wilson Cooper and Christie (2005) also discussed the importance of using a flexible approach when choosing and employing evaluation theory and methodologies, since each evaluation context is entrenched in context-specific political influences. They stated, “because educational programs are situated within complex local, district, and state political contexts, it is virtually impossible to replicate the conduct of an evaluation across settings” (p. 2251). Therefore, the political organization of the program itself, often determined through political contexts and ties (e.g., funding sources, hierarchies, hiring decisions), can reinforce manifestations of political power. As a result, it is important to utilize flexible and responsive evaluation approaches.
Beyond political considerations within the program organization specifically, our review indicated that there are also manifestations of political power that come from a broader context at the societal and macrolevel. As Cardoza Clayson et al. (2002) noted, in cross-cultural evaluations, political conditions within a country of origin influenced practices, beliefs, values, and attitudes of the evaluators, community, and participants themselves. They reflected that political conditions within countries influence migration patterns, cultural practices, attitudes, and participants’ desire to engage in participation. In their evaluation working with Latino migrant workers, Cardoza Clayson et al. (2002) identified the difference in civic engagement activities between Mexico and the United States, which highly influenced the practices of Latino families who lived in the United States but originated in Mexico. Additionally, given the political context of documentation statues within the United States, many Latino families feared deportation, which resulted in issues of trust within public participation and civic engagement, an important facet of their evaluation. Therefore, the evaluators noted the influence of the larger political contexts both from participants’ country of origin and the United States. Given the numerous political ties and implications that impact the evaluation process, data collection, and conclusions, a number of evaluators noted the political context on both an organizational and macropolitical level that highly influenced the evaluation process.
Evaluations in and of themselves have political ramifications
Since evaluation is innately political (Harklau & Norwood, 2005), our synthesis of the evaluation literature revealed that evaluation is not only influenced by political contexts, agendas, and power dynamics, but evaluation also has political implications in and of itself. Evaluation is “a politically contextualized act” (Cardoza Clayson et al., 2002, p. 34). Some evaluators reflected on the importance of considering how the results of an evaluation will impact communities, what the political consequences are, and how the results could be of value to the community rather than negatively impact community members (Mertens & Hopson, 2006). For example, Kirkhart (2011) described how individuals hold more than one cultural identification, making categorical or absolute cultural descriptors irrelevant. However, “categorical labels are often used to organize cultural information, and evaluators must be aware of how differential power and privilege become attached to such descriptors” (p. 76). In other words, evaluations that push categorical labels on participants may inadvertently create power differentials and privilege within a community as a result of the evaluation. Thus, evaluation is powerful and has the potential to positively or negatively influence the community. It is therefore essential to consider whose agenda is being pushed, particularly in cross-culture contexts that have predetermined sets of values and beliefs.
Discursive Power
Discursive power is another dimensions of power that includes the internalized values, beliefs, and norms that govern practice often determined by the societal discourse. Through analyzing the empirical literature, we identified a number of ways that discursive power was manifest throughout evaluations. In what follows, we identify specific issues of discursive power that emerged in our review of the literature. Our findings point to the importance of considering various aspects of societal discourse and language that influence the role of the evaluator and the evaluation process.
Discourses can limit and direct evaluators’ positions and roles
Societal discourse can influence evaluations and evaluators’ position on program specific objectives and outcomes. Through our analysis of the empirical literature, we observed that evaluations often have predetermined ideas of what is considered good and bad or have specific desirable outcomes and values (e.g., Alkon et al., 2001; Anderson-Draper, 2006; Copeland & Carson, 2005; Prilleltensky et al., 2000; Small et al., 2006; Thomas, 2004; Wilson Cooper & Christie, 2005). This can often be driven by funder or evaluators priorities. For example, one of the goals of the projects described by Alkon, Tschann, Ruane, Wolff, and Hittner (2001) was to “improve teachers’ and parents’ knowledge and beliefs about aspects of childrearing” (p. 50). However, the program developers were the ones who determined what was considered good or bad parenting beliefs, and cultures vary greatly in their beliefs around child-rearing. Therefore, if a program or evaluation is attempting to “improve knowledge and beliefs” around a certain topic, it begs the question, who determines what constitutes improvement? These beliefs are often culture specific and determined by individual societies. As Kirkhart (2011) stated, “organizations and institutions reflect their own values, assumptions, and knowledge bases that define organizational culture along dimensions such as disciplinary background, occupational norms and job descriptions, or role requirements” (p. 77). It is therefore important to consider societal and cultural specific discourses around certain topics that shape values, attitudes, and beliefs, which can influence desired program outcomes and evaluation results, as we observed in many evaluations in our review. Therefore, our review indicated the importance of considering discursive power, particularly when working in cross-cultural settings where the dominant culture may be imposing societal norms and beliefs upon participants, which are often determined by the overarching discourse.
Furthermore, our review revealed that discourse often limits and directs evaluators’ roles and positions. In other words, evaluators were not always aware of how societal norms and values influenced their worldview and their perspective of the evaluation at hand such as their goals, objectives, and so on. As Harklau and Norwood (2005) stated,
individuals are never fully self-aware about their own subjectivities and how they are shaped by societal discourses…few evaluators have addressed how institutional and societal forces or discourses (Foucault 1973) limit and direct the subject positions available to evaluation researchers. (p. 282)
Language can be a form of discursive power since it is culturally and socially constructed
Our review indicated that language specifically can be an important manifestation of discursive power. Since language is socially constructed and embedded in the complexities of culture (Cardoza Clayson et al., 2002), it can contribute to forms of discursive power, particularly when working with others who speak another language. A number of evaluators in our analysis discussed the role of language, interpretations, and meanings that were crucial dynamics at play throughout the evaluation process (e.g., Mertens & Hopson, 2006; Small et al., 2006; Stokes et al., 2011). As Cardoza Clayson et al. (2002) noted, an important role of the evaluator is to deconstruct and reconstruct words and concepts when interpreting or translating. This is important to ensure that language is culturally responsive and relevant to the community. Through some of the evaluations, we observed that this not only included communications with participants who speak another language, but also during data analysis. When analyzing data, it’s important to make every effort to derive valid meaning from the language used so it’s interpreted within the correct context (see Manswell Butty et al., 2004). As some evaluators noted, this goes beyond simple translation but should include in depth efforts to truly contextual language and meanings within a specific culture (Mertens & Hopson, 2006). For example, Conner (2004) described the importance of communicating and using the literal language of his participants who spoke Spanish, English, or a unique mixture of both. He stated, “although speaking in the language of the participants would seem to be an easy task for the evaluators to implement, there can be degrees of language use to attend to, both in speaking and in writing” (p. 60). Similarly, Mertens and Hopson (2006) discussed the importance of going beyond simple word-by-word translation to ensure deliberate and proactive effort to consider context and meanings. In other words, interpretation should be conducted without imposing our own language constrictions or simplified translation, which may ignore deeper meanings or specific viewpoints that may ultimately silence voices.
Historical/Temporal Power
Finally, the historical/temporal power dimension was a manifestation of power that was added to our conceptual framework after analysis of the empirical literature. Evaluators in our review described the importance of considering the historical context not only of cultural groups but of the program and evaluation itself. In what follows, we describe the manifestations of historical/temporal power that emerged as critical aspects of power to consider within cultural evaluations. Our findings reveal that the historical context has the potential to influence multiple aspects of the evaluation process.
Experiences in the past can shape present realities
As a number of evaluations revealed, the historical contexts and temporal spaces that individuals, groups, and communities experience can shape their present realities (e.g., Anderson-Draper, 2006; LaPoint & Jackson, 2004; Stokes et al., 2011). More specifically, historical contexts can shape the roles, values, and beliefs of both participants and evaluators. Small et al. (2006) provided an example of the necessity of obtaining an in-depth understanding of an ethnic or cultural groups’ historical context in cross-cultural program evaluation. They described the historical immigration of the Hmong community to the United States as far back as the 1970s with the subsequent challenges this group has faced in the United States since that time. This was an important component of situating their evaluation and understanding the historical context of the Hmong community that might shape their evaluation process. Some evaluators found that gaining a deep understanding of a group’s history was essential for evaluators to understand community political cultures and any historical power dynamics that may influence their evaluation in the present. Other evaluators described the importance of understanding historical challenges that some groups faced and how those influenced the roles and beliefs of participants. For example, Anderson-Draper (2006) identified the challenges inherent within the experience of immigration. In their evaluation on a family violence program, they described the importance of considering how the act of resettlement may have increased stress and family tension, with the potential increase of family violence, an important facet of their evaluation. Thus, historical experiences can shape current values, attitudes, and behaviors, which are important considerations in the evaluation process. This may also extend to within group differences that may exist because of historical experiences. As Alkon et al. (2001) reflected, there may be many within-group differences of a particular cultural group that are based on geographic, historical, and economic differences.
Historical marginalization, racism, and anti-migrant sentiment can influence manifestations of power
Similar to the historical context of a culture group, marginalization, racism, or anti-migrant sentiment in society as a whole, both presently and historically, may shape manifestations of power within the evaluation context (Cardoza Clayson et al., 2002; Christie & Barela, 2005). For example, Cardoza Clayson et al. (2002) described the importance of considering the historical context of their evaluations within Latino communities. They explain that the migrant workers’ lives are shaped by “historical racism” and “anti-migrant sentiment” (p. 36). As Thomas (2004) described, when individuals feel marginalized, it can create challenges in meaningfully engaging them throughout the evaluation. Given the importance of considering historical contexts, some evaluators emphasized the importance of using evaluation approaches that specifically identify historically marginalized and oppressed groups. For example, Wilson Cooper and Christie (2005) used a social justice approach in order to allow evaluators to highlight “historically disempowered and marginalized groups in order to benefit these groups” (p. 2271). They recognized that certain groups have had their voices silenced in the past and so they wanted to highlight these voices in their evaluation.
Consider the history of the program and evaluation itself
Beyond understanding the history and past experiences of participants and evaluators, our review revealed the importance of considering the historical context of evaluations and/or programs themselves within communities. Gaining a deeper understanding of the history of the program may enhance understanding of its progress, lack thereof, or program outcomes (e.g., Copeland-Carson, 2005; Kirkhart, 2011; Thomas, 2004). In other words, participants may have had previous experiences or paradigms of thinking surrounding the project under evaluation or the practice of evaluation, which shaped their roles, beliefs, attitudes, and actions toward the evaluation process. For example, in the educational evaluation conducted by Thomas (2004), evaluators described how many parents of the youth served by the program had previous negative or traumatic experiences within a school system when they were young, resulting in negative or apathetic attitudes toward school staff and others associated with the school system including the evaluators. Thus, evaluators found that historical contexts are critical to explore and understand. As Kirkhart (2011) noted, “the significance of prior history with evaluation itself, for example, is clearly visible” (p. 82). Specifically, Kirkhart (2011) described the importance of understanding the contextual influence of the evaluation itself. As a result, our analysis revealed the importance of having an intimate understanding of a group’s history, and any historical experiences participants or communities may have had with a given program or the evaluation process as a whole.
A Synthesis of the Research Findings
Through our identification of the various components of the manifestations of power, we identified a number of common themes that were integrated throughout the various dimensions of power. Overall, power dynamics are often ignored. Although we could often identify implicit and explicit manifestations of power on a relational, discursive, political, and historical level, a number of evaluators made no specific mention of power (e.g., Alkon et al., 2001; Manswell Butty et al., 2004; Zulli & Frierson, 2004). This is not to say that evaluators did not implement strategies to elicit participant voices or attempt to diminish inequalities within the evaluation process, but mention of power as a specific concept to be addressed at multiple levels was rare. Similarly, evaluators may have attempted to address power, but often lacked comprehensiveness or failed to identify multiple ways that power could be at play. For example, some evaluators attempted to enhance the engagement of participants, but when it ultimately boiled down to decision-making, evaluators or program staff were left with the most power and decision-making control. In addition, a number of evaluators attempted to include various stakeholders and community members, but it was unclear as to who decided which community members were invited to the proverbial table and asked to participate (e.g., Prilleltensky et al., 2000).
Implications
Considering the multiple, complex, and intersecting manifestations of power identified in the CRE literature, a number of important implications can be drawn. First, there is a need to carefully consider the types of evaluation methods employed. The majority of cultural evaluations in our analysis used a mixed methods approach. There is a need to include both qualitative and quantitative research methods to elicit feedback from participants in multiple forms. This is especially important given that the lived experiences of participants are central to creating true knowledge with the awareness that vulnerability and power must be critically analyzed in order to create a level playing field with participants and the evaluator (Dodson, Piatelli, & Schmalzbauer, 2007).
Similarly, it’s important to consider collaborative evaluation processes as opposed to top-down evaluation methods in order to address relational power. Although collaborative and participatory evaluations do not eliminate power dynamics by any means (e.g., Stokes et al., 2011), they are evaluation approaches that have the potential to consider power dynamics from the start of the evaluation process. Although these approaches often conflict with the “gold standard” of traditional research and evaluation instruments (Small et al., 2006; Thomas, 2004), they provide an alternative approach that has the potential to elicit marginalized voices and make the playing field more equitable. Beyond collaborative approaches, it is important that evaluation designs are culturally responsive (Mertens & Hopson, 2006). Evaluation theories should be used as “guides” and not as prescriptive forces when evaluators face different challenges. Having a culturally responsive research design might include having more flexible frameworks that allow for the discussion of power with the potential to remain open to implementing different theoretical frameworks in order to meet the needs of the community (Wilson Cooper & Christie, 2005).
Overall, given the multiple dimensions of power that were identified throughout the empirical literature, it is important that researchers and evaluators start with a comprehensive, strategic, and intentional consideration of all potential forms of power that may dynamically influence the evaluation process. If evaluators do not start with power, it potentially leaves the door open to invalid conclusions. Similar to Kirkhart’s (1994, 2010) multicultural validity, if power is not a central focus in all aspects of the evaluation design, process, and outcome, results may be invalid. As Kirkhart (2010) described, validity is central to good evaluations and multicultural validity ultimately refers to the trustworthiness across “intersecting dimensions of cultural diversity” (p. 401). Similarly, we may need to move in the direction of considering the intersectional dimensions of power as an important component of validity in cross-cultural evaluation.
In addition to implications for research, the identified power manifestations have important implications for evaluation practice. First, it’s important to consider the power of words, particularly when working in cross-cultural contexts. Providing translations in and of themselves are not enough. Translations must consider multiple contexts and meaning. It is important to work toward communication with participants that value interpretation as opposed to simply translation. As Cardoza Clayson et al. (2002) explained, “translation without contextualization can lead to mis-communication, particularly when working with people from different countries of origin. Interpretation and translation are inherently tied” (p. 39). Thus, making every effort to truly communicate and understand participants at a cultural and contextual level goes beyond merely translating words but shifts to understanding meanings.
Similarly, there is a need for evaluators to deeply and fully understand the context. This includes the dynamics of the community, culture, and history of groups and programs. Although this can often be time-consuming and challenging considering various political agendas, it’s important for evaluators to fully immerse themselves within the context of their participants, program, and community, both presently and historically, in order to begin to identify where power might be located. This is in line with starting with power considerations as opposed to reflecting back on them after the evaluation has already been completed (Chouinard, 2014). Taking the time to truly understand cultural contexts, norms, communication styles, and histories is essential.
In summary, it is important to find ways to start with consideration of the various power dynamics at play in the evaluation context, as opposed to merely reflecting on it retrospectively. In order to start with power as a central construct before conducting an evaluation, there are a number of practical questions that evaluators can ask. For example, “How are individuals selected to contribute to the evaluation process?” “Which voices are being heard and why those voices?” and “Whose decision-making is driving the development, implementation, and evaluation of the program and why are they the ones driving the process?” It is essential to ponder these questions particularly when considering collaboration and participation practices, as power dynamics are manifest throughout the selection process. For example, “Are the stakeholders chosen to participate the ones who already support or have connections with program staff?” and “Are there other voices that could be added to the dialogue but may be more difficult to recruit (i.e., family members, teachers, etc.)?” Although these are just a few examples, it would be helpful for evaluation teams to identify a list of questions that are directly relevant to their specific evaluation context and work to answer them as a team in order to uncover various manifestations of power throughout the planning and implementation process.
Concluding Thoughts
Power is an incredibly complex, often intangible aspect of the evaluation process. However, through analysis of the empirical literature on CREs, we identified intersecting power dynamics that manifested across the relational, political, discursive, and historical/temporal dimensions. Through identifying the specific ways that power has manifested in cultural evaluations, we begin to take a step forward in decreasing the gap between theory and practice. Although power is a common topic among conceptual evaluation literature, it is often rarely addressed or specifically identified in empirical practice. Attuning to power throughout the evaluation process may be challenging, time-consuming, and difficult to uncover; however, it is imperative that it is addressed as a central concept throughout the development and implementation of evaluation practice. This is crucial not only for the validity of the evaluation as a whole, but when considering the impact of the evaluation itself. As Stokes et al. (2011) stated, “Power relationships that lead to injustices will have to be sorted out and will need to be addressed in order for program participants to achieve the full benefits of an intervention” (p. 169). Therefore, the current article attempted to take a step forward in our understanding of how power dynamics manifest in practice. It’s important for the evaluation community to continue to take steps forward and strategically identify and address the ways that power manifests in CRE contexts.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.