Understanding relationships in culturally complex evaluation contexts

Introduction

Cross-cultural evaluation (also referred to as ‘culturally competent’, ‘culturally responsive’, ‘transformative’, ‘multicultural’, etc.) is an explicit approach to evaluation that actively positions culture as a key construct in the evaluation of programs, leading to a focus on locating methodological practices commensurate with the culture, context and values of the program community. This approach, most often rooted in a concern for social justice and empowerment, recognizes that context and culture include not only demographic descriptions of communities and programs, but more importantly, diversity in values and the less vocalized issues of power, racism, and economic, class and gender issues that continue to define our society (Senese, 2005; SenGupta et al., 2004). This characterization of cultural context provides a sense of the multidimensional and multifaceted nature of context, its complex ecology, ² which in cross-cultural settings become all the more significant (SenGupta et al., 2004), as the context itself becomes the site of confluence where program, culture and community connect (Chouinard and Cousins, 2009). What defines a community, and where do its boundaries begin and end? These questions become paramount within cross-cultural evaluation settings, as understanding local cultural norms and the parameters of cultural context helps ground the evaluation in the local cultural, historical and political dynamics of the community.

This article is framed by an understanding of social inquiry as a highly contextualized and relational process that is contingent upon the multiple perspectives and voices of researchers and of those being researched. Of particular interest is the relationship between evaluators (researchers) and communities (participants), in what is spoken and not spoken, what is assumed and taken-for-granted, and what we as evaluators bring to the table, and how it is transformed within and out of the interactions that ensue. Fine (1994) refers to this as ‘working the hyphen’ (p. 135), a place where researchers explore the margins of their social location as both self-identities and ‘others’ come together, merge and separate. Relationally, the hyphen is understood not only in terms of how we are ourselves, as evaluators, in relationship to others within the local program context, but also within a larger, multifaceted historical, cultural and sociopolitical system. As Sirin and Fine (2008) elaborate, ‘the psychological texture of the hyphen is informed by history, the media, surveillance, politics, nation of origin, gender, biography, longings, imagination, and loss’ (p. 195). How do researchers work this multi-dimensional and multi-textual hyphen? How do they represent others? Themselves? Whose stories are told? Where are researchers located in the telling of this story?

To explore these questions, I turn to my prior research that explored the influence of culture on the evaluation and on the program setting, and on how it mediates the relationship between evaluators and diverse community stakeholders. Through an interconnected three-phase study that included a comprehensive literature review (Chouinard and Cousins, 2009), interviews with evaluation scholars and practitioners and focus groups with community-based program managers, I found that culture and cultural context influence every dimension of evaluation. My research highlighted the influence of culture on the relationships that are developed with stakeholders (and among stakeholders themselves), in terms of the ecological context of the community’s social and political history, in the methodologies and research methods selected, in the politics of the context as power and privilege interweave throughout the setting, through the organizational norms, policies and agendas that structure the evaluation, and in the personal biases and traits of the evaluator him or herself. The multi-dimensional influence of culture, what can really be defined as cultural complexity in terms of the influence on the evaluation and on the program setting, leads to a number of dilemmas in terms of implementing evaluations in diverse community contexts, dilemmas that are conceptualized at both the level of process (i.e. in terms of conducting culturally valid and responsive evaluations) and philosophically (i.e. in terms of the social constructivist and hermeneutic dimensions of our research in cultural communities). To further explore these dilemmas, in this article I begin with an initial discussion of the key theoretical constructs that guide and frame this research, positioning research and evaluation methodologies as cultural and political expressions that influence the knowledge that is created in the field. Second, I identify three dilemmas, or what Smith (1987) refers to as ‘problematics’, that came to the surface in my analysis and that I believe remain ongoing dilemmas in cross-cultural evaluation contexts. I conclude with a brief discussion of enduring issues and questions of practice.

Theoretical underpinnings

In the following section, I provide a brief summary of postmodern theory and connections to how we create and construct meaning in cross-cultural evaluation and program contexts.

Three ‘problematics’ identified in conducting cross-cultural evaluation

My initial research highlighted the many inter-related cultural dimensions that surface in interactions between evaluators and stakeholders in cross-cultural evaluation and program settings, underlying which are a number of dilemmas that persist. These dilemmas, described in institutional ethnography by Smith (1987) as ‘problematics’, are partially expressed within the local context, but historically and politically mediated through institutional and social forces that originate outside of the parameters of the local setting. A problematic, according to Smith (2005), ‘sets out a project of research’ (p. 227) that focuses on understanding how people’s everyday world of experience is externally structured and ‘put together by relations that extend vastly beyond the everyday’ (p. 1). Through analysis of interview transcripts and a comprehensive review of the literature, I provide three specific problematics that I believe continue to create ongoing dilemmas in cross-cultural program and evaluation contexts.

The first problematic is related to the fact that one can be both an insider and an outsider (at one and the same time), further obfuscating initial assumptions that we might make based on gender, ethnicity, education, age, etc. Do we pair evaluators and research participants along gender, ethnic or other identity lines? How do we determine which parameters and definitions of identity are important within a specific context? While this continues to be a contested area of research and practice, my research suggests that as identity is a multifaceted and fluid concept, it becomes increasingly difficult to accurately determine insider or outsider status based on simple, singular dimensional notions of cultural identity.

The second problematic is concerned with the role evaluators play in cross-cultural contexts, as their identities cannot be divorced from the politics of the evaluation and context (the ‘politics of place’), nor from their political/ethical stance in terms of process, results and uses of evaluation (Weiss, 1993). How do evaluators make sense of the diverse roles and role expectations within cross-cultural contexts? How do they balance the multiple and potentially conflictual role expectations among the diverse stakeholder groups? Evaluators have choices to make, many of which occur outside of the program and evaluation context, choices that concern their ‘performance’ of evaluation, and that ultimately reflect upon their role and identity as researchers.

The third problematic represents the myriad political, social and cultural implications behind the methodological selections and choice of methods, and the adoption of collaborative approaches as a way to bridge the cultural chasm. Can collaborative methodologies provide a context to bridge the space between evaluators and stakeholders? Can these methodologies mitigate issues of power between evaluators and stakeholders (or between stakeholders themselves)? While collaborative approaches to evaluation may help bring all stakeholders to the table and may ameliorate some power differentials, challenges of creating meaningful participation in the context of evaluation in cross-cultural contexts persist.

Problematic one: the multiple faces of identity

One of the challenges in conducting evaluations in diverse community contexts is finding evaluators or researchers who have knowledge of the community and its culture, language and history. The notion of identity, of who we are and of how we are positioned and categorized, is of fundamental concern in cross-cultural research, as we make assumptions about self and others that guide our decisions as evaluators in the field. The challenge thus becomes determining what is salient within a specific context, as it is not always self-evident who is in or who is out, thus moving the emphasis to developing relationships to provide needed contextual and cultural clarity about what is relevant and what is not, in a particular setting.

This problematic represents the ecological and special connection of self and others to social, political, historical and cultural locations, as we move from being outsiders to insiders (or vice versa), and as our proximity to the community and to ourselves shifts and changes over time. The awareness of self as cultural being and as evaluator leads to an internal dialogue about identity and about the social categories that are assumed throughout the research encounter. Identities are not fixed but mutable and transformed through inter-relationships and interactions with others. There is no one category or single all-encompassing label that can capture the essence of participants and researchers, particularly if we consider the notion of identity as a ‘production’ that is in a constant state of change, being transformed within the specific historical and cultural discourses of the time (Hall, 1990), and through our social interactions with others. Identity is understood, not as a fixed, static essence, but a positioning (Hall, 1990), a situational performance (Goffman, 1959) that is never complete, leading to a multi-dimensional notion of identity that is created and recreated through discursive practices (Hall, 1996). As Hall (1990) describes:

Cultural identities come from somewhere, have histories. But like everything which is historical, they [identities] undergo constant transformation. Far from being eternally fixed in some essentialized past, they are subject to the continuous ‘play’ of history, culture and power. Far from being grounded in a mere ‘recovery’ of the past, which is waiting to be found, and which, when found, will secure our sense of ourselves into eternity, identities are the name we give to the different ways we are positioned by, and position ourselves within, the narratives of the past. (p. 70)

These overlapping and intertwining notions of identity thus occur across cultural, class, gender and racial divides, and across political, economic and social histories that are saturated with unequal status, power and privilege. The many points of inclusion and exclusion, of being a part of and being outside, underscores the very complexity of our relationships with self and others in the research encounter. Relationships between evaluators and the stakeholder community, between notions of personal identity and the identity of others, thus influence the opportunities and limitations within the evaluative context, circumscribing the program and the co/construction of knowledge as the evaluation unfolds (Abma, 2006).

My research underscored the point that intersections of identity were particularly marked for those who considered themselves as both cultural insiders and academic or privileged outsiders (Brayboy, 2000; Thapar-Bjorkert, 1999), both uneasy dualities considering the multiplicity of roles and identities potentially enacted throughout the evaluative and research encounter. Rosaldo (1989) refers to this duality as one of ‘danc[ing] on the edge of a paradox’ (p. 180), simultaneously living positions both of colonized and colonizer (Brayboy, 2000). The insider/outsider binary thus underscores the difficulty of ascribing fixed notions of identity, as it neglects the interactive process in the creation of identities (Naples, 1997) while it belies the complexity of the cross-cultural encounter. The search for the right cultural informants and the use of evaluators from similar ethnic backgrounds as program participants may thus prove problematic, as it may reify cultural traits by oversimplifying the factors that create identity (Schick, 2002). At the same time, it does not account for the heterogeneity among cultural members nor intra-cultural variables, confusing issues of power, knowledge and exclusion with finding the right informants (Schick, 2002). The interactions and inter-relationships between evaluators and stakeholders thus raises important questions about the construction of identity, and about the multiplicity of often overlapping and intersecting identities (tied to concepts of gender, race or class) that abound in the cross-cultural program context.

Globalization ensures that cultures are no longer insular (Bandura, 2002), as they are constantly being constructed and remade through processes of migration and acculturation (Guarnaccia and Rodriguez, 1996). Identities are thus created discursively, constructed in specific historical and political contexts (Hall, 1996), in relation to others (McRobbie, 1992), and through continuous struggle, as they are constructed and reconstructed over time. Given the complexity of culture and the formation of identity, it thus becomes necessary to challenge the epistemological status of our definitions of what constitutes a social category as well as the terms of inclusion (Rogoff, 2003; Schick, 2002), making decisions concerning who’s in and who’s out, as a way of bridging the cultural chasm. As Moharty (1987, as cited in Pickering, 2008) explains, ‘the experience of being a woman can create an illusory unity, for it is not the experience of being a woman, but the meanings attached to gender, race, class, and age at various moments that is of strategic significance’ (p. 39). The sense of ‘illusory unity’ based on shifting categories of identity, of ascribed status (Merton, 1972) whether of woman, ethnicity, class or age, can lead to a false sense of privileged access to social and cultural truths (Merton, 1972). While this underscores the complexity of the insider/outsider dilemma (see Brayboy, 2000; Bridges, 2001; Howarth, 2002), it also raises key epistemological questions about the social construction of knowledge, assertions of truth, and about who has privileged access to, and has the right to speak, for others. Thus, while the complexity of identity and the philosophical and empirical problems associated with creating fixed categories in terms of who is in and who is out are significant, it fails to address the significance of research relationships (and ‘interactive roles’) and the influence of community and program context in shaping research outcomes (Schick, 2002). As the problematic of perspective and location of self and others implies, it is our coming together as evaluators and stakeholders that creates the very possibility of change.

Problematic two: the role of the evaluator amidst situational and cultural complexity

The evaluator’s role in the cross-cultural setting is contextual, dynamic, multiple, political, and engaged, helping to shape relationships and build understanding among stakeholders (including both program sponsors and communities) as the evaluation unfolds. My findings suggest that understanding of self and others provides contextual and cultural grounding, as the multiple roles we assume also reflects the dynamic and multifaceted identities that we play in the field. The challenge thus becomes how we come to understand and enact our roles as evaluators in cross-cultural contexts, and with whether our roles capture the situational and cultural complexity with which the program and evaluation context demands.

While there is tremendous conceptual variation in terminology and in the use of the concept across multiple applications and fields, the sociological literature on role theory (Bailey and Yost, 2001; Outhwaite, 2008) provides some clarity for understanding the evaluator role(s) in diverse community contexts. The literature focuses on roles from a structural perspective, envisioning it as an orchestration, a process, or phenomenologically in terms of role dynamics (Outhwaite, 2008). Roles can be understood at the individual level as an enactment, a dramaturgical construction, as well as from a social systems perspective, where roles are tied to the historical, social, political and cultural context thus embodying the values of the social system (Bailey and Yost, 2001). The notion of role, much like identity, is also a very fluid complex concept that is both socially and personally expressed, occupying an ‘in-between-ness’ (Ibrahim, 2008), and a sense of movement and confusion where we can be at once connected and disconnected from self and others. As Goffman (1959) vividly portrays, ‘the image that emerges of the individual is that of a juggler and synthesizer, an accommodator and appeaser, who fulfils one function while he is apparently engaged in another’ (p. 40).

Symbolic interactionism takes a middle position, envisioning roles in reciprocal relationship between the individual and society (Bailey and Yost, 2002), not as static entities, but situationally complex, emerging and evolving in relationship to others and to society (Turner, 2002). From Goffman’s (1959) perspective, there is a complex relationship between the roles that we perform in interaction with others, and in our construction of identity, as both persons and roles are dependent upon social definition in equal measure. While both role and identity can be understood as broadly relational and ecologically connected constructs, with role the emphasis is on the performative and enactive, rather than on the social categories that we create.

This characterization of role is thus very much attuned to social constructivist notions, as the roles evaluators assume in cross-cultural contexts imply a different understanding of evaluation (of what it is and what it can be), from that of a disinterested social science to a more engaged method of social inquiry (Schwandt, 2002). This highly contextualized and relational approach to evaluator role is also reflective of a new conceptual understanding of evaluation (Abma et al., 2001; King and Stevahn, 2002), from a social science defined by neutrality, objectivity and dispassionate observation, to one of engagement and richly textured and interpretive inter-relationships (Ryan and Schwandt, 2002).

Some researchers abandon the traditional conception of role altogether, as they argue that it fails to accurately capture the range of personal and political relationships that are embedded in evaluation and that are considered paramount in the cross-cultural evaluation and program context (King and Stavahn, 2002). Mathison (1994) describes a shift in the role of the evaluator, from expert to one of collaborative partnership, emphasizing the ongoing, embedded and relational aspect of role. For Greene (2000), the influence of context and multiple relationships on our roles (as evaluators in the field) is conceptualized as ‘evaluator-as-engaged person’, a stance that reflects how we are present in our work, our engagement with moral and ethical complexity, as well as our engagement from the perspective of various positions and locations within society (where we come from). Schwandt (2002) also sees role as a relational concept, but he draws a distinction between role as something that is part of our core, stable self, and as something that can be put on and taken off at will, and role as a praxis-oriented concept that is discursively constructed, tied to notions of identity, and continuously enacted and re-enacted in practice. Abma and Widdershoven (2008) also note three distinctive evaluator roles that include evaluator as social science expert and judge (signifying relational distance), evaluator as provocateur who challenges societal power and social relations, and evaluator as someone who understands the inter-relational aspect of evaluation as fundamental to the very process of evaluation itself. This latter view is based on a hermeneutic and constructivist perspective that, as Abma and Widdershoven (2008) explain, is premised on the belief that ‘understanding our socially constructed world can only be generated by developing a relation and dialogue with and between the inhabitants of this world’ (p. 211).

The role of the evaluator is thus characterized as situationally and contextually fluid and complex, as well as relationally, interactionally and discursively enacted (through our relations to self and others), within the evaluation and program context. Conceptualizing the evaluator role from a relational, socially constructed and performative perspective, recasts relationships within the cross-cultural evaluative context as an asset rather than as something that needs to be overcome (Hopson, 2002), and brings the focus to an understanding of diverse stakeholder needs, interests and program goals. The point is that we bring who we are to the evaluation, including our values, preferences, aspirations, biases, and what I have referred to previously as our identities, from the questions we pose, to the stakeholders we include or exclude, and to what we envision evaluation to be. The role of the evaluator, much like identity, is thus deeply embedded and shaped by the local program and community context, as well as by the broader social, historical and political forces that influence the local program setting (Hopson, 2003). The traditional characterization of role thus underplays the dynamic aspect of our connections with others and with ourselves (Schwandt, 2002), both critical components to understanding within the cross-cultural program and evaluation context.

Problematic three: dissonance despite methodological pluralism

Cross-cultural evaluation is an explicit approach to evaluation that considers culture a key consideration in the evaluation of programs, leading to the search for methodological practices that are commensurate with the culture, context and values of the program community. One of the key assumptions behind this approach is the idea that culture is an integral part of the context of evaluation, not only in terms of the program setting, but also in terms of program design, and in the methods and approaches that evaluators adopt for use in their work (SenGupta et al., 2004). My findings suggest that in an effort to conduct culturally responsive evaluation in culturally distinct communities, evaluation is seen as a form of collaboration and partnership between evaluators and community members, leading to an increased understanding of unique socio-cultural characteristics, processes and perspectives (Chouinard and Cousins, 2009). While some of the challenges result from the dynamics of the relationship between evaluators and stakeholders (e.g. power differentials, community and community member voice), other challenges relate to the political and social history of the community vis-à-vis the dominant society, as well as concomitant levels of trust and respect. Still other challenges relate to the fact that evaluation itself represents a form of cultural authority (House, 1993), that is deeply implicated and influenced by the political, social, historical and economic context in which it occurs, all of which underscores the need to critically look at the research methodologies that are adopted and practiced in the field, as well as the knowledge that is generated through the collection, analysis and dissemination of findings.

This problematic thus represents the political, social and cultural implications behind our methodologies, as evaluators and stakeholders together construct and co-construct the knowledge that is created through the process of evaluation. While the participatory approach is not without its challenges in cross-cultural contexts, there are three primary benefits cited for its use in multiethnic and multicultural settings: 1) it increases the validity and relevance of research findings; 2) it builds the cultural competency of evaluators; and 3) it builds local capacity, community knowledge and empowerment. Despite these benefits, the co-construction of knowledge and meaning between diverse stakeholders and evaluators is not uncontested, as one cannot discount the imbalance of power between evaluators and stakeholders (House and Howe, 2000), particularly in communities with a history of exploitation and marginalization. While collaborative approaches may create a more inclusive and participatory context, and may in fact help mitigate some power differentials, merely inviting everyone to the table is not enough.

While evaluators who work in cross-cultural program settings acknowledge the challenges of conducting evaluations and building collaborative relationships amidst unequal power and status, my research findings suggest that this fact remains under-reported in both the literature and empirical findings. As a relational and productive concept (Foucault, 1979), power informs both the relationships between evaluators and stakeholders, as well as the broader socio-political level, what Gaventa and Cornwall (2006) refer to respectively as the ‘micro-political relationships of power’ and the ‘macro-political relationships of power’ (p. 73). Power is not conceptualized as something that is possessed by any one individual, state or higher authority, but as something that circulates and structures knowledge and its social constructions (Foucault, 1980). Power can thus be conceptualized as a ‘network of social boundaries’ (Hayward, 1998: 2), that includes laws, norms, institutional structures, and identities, and that delimit what can and cannot be done. This relational view of power draws attention to how relations of power (and powerlessness) are socially and politically shaped, and how they influence relationships between evaluators and stakeholders in the field.

The knowledge that emerges out of this collaborative and fundamentally social process involves multiple, complex and conflicting social, contextual, political, institutional and cultural factors, as relations of power continue to define the parameters of place and possibility. The use of collaborative methodologies and culturally resonant methods cannot mask evaluation’s inherent cultural authority (House, 1993), nor obscure its power to define the parameters of what constitutes legitimate discourse and knowledge in the social sciences (Reagan, 1996). Reliance on collaborative methodologies, through the use of stakeholder agreement or terms of reference (for example), is thus insufficient, particularly if one considers the broader social, cultural and political context within which our methodologies are embedded. Consider, for example, over 2000 years of Western civilization, the Western canon. That is what evaluators carry in their bags, in their evaluation toolkit.

It thus becomes essential that evaluators themselves develop cultural fluency, not only about themselves and about the program and the people being evaluated, but about the wider cultural systems within which we are all enmeshed. Evaluators, for example, need to see themselves not only as ‘situated in the action of [their] research’ (Rupp, 1983, as cited in Anderson, 1993), but also as purveyors of Westernized inquiry methodologies that might not advance understanding of diverse cultural lives and experiences. As conscientious social researchers working in cross-cultural settings, it is important to acknowledge that research and evaluation methods continue to reflect Western European perspectives (Cannella and Lincoln, 2004; Conner, 1985), raising the question of whether these methods can ever accurately reflect and describe the experiences of the program communities. According to Smith (1999), Westernized research methods ‘are underpinned by a cultural system of classification and representation, by views about human nature, human morality and virtue, by conceptions of space and time, by conceptions about gender and race’ (Smith, 1999: 44), all of which serves to potentially misrepresent the very communities that evaluators and researchers seek to understand.

Concluding thoughts

In this article, I have used a critical theoretical perspective in which to explore the dynamics of what I have termed ‘problematics’ in cross cultural evaluation. This critical perspective has enabled me to re-examine the methodological approaches that we use, and to better appreciate the complex and often defining social, cultural and political phenomenon involved in cross-cultural research and evaluation. While each of the three problematics that I identified is distinct, there is much conceptual overlap, as each problematic illustrates the deeply re-presentational issues that are potentially (and often) raised as we conduct evaluations in culturally complex contexts. The first problematic (the many faces of identity), underscores the complexity of the relationship between self and others in the research encounter, as intersections of our identities (and others) are given expression depending upon the contingencies of presenting contexts. This problematic brings the focus to how we as evaluators locate ourselves (whether we position ourselves as insiders or outsiders) and on how that positioning influences our relationship to the text and to the program community. The second problematic (the role of the evaluator amidst situational and cultural complexity), shifts the focus to evaluator roles that are discursively and socially constructed within the demands and parameters of the research and evaluation context, as evaluators focus on building relationships to bridge what is often a cultural divide. While conceptually tied to the notion of identity, this construction of evaluator role repositions our understanding of evaluation (what it is, who it serves) and repositions the evaluator as relationally engaged with the stakeholder community (whose voices we include, whose voices we exclude). The third problematic (dissonance despite methodological pluralism), highlights the socio-political implications of our methodological choices, as evaluators adopt collaborative approaches to mitigate issues of power and privilege so often pervasive in the cross-cultural research encounter. This problematic brings the focus directly and critically to the cultural validity of our method choices and to the challenges involved in representing the diversity of voices and perspectives despite the use of collaborative approaches.

These three problematics raise what I term ‘enduring issues of practice’, defined as underlying areas of practice that generate perplexing questions requiring further consideration and ongoing research. First, we discuss categories, identities and roles as fluid and contextualized constructions, yet they do have a very concrete reality inside of our institutions. The question then becomes, how do we bridge these seemingly theoretical and yet very real cultural territories, in a meaningful way? Second, the evaluator is deeply implicated in matters of methodology and epistemology, as they ultimately shape what they see, what they hear, and the information that they gather. The question thus becomes, where are evaluators located in the notion of ‘the master’s tools will never dismantle the master’s house’ (Lorde, 1984). Third, despite the need for culturally resonant methods particularly in diverse communities, the demand for impartiality and objectivity persists. The question thus becomes, how do evaluators balance the work that they do cross-culturally, with the demands of accountability and evidence-based practice that is so prevalent today? And finally, power is a persistent theme throughout all of the literature on culturally responsive research and evaluation. Notably, studies conclude with enduring power differentials between researchers or evaluators and the program community. It is important to note, however, that we adopt these culturally specific methods precisely because these power differentials exist. The question is thus that if we start with power as a central construct (rather than end with it), what will it allow us to see?