Understanding relationships in culturally complex evaluation contexts

Abstract

Keywords

critical methodology cross-cultural evaluation culturally responsive evaluation culture identity

To break a cycle of repetition, it becomes necessary at some point to go past the edge of the familiar and enter a place that is truly unknown . . . in the absence of the willingness to risk relationship – the experience of really hearing and taking the other’s voice into oneself – the talking just goes on and on, because in the absence of relationship, change is impossible. (Taylor et al., 1995: 209)

Introduction

Cross-cultural evaluation (also referred to as ‘culturally competent’, ‘culturally responsive’, ‘transformative’, ‘multicultural’, etc.) is an explicit approach to evaluation that actively positions culture as a key construct in the evaluation of programs, leading to a focus on locating methodological practices commensurate with the culture, context and values of the program community. This approach, most often rooted in a concern for social justice and empowerment, recognizes that context and culture include not only demographic descriptions of communities and programs, but more importantly, diversity in values and the less vocalized issues of power, racism, and economic, class and gender issues that continue to define our society (Senese, 2005; SenGupta et al., 2004). This characterization of cultural context provides a sense of the multidimensional and multifaceted nature of context, its complex ecology,² which in cross-cultural settings become all the more significant (SenGupta et al., 2004), as the context itself becomes the site of confluence where program, culture and community connect (Chouinard and Cousins, 2009). What defines a community, and where do its boundaries begin and end? These questions become paramount within cross-cultural evaluation settings, as understanding local cultural norms and the parameters of cultural context helps ground the evaluation in the local cultural, historical and political dynamics of the community.

This article is framed by an understanding of social inquiry as a highly contextualized and relational process that is contingent upon the multiple perspectives and voices of researchers and of those being researched. Of particular interest is the relationship between evaluators (researchers) and communities (participants), in what is spoken and not spoken, what is assumed and taken-for-granted, and what we as evaluators bring to the table, and how it is transformed within and out of the interactions that ensue. Fine (1994) refers to this as ‘working the hyphen’ (p. 135), a place where researchers explore the margins of their social location as both self-identities and ‘others’ come together, merge and separate. Relationally, the hyphen is understood not only in terms of how we are ourselves, as evaluators, in relationship to others within the local program context, but also within a larger, multifaceted historical, cultural and sociopolitical system. As Sirin and Fine (2008) elaborate, ‘the psychological texture of the hyphen is informed by history, the media, surveillance, politics, nation of origin, gender, biography, longings, imagination, and loss’ (p. 195). How do researchers work this multi-dimensional and multi-textual hyphen? How do they represent others? Themselves? Whose stories are told? Where are researchers located in the telling of this story?

To explore these questions, I turn to my prior research that explored the influence of culture on the evaluation and on the program setting, and on how it mediates the relationship between evaluators and diverse community stakeholders. Through an interconnected three-phase study that included a comprehensive literature review (Chouinard and Cousins, 2009), interviews with evaluation scholars and practitioners and focus groups with community-based program managers, I found that culture and cultural context influence every dimension of evaluation. My research highlighted the influence of culture on the relationships that are developed with stakeholders (and among stakeholders themselves), in terms of the ecological context of the community’s social and political history, in the methodologies and research methods selected, in the politics of the context as power and privilege interweave throughout the setting, through the organizational norms, policies and agendas that structure the evaluation, and in the personal biases and traits of the evaluator him or herself. The multi-dimensional influence of culture, what can really be defined as cultural complexity in terms of the influence on the evaluation and on the program setting, leads to a number of dilemmas in terms of implementing evaluations in diverse community contexts, dilemmas that are conceptualized at both the level of process (i.e. in terms of conducting culturally valid and responsive evaluations) and philosophically (i.e. in terms of the social constructivist and hermeneutic dimensions of our research in cultural communities). To further explore these dilemmas, in this article I begin with an initial discussion of the key theoretical constructs that guide and frame this research, positioning research and evaluation methodologies as cultural and political expressions that influence the knowledge that is created in the field. Second, I identify three dilemmas, or what Smith (1987) refers to as ‘problematics’, that came to the surface in my analysis and that I believe remain ongoing dilemmas in cross-cultural evaluation contexts. I conclude with a brief discussion of enduring issues and questions of practice.

Theoretical underpinnings

In the following section, I provide a brief summary of postmodern theory and connections to how we create and construct meaning in cross-cultural evaluation and program contexts.

A view of postmodernism

The turn to postmodernism, to a philosophical perspective that is based on the ‘incredulity of metanarratives’ (Lyotard, 1979: xxiv), enables a conscious acknowledgement of the plurality of voices in the field and a questioning of the taken-for-granted assumptions that guide social science inquiry methods. As Kinchloe and McLaren (1998) explain, ‘the thrust [of postmodernism’s] critique is aimed at deconstructing Western metanarratives of truth and the ethnocentrism implicit in the European view of history and the unilinear progress of universal reason’ (p. 271). In rejecting the belief in a single, monolithic criterion of truth that is applicable in all contexts and at all times, postmodernism opens up the possibility of a culturally relevant approach to social inquiry that does not privilege one perspective over others, and that recognizes the contextual and localized nature of knowledge construction, thus giving priority to the inclusion of diverse voices in the field. The recognition of multiple, localized and partial truths (over one universal meta-truth) means that no single group has a monopoly on the truth (Howe, 1994), and that knowledge (and its construction) must be understood and appraised within its own cultural and social setting, all very important considerations in the context of cross-cultural evaluation.

In denying the possibility of a value-free and objective social science (Schwandt, 1997), a postmodernist approach remains very much attuned to the political nature of our research methods and to the underlying social, cultural and political discourses that guide our social inquiry processes. Postmodernism questions the ‘universalizing presumptions’ (Harvey, 1989) of our research methods and helps in decentering the cultural authority of our westernized methods of social inquiry (Schwandt, 1996). In disputing the ‘apolitical illusion’ (Cannella and Lincoln, 2004) and belief in objective, neutral and universal social inquiry methods, a postmodernist perspective provides a way for us to look at how we create truth, rather than merely providing another method that serves to reveal truth (Everitt, 1996).

Questioning the taken-for-granted assumptions about a neutral, objective and detached social science has significant methodological implications (Agger, 1991). As Smith (1990) points out, ‘objectified forms of knowledge structure the relation between knower and known’ (p. 63), shifting the construction of knowledge to somewhere outside of the locally situated context and social relations. Postmodernism thus provides researchers with the discursive space in which to seriously challenge dominant frameworks for understanding how knowledge is constructed and maintained in our society. Foucault (1980) uses the concept of geneology to describe how dominant discourses structure reality, shape and normalize personal identities, and regulate society. By returning more traditionally subjugated knowledge into his concept of geneology, Foucault means to disturb and interrupt the more accepted and dominant forms of knowledge. As Best and Kellner (1997) argue, Foucault uses geneology to ‘liberate suppressed voices and struggles in history from the dominant narratives that reduce them to silence’ (p. 273). As Foucault (1980) clarifies, ‘there is not one but many silences, and they are an integral part of the strategies that underlie and permeate discourses’ (p. 27). It is subjugated knowledge, the silenced voices of the least powerful that is concealed and that must be raised above the historical din of the more dominant voices in our society.

A postmodern frame helps capture the plurality and complexity of research participant voices, as well as helping evaluators better understand their own embeddedness in the research process. Moreover, and most importantly, abandoning notions of objectivity also enables a critical reflection on self as researcher, on what Guba and Lincoln (1981) refer to as the ‘human as instrument’ (p. 210), and on the multiple, fluid and changing roles that researchers and evaluators assume in the field (Adler and Adler, 1987). As Dwyer and Buckle (2005) point out, ‘postmodernism emphasizes the importance of understanding the researcher’s context (gender, class, ethnicity, etc.) as part of narrative interpretation’ (p. 55). From this perspective, research is very much considered a dialogical process that involves the interaction of both the researcher and participants who together co-construct the narrative text (Fontana, 2002; Packwood and Sikes, 1996). Postmodernism thus moves research towards a less reductionist and more pluralistic conception, towards participatory methods of social inquiry (Lincoln, 1994; Reason, 1988), and towards the recognition that knowledge can only be defined by the plurality of perspectives and the multiplicity of subject positions (Agger, 1991). As Howe (1994) points out, ‘the full participation of all those involved in decisions about what is going on and what should be done is the only way to determine non-oppressive, culturally pertinent truths and working, practical judgments’ (p. 525). In acknowledging the multiplicity of voices, postmodernism thus helps to make visible or uncover the many, contradictory and varied social, political and cultural forces that guide our social inquiry methods and practices (Giroux, 2005).

Re-presentation within a postmodernist frame

In program evaluation (and other social inquiry practices), data is collected using specific methods that are aligned with specific methodological approaches, and we analyze and report on our findings to the broad stakeholder community. How we represent our findings is rarely questioned, so long as they are clearly written and well-articulated and make sense to the program community. In fact, how we report on our findings is thought to follow from the methods that we use to collect our data; methods and reporting are considered inseparable. A postmodernist perspective on social inquiry questions this assumption (Kincheloe, 1997), opening up the possibility of taking a closer look at how we represent our data and our findings, how we represent others’ voices, and how we locate ourselves (how we are represented) within the text (as either outsiders or insiders) (Merriam et al., 2001). Postmodernism enables us to see ourselves within the text, writing from a particular context, reflexively (Richardson, 1994). As Denzin (1998) explains, ‘representation is always self-presentation’ (p. 319).

The choices that we make as we write – how and for whom (whose voice(s) will be heard? How will their story be told?), is reflective of the issue that all knowledge is socially constructed (Foucault, 1980; Richardson, 1994). As Richardson (1994) explains, ‘writing is not simply a true representation of an objective reality, out there, waiting to be seen . . . writing creates a particular view of reality’ (p. 9). The language that we use to represent our findings thus helps to shape the reality and the perception of reality that we attempt to describe (Said, 2003 [1979]), underscoring the point that meaning is itself produced, rather than merely reflected within language (Weedon, 1997).

How we represent our findings and how we understand the voices of others and ourselves in the text, while posing fundamental questions in any qualitative social science inquiry, becomes even more critical in cross-cultural research and evaluation. How do we represent cultures other than our own? How do we understand them and ensure that our re-presentation resonates? What methodological approach might we adopt and how might we present our findings to a diverse audience? Are the methods that we use to collect our data valid in the cultural context within which we conduct our research? In the following section, I begin to explore what it means to understand others and ourselves within the cross-cultural research context, problematizing the nature of identity and self-identity, as we begin to make sense of the multiple and often conflicting categories of being that we confront both within ourselves and among our research participants in the field.

Three ‘problematics’ identified in conducting cross-cultural evaluation

My initial research highlighted the many inter-related cultural dimensions that surface in interactions between evaluators and stakeholders in cross-cultural evaluation and program settings, underlying which are a number of dilemmas that persist. These dilemmas, described in institutional ethnography by Smith (1987) as ‘problematics’, are partially expressed within the local context, but historically and politically mediated through institutional and social forces that originate outside of the parameters of the local setting. A problematic, according to Smith (2005), ‘sets out a project of research’ (p. 227) that focuses on understanding how people’s everyday world of experience is externally structured and ‘put together by relations that extend vastly beyond the everyday’ (p. 1). Through analysis of interview transcripts and a comprehensive review of the literature, I provide three specific problematics that I believe continue to create ongoing dilemmas in cross-cultural program and evaluation contexts.

The first problematic is related to the fact that one can be both an insider and an outsider (at one and the same time), further obfuscating initial assumptions that we might make based on gender, ethnicity, education, age, etc. Do we pair evaluators and research participants along gender, ethnic or other identity lines? How do we determine which parameters and definitions of identity are important within a specific context? While this continues to be a contested area of research and practice, my research suggests that as identity is a multifaceted and fluid concept, it becomes increasingly difficult to accurately determine insider or outsider status based on simple, singular dimensional notions of cultural identity.

The second problematic is concerned with the role evaluators play in cross-cultural contexts, as their identities cannot be divorced from the politics of the evaluation and context (the ‘politics of place’), nor from their political/ethical stance in terms of process, results and uses of evaluation (Weiss, 1993). How do evaluators make sense of the diverse roles and role expectations within cross-cultural contexts? How do they balance the multiple and potentially conflictual role expectations among the diverse stakeholder groups? Evaluators have choices to make, many of which occur outside of the program and evaluation context, choices that concern their ‘performance’ of evaluation, and that ultimately reflect upon their role and identity as researchers.

The third problematic represents the myriad political, social and cultural implications behind the methodological selections and choice of methods, and the adoption of collaborative approaches as a way to bridge the cultural chasm. Can collaborative methodologies provide a context to bridge the space between evaluators and stakeholders? Can these methodologies mitigate issues of power between evaluators and stakeholders (or between stakeholders themselves)? While collaborative approaches to evaluation may help bring all stakeholders to the table and may ameliorate some power differentials, challenges of creating meaningful participation in the context of evaluation in cross-cultural contexts persist.

Problematic one: the multiple faces of identity

One of the challenges in conducting evaluations in diverse community contexts is finding evaluators or researchers who have knowledge of the community and its culture, language and history. The notion of identity, of who we are and of how we are positioned and categorized, is of fundamental concern in cross-cultural research, as we make assumptions about self and others that guide our decisions as evaluators in the field. The challenge thus becomes determining what is salient within a specific context, as it is not always self-evident who is in or who is out, thus moving the emphasis to developing relationships to provide needed contextual and cultural clarity about what is relevant and what is not, in a particular setting.

This problematic represents the ecological and special connection of self and others to social, political, historical and cultural locations, as we move from being outsiders to insiders (or vice versa), and as our proximity to the community and to ourselves shifts and changes over time. The awareness of self as cultural being and as evaluator leads to an internal dialogue about identity and about the social categories that are assumed throughout the research encounter. Identities are not fixed but mutable and transformed through inter-relationships and interactions with others. There is no one category or single all-encompassing label that can capture the essence of participants and researchers, particularly if we consider the notion of identity as a ‘production’ that is in a constant state of change, being transformed within the specific historical and cultural discourses of the time (Hall, 1990), and through our social interactions with others. Identity is understood, not as a fixed, static essence, but a positioning (Hall, 1990), a situational performance (Goffman, 1959) that is never complete, leading to a multi-dimensional notion of identity that is created and recreated through discursive practices (Hall, 1996). As Hall (1990) describes:

Cultural identities come from somewhere, have histories. But like everything which is historical, they [identities] undergo constant transformation. Far from being eternally fixed in some essentialized past, they are subject to the continuous ‘play’ of history, culture and power. Far from being grounded in a mere ‘recovery’ of the past, which is waiting to be found, and which, when found, will secure our sense of ourselves into eternity, identities are the name we give to the different ways we are positioned by, and position ourselves within, the narratives of the past. (p. 70)

These overlapping and intertwining notions of identity thus occur across cultural, class, gender and racial divides, and across political, economic and social histories that are saturated with unequal status, power and privilege. The many points of inclusion and exclusion, of being a part of and being outside, underscores the very complexity of our relationships with self and others in the research encounter. Relationships between evaluators and the stakeholder community, between notions of personal identity and the identity of others, thus influence the opportunities and limitations within the evaluative context, circumscribing the program and the co/construction of knowledge as the evaluation unfolds (Abma, 2006).

My research underscored the point that intersections of identity were particularly marked for those who considered themselves as both cultural insiders and academic or privileged outsiders (Brayboy, 2000; Thapar-Bjorkert, 1999), both uneasy dualities considering the multiplicity of roles and identities potentially enacted throughout the evaluative and research encounter. Rosaldo (1989) refers to this duality as one of ‘danc[ing] on the edge of a paradox’ (p. 180), simultaneously living positions both of colonized and colonizer (Brayboy, 2000). The insider/outsider binary thus underscores the difficulty of ascribing fixed notions of identity, as it neglects the interactive process in the creation of identities (Naples, 1997) while it belies the complexity of the cross-cultural encounter. The search for the right cultural informants and the use of evaluators from similar ethnic backgrounds as program participants may thus prove problematic, as it may reify cultural traits by oversimplifying the factors that create identity (Schick, 2002). At the same time, it does not account for the heterogeneity among cultural members nor intra-cultural variables, confusing issues of power, knowledge and exclusion with finding the right informants (Schick, 2002). The interactions and inter-relationships between evaluators and stakeholders thus raises important questions about the construction of identity, and about the multiplicity of often overlapping and intersecting identities (tied to concepts of gender, race or class) that abound in the cross-cultural program context.

Globalization ensures that cultures are no longer insular (Bandura, 2002), as they are constantly being constructed and remade through processes of migration and acculturation (Guarnaccia and Rodriguez, 1996). Identities are thus created discursively, constructed in specific historical and political contexts (Hall, 1996), in relation to others (McRobbie, 1992), and through continuous struggle, as they are constructed and reconstructed over time. Given the complexity of culture and the formation of identity, it thus becomes necessary to challenge the epistemological status of our definitions of what constitutes a social category as well as the terms of inclusion (Rogoff, 2003; Schick, 2002), making decisions concerning who’s in and who’s out, as a way of bridging the cultural chasm. As Moharty (1987, as cited in Pickering, 2008) explains, ‘the experience of being a woman can create an illusory unity, for it is not the experience of being a woman, but the meanings attached to gender, race, class, and age at various moments that is of strategic significance’ (p. 39). The sense of ‘illusory unity’ based on shifting categories of identity, of ascribed status (Merton, 1972) whether of woman, ethnicity, class or age, can lead to a false sense of privileged access to social and cultural truths (Merton, 1972). While this underscores the complexity of the insider/outsider dilemma (see Brayboy, 2000; Bridges, 2001; Howarth, 2002), it also raises key epistemological questions about the social construction of knowledge, assertions of truth, and about who has privileged access to, and has the right to speak, for others. Thus, while the complexity of identity and the philosophical and empirical problems associated with creating fixed categories in terms of who is in and who is out are significant, it fails to address the significance of research relationships (and ‘interactive roles’) and the influence of community and program context in shaping research outcomes (Schick, 2002). As the problematic of perspective and location of self and others implies, it is our coming together as evaluators and stakeholders that creates the very possibility of change.

Problematic two: the role of the evaluator amidst situational and cultural complexity

The evaluator’s role in the cross-cultural setting is contextual, dynamic, multiple, political, and engaged, helping to shape relationships and build understanding among stakeholders (including both program sponsors and communities) as the evaluation unfolds. My findings suggest that understanding of self and others provides contextual and cultural grounding, as the multiple roles we assume also reflects the dynamic and multifaceted identities that we play in the field. The challenge thus becomes how we come to understand and enact our roles as evaluators in cross-cultural contexts, and with whether our roles capture the situational and cultural complexity with which the program and evaluation context demands.

While there is tremendous conceptual variation in terminology and in the use of the concept across multiple applications and fields, the sociological literature on role theory (Bailey and Yost, 2001; Outhwaite, 2008) provides some clarity for understanding the evaluator role(s) in diverse community contexts. The literature focuses on roles from a structural perspective, envisioning it as an orchestration, a process, or phenomenologically in terms of role dynamics (Outhwaite, 2008). Roles can be understood at the individual level as an enactment, a dramaturgical construction, as well as from a social systems perspective, where roles are tied to the historical, social, political and cultural context thus embodying the values of the social system (Bailey and Yost, 2001). The notion of role, much like identity, is also a very fluid complex concept that is both socially and personally expressed, occupying an ‘in-between-ness’ (Ibrahim, 2008), and a sense of movement and confusion where we can be at once connected and disconnected from self and others. As Goffman (1959) vividly portrays, ‘the image that emerges of the individual is that of a juggler and synthesizer, an accommodator and appeaser, who fulfils one function while he is apparently engaged in another’ (p. 40).

Symbolic interactionism takes a middle position, envisioning roles in reciprocal relationship between the individual and society (Bailey and Yost, 2002), not as static entities, but situationally complex, emerging and evolving in relationship to others and to society (Turner, 2002). From Goffman’s (1959) perspective, there is a complex relationship between the roles that we perform in interaction with others, and in our construction of identity, as both persons and roles are dependent upon social definition in equal measure. While both role and identity can be understood as broadly relational and ecologically connected constructs, with role the emphasis is on the performative and enactive, rather than on the social categories that we create.

This characterization of role is thus very much attuned to social constructivist notions, as the roles evaluators assume in cross-cultural contexts imply a different understanding of evaluation (of what it is and what it can be), from that of a disinterested social science to a more engaged method of social inquiry (Schwandt, 2002). This highly contextualized and relational approach to evaluator role is also reflective of a new conceptual understanding of evaluation (Abma et al., 2001; King and Stevahn, 2002), from a social science defined by neutrality, objectivity and dispassionate observation, to one of engagement and richly textured and interpretive inter-relationships (Ryan and Schwandt, 2002).

Some researchers abandon the traditional conception of role altogether, as they argue that it fails to accurately capture the range of personal and political relationships that are embedded in evaluation and that are considered paramount in the cross-cultural evaluation and program context (King and Stavahn, 2002). Mathison (1994) describes a shift in the role of the evaluator, from expert to one of collaborative partnership, emphasizing the ongoing, embedded and relational aspect of role. For Greene (2000), the influence of context and multiple relationships on our roles (as evaluators in the field) is conceptualized as ‘evaluator-as-engaged person’, a stance that reflects how we are present in our work, our engagement with moral and ethical complexity, as well as our engagement from the perspective of various positions and locations within society (where we come from). Schwandt (2002) also sees role as a relational concept, but he draws a distinction between role as something that is part of our core, stable self, and as something that can be put on and taken off at will, and role as a praxis-oriented concept that is discursively constructed, tied to notions of identity, and continuously enacted and re-enacted in practice. Abma and Widdershoven (2008) also note three distinctive evaluator roles that include evaluator as social science expert and judge (signifying relational distance), evaluator as provocateur who challenges societal power and social relations, and evaluator as someone who understands the inter-relational aspect of evaluation as fundamental to the very process of evaluation itself. This latter view is based on a hermeneutic and constructivist perspective that, as Abma and Widdershoven (2008) explain, is premised on the belief that ‘understanding our socially constructed world can only be generated by developing a relation and dialogue with and between the inhabitants of this world’ (p. 211).

The role of the evaluator is thus characterized as situationally and contextually fluid and complex, as well as relationally, interactionally and discursively enacted (through our relations to self and others), within the evaluation and program context. Conceptualizing the evaluator role from a relational, socially constructed and performative perspective, recasts relationships within the cross-cultural evaluative context as an asset rather than as something that needs to be overcome (Hopson, 2002), and brings the focus to an understanding of diverse stakeholder needs, interests and program goals. The point is that we bring who we are to the evaluation, including our values, preferences, aspirations, biases, and what I have referred to previously as our identities, from the questions we pose, to the stakeholders we include or exclude, and to what we envision evaluation to be. The role of the evaluator, much like identity, is thus deeply embedded and shaped by the local program and community context, as well as by the broader social, historical and political forces that influence the local program setting (Hopson, 2003). The traditional characterization of role thus underplays the dynamic aspect of our connections with others and with ourselves (Schwandt, 2002), both critical components to understanding within the cross-cultural program and evaluation context.

Problematic three: dissonance despite methodological pluralism

Cross-cultural evaluation is an explicit approach to evaluation that considers culture a key consideration in the evaluation of programs, leading to the search for methodological practices that are commensurate with the culture, context and values of the program community. One of the key assumptions behind this approach is the idea that culture is an integral part of the context of evaluation, not only in terms of the program setting, but also in terms of program design, and in the methods and approaches that evaluators adopt for use in their work (SenGupta et al., 2004). My findings suggest that in an effort to conduct culturally responsive evaluation in culturally distinct communities, evaluation is seen as a form of collaboration and partnership between evaluators and community members, leading to an increased understanding of unique socio-cultural characteristics, processes and perspectives (Chouinard and Cousins, 2009). While some of the challenges result from the dynamics of the relationship between evaluators and stakeholders (e.g. power differentials, community and community member voice), other challenges relate to the political and social history of the community vis-à-vis the dominant society, as well as concomitant levels of trust and respect. Still other challenges relate to the fact that evaluation itself represents a form of cultural authority (House, 1993), that is deeply implicated and influenced by the political, social, historical and economic context in which it occurs, all of which underscores the need to critically look at the research methodologies that are adopted and practiced in the field, as well as the knowledge that is generated through the collection, analysis and dissemination of findings.

This problematic thus represents the political, social and cultural implications behind our methodologies, as evaluators and stakeholders together construct and co-construct the knowledge that is created through the process of evaluation. While the participatory approach is not without its challenges in cross-cultural contexts, there are three primary benefits cited for its use in multiethnic and multicultural settings: 1) it increases the validity and relevance of research findings; 2) it builds the cultural competency of evaluators; and 3) it builds local capacity, community knowledge and empowerment. Despite these benefits, the co-construction of knowledge and meaning between diverse stakeholders and evaluators is not uncontested, as one cannot discount the imbalance of power between evaluators and stakeholders (House and Howe, 2000), particularly in communities with a history of exploitation and marginalization. While collaborative approaches may create a more inclusive and participatory context, and may in fact help mitigate some power differentials, merely inviting everyone to the table is not enough.

While evaluators who work in cross-cultural program settings acknowledge the challenges of conducting evaluations and building collaborative relationships amidst unequal power and status, my research findings suggest that this fact remains under-reported in both the literature and empirical findings. As a relational and productive concept (Foucault, 1979), power informs both the relationships between evaluators and stakeholders, as well as the broader socio-political level, what Gaventa and Cornwall (2006) refer to respectively as the ‘micro-political relationships of power’ and the ‘macro-political relationships of power’ (p. 73). Power is not conceptualized as something that is possessed by any one individual, state or higher authority, but as something that circulates and structures knowledge and its social constructions (Foucault, 1980). Power can thus be conceptualized as a ‘network of social boundaries’ (Hayward, 1998: 2), that includes laws, norms, institutional structures, and identities, and that delimit what can and cannot be done. This relational view of power draws attention to how relations of power (and powerlessness) are socially and politically shaped, and how they influence relationships between evaluators and stakeholders in the field.

The knowledge that emerges out of this collaborative and fundamentally social process involves multiple, complex and conflicting social, contextual, political, institutional and cultural factors, as relations of power continue to define the parameters of place and possibility. The use of collaborative methodologies and culturally resonant methods cannot mask evaluation’s inherent cultural authority (House, 1993), nor obscure its power to define the parameters of what constitutes legitimate discourse and knowledge in the social sciences (Reagan, 1996). Reliance on collaborative methodologies, through the use of stakeholder agreement or terms of reference (for example), is thus insufficient, particularly if one considers the broader social, cultural and political context within which our methodologies are embedded. Consider, for example, over 2000 years of Western civilization, the Western canon. That is what evaluators carry in their bags, in their evaluation toolkit.

It thus becomes essential that evaluators themselves develop cultural fluency, not only about themselves and about the program and the people being evaluated, but about the wider cultural systems within which we are all enmeshed. Evaluators, for example, need to see themselves not only as ‘situated in the action of [their] research’ (Rupp, 1983, as cited in Anderson, 1993), but also as purveyors of Westernized inquiry methodologies that might not advance understanding of diverse cultural lives and experiences. As conscientious social researchers working in cross-cultural settings, it is important to acknowledge that research and evaluation methods continue to reflect Western European perspectives (Cannella and Lincoln, 2004; Conner, 1985), raising the question of whether these methods can ever accurately reflect and describe the experiences of the program communities. According to Smith (1999), Westernized research methods ‘are underpinned by a cultural system of classification and representation, by views about human nature, human morality and virtue, by conceptions of space and time, by conceptions about gender and race’ (Smith, 1999: 44), all of which serves to potentially misrepresent the very communities that evaluators and researchers seek to understand.

Concluding thoughts

In this article, I have used a critical theoretical perspective in which to explore the dynamics of what I have termed ‘problematics’ in cross cultural evaluation. This critical perspective has enabled me to re-examine the methodological approaches that we use, and to better appreciate the complex and often defining social, cultural and political phenomenon involved in cross-cultural research and evaluation. While each of the three problematics that I identified is distinct, there is much conceptual overlap, as each problematic illustrates the deeply re-presentational issues that are potentially (and often) raised as we conduct evaluations in culturally complex contexts. The first problematic (the many faces of identity), underscores the complexity of the relationship between self and others in the research encounter, as intersections of our identities (and others) are given expression depending upon the contingencies of presenting contexts. This problematic brings the focus to how we as evaluators locate ourselves (whether we position ourselves as insiders or outsiders) and on how that positioning influences our relationship to the text and to the program community. The second problematic (the role of the evaluator amidst situational and cultural complexity), shifts the focus to evaluator roles that are discursively and socially constructed within the demands and parameters of the research and evaluation context, as evaluators focus on building relationships to bridge what is often a cultural divide. While conceptually tied to the notion of identity, this construction of evaluator role repositions our understanding of evaluation (what it is, who it serves) and repositions the evaluator as relationally engaged with the stakeholder community (whose voices we include, whose voices we exclude). The third problematic (dissonance despite methodological pluralism), highlights the socio-political implications of our methodological choices, as evaluators adopt collaborative approaches to mitigate issues of power and privilege so often pervasive in the cross-cultural research encounter. This problematic brings the focus directly and critically to the cultural validity of our method choices and to the challenges involved in representing the diversity of voices and perspectives despite the use of collaborative approaches.

These three problematics raise what I term ‘enduring issues of practice’, defined as underlying areas of practice that generate perplexing questions requiring further consideration and ongoing research. First, we discuss categories, identities and roles as fluid and contextualized constructions, yet they do have a very concrete reality inside of our institutions. The question then becomes, how do we bridge these seemingly theoretical and yet very real cultural territories, in a meaningful way? Second, the evaluator is deeply implicated in matters of methodology and epistemology, as they ultimately shape what they see, what they hear, and the information that they gather. The question thus becomes, where are evaluators located in the notion of ‘the master’s tools will never dismantle the master’s house’ (Lorde, 1984). Third, despite the need for culturally resonant methods particularly in diverse communities, the demand for impartiality and objectivity persists. The question thus becomes, how do evaluators balance the work that they do cross-culturally, with the demands of accountability and evidence-based practice that is so prevalent today? And finally, power is a persistent theme throughout all of the literature on culturally responsive research and evaluation. Notably, studies conclude with enduring power differentials between researchers or evaluators and the program community. It is important to note, however, that we adopt these culturally specific methods precisely because these power differentials exist. The question is thus that if we start with power as a central construct (rather than end with it), what will it allow us to see?

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Jill Anne Chouinard, PhD, is an Assistant Professor in the Department of Educational Research Methodology at the University of North Carolina at Greensboro. Her main interests are in cross-cultural evaluation and the use of participatory approaches as a leverage for social change. She received her PhD in organizational studies and evaluation in 2010 from the University of Ottawa.

References

Abma

(2006) The social relations of evaluation. In: Greene

Mark

(eds), Handbook of Evaluation: Policies, Programs and Practices. Thousand Oaks, CA: SAGE, 184–99.

Abma

Widdershoven

GAM

(2008) Evaluation and/as social relation. Evaluation 14(2): 209–25.

Abma

Greene

Karlson

Ryan

Schwandt

Widdershoven

(2001) Dialogue on dialogue. Evaluation 7(2): 164–80.

Adler

(1987) Membership Roles in the Field. Newbury Park, CA: SAGE.

Agger

(1991) Critical theory, poststructuralism, postmodernism: their sociological relevance. Annual Review of Sociology 17: 105–31.

Bailey

Yost

(2001) Role theory: foundations, extensions, and application. In: Encyclopedia of Sociology, vol. 4, 2nd edn. New York: Macmillan Reference USA, 2420–5.

Bandura

(2002) Social cognitive theory in cultural context. Applied Psychology: An International Review 51(2): 269–90.

Bauman

(1992) Intimations of Postmodernity. London: Routledge.

Best

Kellner

(1997) The Postmodern Turn. New York: The Guilford Press.

10.

Brayboy

(2000) The Indian and the researcher: tales from the field. Qualitative Studies in Education 13(4): 415–26.

11.

Bridges

(2001) The ethics of outsider research. Journal of Philosophy of Education 35(3): 371–86.

12.

Cannella

Lincoln

(2004) Epilogue: claiming a critical public social science. Reconceptualizing and redeploying research. Qualitative Inquiry 10(2): 298–309.

13.

Chouinard

Cousins

(2009) A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation 30(4): 457–94.

14.

Cockburn

(1998) The Space Between Us: Negotiating Gender and National Identities in Conflict. London: Zed Books.

15.

Collins

(1998) It’s all in the family: intersections of gender, race, and nation. Hypatta 13(3): 62–82.

16.

Conner

(1985) International and domestic evaluation: comparisons and insights. New Directions for Evaluation 25: 19–28.

17.

Danziger

(1990) Constructing the Subject: Historical Origins of Psychological Research. New York: Cambridge University Press.

18.

Davis

(2008) Intersectionality as buzzword: A sociology of science perspective on what makes a feminist theory successful. Feminist Theory 9(1): 67–85.

19.

Denzin

(1998) The art and politics of interpretation. In: Denzin

Lincoln

(eds), Collecting and Interpreting Qualitative Materials. Thousand Oaks, CA: SAGE, 313–44.

20.

Dwyer

Buckle

(2009) The space between: on being an insider-outsider in qualitative research. International Journal of Qualitative Methods 8(1): 54–63.

21.

Everitt

(1996) Developing critical evaluation. Evaluation 2(2): 173–88.

22.

Fine

(1994) Working the hyphens: reinventing the self and other in qualitative research. In: Denzin

Lincoln

(eds), Handbook of Qualitative Research. Newbury Park, CA: SAGE.

23.

Fontana

(2002) Postmodern trends in interviewing. In: Gubrium

Holstein

(eds), Handbook of Interview Research. Thousand Oaks, CA: SAGE, 161–75.

24.

Foucault

(1972) The Archeology of Knowledge and the Discourse on Language. New York: Pantheon Books.

25.

Foucault

(1979) ‘What is an author?’ Screen 20(1): 13–33.

26.

Foucault

(1980) The History of Sexuality, vol. 1. New York: Vintage.

27.

Gaventa

Cornwall

(2006) Power and knowledge. In: Reason

Bradbury

(eds), Handbook of Action Research: The Concise Paperback Edition. Thousand Oaks, CA: SAGE, 71–82.

28.

Giroux

(2005) Border Crossings: Cultural Workers and the Politics of Education, 2nd edn. New York: Routledge.

29.

Goffman

(1959) The Presentation of Self in Everyday Life. Garden City, NJ: Doubleday, Anchor Books.

30.

Greene

(2000) Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation 85: 27–38.

31.

Gregory

(2000) Problematizing participation: a critical review of approaches to participation in evaluation theory. Evaluation 6(2): 179–99.

32.

Guarnaccia

Rodriguez

(1996) Concepts of culture and their role in the development of culturally competent mental health services. Hispanic Journal of Behavioral Sciences 18(4): 419–43.

33.

Guba

Lincoln

(1981) Effective Evaluation. San Francisco, CA: Jossey-Bass.

34.

Hall

(1990) Cultural identity and diaspora. In: Rutherford

(ed.), Identity: Community, Culture, Difference. London: Lawrence & Wishart, 222–37.

35.

Hall

(1992) The west and the rest: discourse and power. In: Hall

Gieben

(eds), Formations of Modernity. Cambridge: Polity Press, 275–331.

36.

Hall

(1996) Introduction: who needs identity? In: Hall

Du Gay

(eds), Questions of Cultural Identity. Thousand Oaks, CA: SAGE, 1–17.

37.

Harvey

(1989) The Condition of Postmodernity. Malden, MA: Blackwell.

38.

Hayward

(1998) De-facing power. Polity 31(1): 1–22.

39.

Hopson

(2001) Global and local conversations on culture, diversity, and social justice in evaluation: issues to consider in a 9?11 era. American Journal of Evaluation 22(3): 375–80.

40.

Hopson

(2002) Making (more) room at the evaluation table for ethnography: contributions to the responsive constructivist generation. In: Ryan

Schwandt

(eds), Exploring Evaluator Role and Identity. Greenwich, CT: Information Age Publishing, 37–56.

41.

Hopson

R K

(2003) Overview of Multicultural and Culturally Competent Program Evaluation: Issues, Challenges and Opportunities. Social Policy Research Associates, The California Endowment.

42.

House

(1993) Professional Evaluation: Social Impact and Political Consequences. Thousand Oaks, CA: SAGE.

43.

House

Howe

(2000) Deliberative democratic evaluation. New Directions for Evaluation 85: 3–12.

44.

Howarth

(2002) Using the theory of social representations to explore difference in the research relationship. Qualitative Research 2(1): 21–34.

45.

Howe

(1994) Modernity, postmodernity and social work. British Journal of Social Work 24: 513–32.

46.

Ibrahim

(2008) The new flaneur: subaltern cultural studies, African youth in Canada and the semiology of in-betweenness. Cultural Studies 22(2): 234–53.

47.

Kincheloe

(1997) Fiction formulas: critical constructivism and the representation of reality. In: Denzin

Lincoln

(eds), Representation and the Test: Re-framing the Narrative Voice. Albany, NY: State University of New York Press, 58–79.

48.

Kincheloe

McLaren

(1998) Rethinking critical theory and qualitative research. In: Denzin

Lincoln

(eds), The Landscape of Qualitative Research: Theories and Issues. Thousand Oaks, CA: SAGE.

49.

King

Stevahn

(2002) Three frameworks for considering evaluator role. In: Ryan

Schwandt

(eds), Exploring Evaluator Role and Identity. Greenwich, CT: Information Age Publishing, 1–16.

50.

Kushner

(2000) Personalizing Evaluation. London: SAGE.

51.

Lincoln

(1994) Tracks toward a postmodern politics of evaluation. Evaluation Practice 15(3): 299–309.

52.

Long

(1992) Introduction. In: Long

Long

(eds), Battlefields of Knowledge: The Interlocking of Theory and Practice in Social Research and Development. London: Routledge, 3–43.

53.

Lorde

(1984) Sister Outsider: Essays and Speeches. Berkeley, CA: The Crossing Press.

54.

Lyotard

(1979) The Postmodern Condition: A Report on Knowledge. Minneapolis, MN: The University of Minnesota Press.

55.

Mathison

(1994) Rethinking the evaluator role: partnerships between organizations and evaluators. Evaluation and Program Planning 17(3): 299–304.

56.

McCall

(2005) The complexity of intersectionality. Signs: Journal of Women in Culture and Society 30(3): 1771–800.

57.

McRobbie

(1992) Post-Marxism and cultural studies: a post-script. In: Grossberg

Nelson

Treicher

(eds), Cultural Studies. New York: Routledge, 719–30.

58.

Merriam

Lee

Kee

Ntseane

Muhamad

(2001) Power and postionality: negotiating insider/outsider status within and across cultures. International Journal of Lifelong Education 20(5): 405–16.

59.

Mertens

(1999) Inclusive evaluation: implications of transformative theory for evaluation. American Journal of Evaluation 20(1): 1–14.

60.

Merton

(1972) Insiders and outsiders: a chapter in the sociology of knowledge. The American Journal of Sociology 78(1): 9–47.

61.

Naples

(1997) A feminist revisiting of the insider/outsider debate. In: Hertz

(ed.), Reflexivity and Voice. Thousand Oaks, CA: SAGE, 70–94.

62.

Nelson-Barber

LaFrance

Trumbull

Aburto

(2005) Promoting culturally reliable and valid evaluation practice. In: Hood

Hopson

Frierson

(eds), The Role of Culture and Cultural Context: A Mandate for Inclusion, the Discovery of Truth, and Understanding in Evaluative Theory and Practice. Greenwich, CT: Information Age Publishing, 61–85.

63.

Packwood

Sikes

(1996) Adopting a postmodern approach to research. Qualitative Studies in Education 9(3): 335–45.

64.

Pachai

Bishop

(2006) Images of Our Past: Historic Black Nova Scotia. Halifax, NS: Nimbus Publishing Ltd.

65.

Pickering

(2008) Experience and the social world. In: Pickering

(ed.), Research Methods for Cultural Studies. Edinburgh: Edinburgh University Press, 17–31.

66.

Reagan

(1996) Non-Western Educational Traditions: Alternative Approaches to Educational Thought and Practice. Mahwah, NJ: Lawrence Erlbaum Associates.

67.

Reason

(1988) Human Inquiry in Action: Developments in New Paradigm Research. London: SAGE.

68.

Richardson

(1994) Writing: a method of inquiry. In: Denzin

Lincoln

(eds), Handbook of Qualitative Research. Thousand Oaks, CA: SAGE, 516–29.

69.

Rogoff

(2003) The Cultural Nature of Human Development. New York: Oxford University Press.

70.

Rosaldo

(1989) Culture and Truth: The Remaking of Social Analysis. Boston, MA: Beacon Press.

71.

Ryan

Schwandt

(2002) Introduction. In: Ryan

Scwandt

(eds), Exploring Evaluator Role and Identity. Greenwich, CT: Information Age Publishing.

72.

Said

(2003 [1979]) Orientalism (25th Anniversary Edition). New York: Vintage Books.

73.

Samuels

Ross-Sheriff

(2008) Identity, oppression, and power: feminisms and intersectionality theory. Affillia: Journal of Women and Social Work 23(1): 5–9.

74.

Scheurich

Young

(2002) Coloring epistemology: are our research epistemologies racially biased? In: Scheurich

(ed.), Anti-Racist Scholarship: An Advocacy, Albany, NY: State University of New York Press.

75.

Schick

(2002) When the subject is difference: conditions of voice in policy-oriented qualitative research. Qualitative Inquiry 8(5): 632–51.

76.

Schwandt

(1996) Farewell to criteriology. Qualitative Inquiry 2(1): 58–72.

77.

Schwandt

(1997) Reading the ‘problem of evaluation’ in social inquiry. Qualitative Inquiry 3(1): 4–25.

78.

Schwandt

(2002) Traversing the terrain of role, identity and self. In: Ryan

Schwandt

(eds), Exploring Evaluator Role and Identity. Greenwich, CT: Information Age Publishing, 193–207.

79.

Senese

(2005) The PENAL project: program evaluation and Native American liability. In: Hood

Hopson

Frierson

80.

SenGupta

Hopson

Thompson-Robinson

(2004) Cultural competence in evaluation: an overview. New Directions for Evaluation 102: 5–19.

81.

Sirin

Fine

(2008) Muslim American Youth: Understanding Hyphenated Identities through Multiple Methods. New York: New York University Press.

82.

Smith

(1987) The Everyday World as Problematic: A Sociology for People. Lantham, MD: Altamira Press.

83.

Smith

(1990) The Conceptual Practices of Power: A Feminist Sociology of Knowledge. Toronto: University of Toronto Press.

84.

Smith

(2005) Institutional Ethnography: A Sociology for People. Toronto: Altamira Press.

85.

Smith

(1999) Decolonizing Methodologies: Research and Indigenous Peoples. London: Zed Books.

86.

Stanfield

Jr (1999) Slipping through the front door: relevant social scientific evaluation in the people of color century. American Journal of Evaluation 20(3): 415–31.

87.

Sue

(1999) Counselling the Culturally Different: Theory and Practice, 3rd edn. New York: John Wiley & Sons, Inc.

88.

Thapar-Bjorkert (1999) Negotiating otherness: dilemmas for a non-western researcher in the Indian sub-continent. Journal of Gender Studies 8(1): 57–69.

89.

Thomas

(2004) Building a contextually responsive evaluation framework: lessons from working with urban school interventions. New Directions for Evaluation 101: 3–23.

90.

Turner

(2002) Role theory. In: Turner

(ed.) Handbook of Sociological Theory. New York: Springer.

91.

Wallerstein

Duran

(2003) The conceptual, historical, and practical roots of community-based participatory research and related participatory traditions. In: Minkler

Wallerstein

(eds), Community Based Participatory Research. San Francisco, CA: Jossey-Bass, 27–52.

92.

Weedon

(1997) Feminist Practice & Poststructuralist Theory, 2nd edn. Cambridge, MA: Blackwell.

93.

Weiss

(1993) Evaluation, 2nd edn. Upper Saddle River, NJ: Prentice-Hall Inc.