Navigating the awkward,challenging social context of evaluation

Introduction

Evaluators work in an awkward and challenging social context. The fear, resistance and anxiety that are associated with evaluation call for complex sensitivities and skills, in response to infinite combinations of stakeholders and circumstances. Evaluation interactions are demanding in terms of the social competencies required, as well as demanding on evaluator well-being. In brief, ‘One of the joys of doing evaluation is the unending opportunity to grow from challenging experiences’ (McDonald & Myrick, 2008, p. 343).

This article has three main parts. It begins by examining how qualitative and relational approaches have been positioned in the evaluation field over time. These have been increasingly acknowledged and now widely accepted as crucial to the practice of evaluation. In the second part, more recent literature is used to identify six particularly complex and demanding social competencies that are essential to on-the-ground evaluation practice. These competencies are in allaying fear and anxiety, establishing rapport, building and maintaining professional credibility, recognising tacit social dynamics, preventing and managing coercion attempts, and preventing and managing hostility.

The third part of the article explores what this all means for evaluator well-being. Research literature from the evaluation field and broader social sciences is used to identify issues and possible mitigating strategies. Difficulties around self-assessing competency levels, emotional labour and self-care, contending with a poor reputation and the limited research specific to these challenges are discussed. The article posits that practising evaluators should routinely reflect and take active steps to improve their social competencies and to maintain their own well-being.

In this article, the term social competencies is used to refer to the ‘skills, knowledge, experience, abilities, attributes and dispositions needed to successfully practice evaluation’ (Aotearoa New Zealand Evaluation Association [ANZEA], 2011, p. 12) that relate to social, interactive or interpersonal aspects of evaluation practice. The article focuses on six social competencies that have particular importance and complexity within evaluation practice. This list is not exhaustive and does not explicitly include broader concepts such as cultural responsiveness or written and verbal communication skills.

Looking back: qualitative and relational approaches in evaluation

Contextually, it is important to note that the field of evaluation emerged at the same time as the 20th-century positivist and non-positivist paradigm debates. It took some time and effort for social sciences and qualitative approaches to gain credibility in comparison to traditional scientific methods (Sarantakos, 2013). Across the social sciences, problems arose when work reflected certain values or politics, so many social scientists preferred to use quantitative methods to avoid allegations of bias (House, 1995). As evaluation expanded in the 1950s and 1960s from focusing on educational assessment to a wider range of government programmes and initiatives, there was an increased need for qualitative and interactive methods. This placed evaluation firmly on the social side of science, where controlled, gold-standard settings are impossible or unethical. This placement was not ideal for the evaluation field in terms of esteem and credibility: ‘the social sciences have long had an inferiority complex’ (Douglas, 2014, p. 162). Donaldson (2001) describes the persistence of unfavourable comparisons:

One rather common view by nonevaluator colleagues in related areas is that evaluation is at best a second rate scientific activity. Well controlled scientific inquiry is typically the gold standard for exemplary work in these settings. Evaluators take on messy ‘real world’ problems and questions that often require giving up scientific controls . . . To some, evaluation is a pseudoscience. (p. 358, emphasis in original)

Unlike most academic and scientific professions, training and accreditation in evaluation is considered optional and the reality of being ‘for hire’ can be seen to compromise credibility. Evaluation is a young field and still finding its place. As recently as 1996, Scriven wrote of ‘these early days while we’re trying to get the discipline conceptualised, validated, and credible’ (p. 161). In this context, it is unsurprising that interpersonal aspects of evaluation practice have been dismissed as ‘atheoretical common sense’ (Leviton, 2001, p. 6).

Early evaluators had focused on the theoretical side of evaluation and the development of methods and models (Greene, 2001; King & Stevahn, 2013; Leviton, 2001; Mark, 2002). Different approaches to data gathering and analysis were debated and refined, building a foundation of credibility on the rigour of various models. Theorists prioritised methodological matters over social and relational aspects, with what has been called an ‘optimistic’ (Chelimsky, 2012, p. 93) expectation that good work and good findings would naturally translate into action. Over time, practical evidence mounted that evaluation findings were largely ignored by those who were expected to act on them. The rise of use-oriented evaluation approaches in the 1970s ‘broaden[ed] the scope of evaluation to include the social’ (Abma, 2006, p. 196). Social and interpersonal factors were foregrounded in the development of responsive (Stake, 1975), utilisation-focused (Patton, 1978) and naturalistic (Guba & Lincoln, 1981) evaluation. Mark (2002) cites transformative and empowerment approaches as further evidence of the shift, over time, to privileging an evaluation’s intended user or purpose over fidelity to methods and models.

Many contemporary approaches tend to define themselves primarily in terms of the relationship of the evaluator (and evaluation) to others. Advocates of these approaches, as I read them, see the relationship with some relevant other(s) as the guiding star, and see methods and evaluation purpose as things to be brought into alignment to better serve that other. (p. 22, brackets and italics in original)

Since the 1980s, ‘interactive evaluation approaches have replaced distanced approaches’ (Donaldson et al., 2002, p. 262). The rise in participatory, culturally responsive and socially responsible evaluation discourse indicates a decisive move away from the positivist underpinnings of the traditional sciences. Personal, psychological, political and other human influences are now acknowledged as realities of practice (Donaldson et al., 2002; Grob, 2018; Morris, 2011): the application of an evaluation method or model in pure form cannot be any more than a theoretical notion.

In recent decades, evaluation literature has been exploring the interpersonal challenges and opportunities of the discipline. Discussion on the many ‘roles’ of the practising evaluator was widespread in 1990s and early 2000s, with laundry lists of traits, techniques and social value positions for consideration (Caracelli, 2000; Ryan & Schwandt, 2002; Skolits et al., 2009). Ethical challenges faced by practising evaluators were explored and debated (e.g., Morris, 2003, 2008). Relational methods were shown to be more effective in reaching marginalised populations (Mertens, 2001). Research showed that ‘trust’, ‘relationship[s]’ and ‘personalities’ of evaluators had significant impacts on process and use (Taut & Alkin, 2003, p. 217). Literature on cultural competence, cultural responsiveness and Indigenous methodologies in evaluation began to improve understandings about socially located ways of being, knowing and doing, and cement evaluators’ appreciation of the culturally contexted nature of all interactions. There has been development and then revision of taxonomies, standards and competency frameworks, which include nuanced interpersonal skills (American Evaluation Association [AEA], 2018; ANZEA, 2011; Australian Evaluation Society [AES], 2013; Stevahn et al., 2005). The quality of the evaluation process and findings have been strongly linked to the quality of interactions and engagement (Abma, 2006; King & Stevahn, 2013; Perrin, 2019; Taut & Alkin, 2003).

Evaluators need skills in building relationships, facilitating groups, managing conflict, walking political tight ropes, and effective interpersonal communications to capitalise on the importance of the personal factor. Technical skills and social science knowledge aren’t sufficient . . . People skills are critical. (Patton, 2012, p. 76)

Literature today acknowledges that interpersonal, social and cultural factors are embedded in the art and science of evaluation, and critical to effective practice: ‘Evaluations where evaluators hide behind their data and fail to engage with stakeholders are not likely to have much credibility or to lead to use’ (Perrin, 2019, p. 373). Research that closely examines social competencies is not commonplace (Abma, 2006), though practical guides that include evaluation-specific interaction techniques are emerging (e.g. Cousins, 2019; Goodyear et al., 2014; King & Stevahn, 2013; Rodríguez-Campos & Rincones-Gómez, 2012; Podems, 2018).

Six essential social competencies for evaluators

This section draws on the literature to identify and summarise six social competencies that are particularly important in evaluation practice. Within evaluation contexts, these competencies are especially complex and demanding. They are: allaying fear and anxiety, establishing rapport, building and maintaining professional credibility, recognising tacit social dynamics, preventing and managing coercion attempts, and preventing and managing hostility.

What does this mean for practising evaluators and their well-being?

The previous section outlined some of the ways in which evaluation practice occupies a particularly awkward and challenging social space due to the fear, resistance and anxiety that is associated with evaluation. In this context, professional interactions are abnormally high risk and demanding. This has implications for the sorts of social competencies required, as well as implications for evaluator well-being.

Practising evaluation using relational methods demands a ‘sixth sense’ (Stevahn & King, 2014, p. 142). Within a small sample of relevant literature, interactions are variously described as ‘delicate and challenging’ (Donaldson, 2001, p. 360), ‘enormously important’ (Abma, 2006, p. 187), ‘overwhelmingly complex’ (King & Stevahn, 2013, p. 10), ‘sophisticated’ (Warner, 2020, p. 432) and a ‘very difficult – and perhaps even impossible – juggling act’ (McKegg, 2003, p. 232). Evaluators must balance the dual messages: ‘Don’t worry, I come in peace’, and ‘. . . But I’m good at my job which admittedly might not be good for you’. While most social science research and qualitative methods involve relational practice, in an evaluation context a sense of impending risk or reward for stakeholders, whether real or perceived, can have significant impacts on authenticity and usefulness. For this reason, Scriven (1997, p. 484) and Kirkhart (1995, p. 4) both question the ‘validity’ of evaluation information gained through interactions.

The awkward social context also hinders evaluators’ ability to self-assess their social competencies. Practising evaluators need constructive feedback to do their jobs well, just like any other profession – and indeed the programmes and organisations they evaluate. The social context of evaluation makes this difficult. Positive or negative responses from stakeholders can easily be attributed to evaluation-related heightened emotion, rather than the evaluator’s social competence. To illustrate, consider forced laughter at a powerful manager’s joke, an exaggerated story from a nerve-wracked job interviewee or a deer-in-headlights response to an intimate question from a relative stranger. The more socially competent the evaluator, of course, the more they can notice discomfort or disingenuity, and create the conditions for authentic and useful responses from stakeholders. This is problematic for less-skilled evaluators: without the skills to note subtle evidence of unsuccessful social interactions, how can they know their skills need improvement?

Social skilfulness around friends or family is not a reliable measure of effectiveness, because in social and communication matters, context is key. ‘An individual might be a top-flight corporate negotiator but at the same time a dreadfully poor manager of conflict with her spouse and children’ (Berger, 2005, p. 434). Various self-assessment tools that include social aspects have been developed for the field, such as that put forward by Ghere et al. (2006). Self-assessment is also one of the intended functions of national competency frameworks (AEA, 2018; AES, 2013; ANZEA, 2011). When considering self-assessment in this area of practice, though, it should be remembered that social effectiveness is, by definition, located outside of the self. Evaluators are likely to find it difficult to measure and be confident in their performance.

Evaluation is often viewed negatively, which can impact well-being and is usually markedly out of sync with evaluators’ positive motives. Podems (2018) writes, ‘The reason that I am an evaluator is not that I enjoy judging things or being critical. I am an evaluator because I believe that evaluators have the potential to make the world a better place to live in’ (p. ix). Grob (2014) also notes that a significant proportion of evaluators enter the field ‘to make a difference’ (p. 393). Chouinard and Cousins (2021, p. 9) describe how the ‘virtues of care, empathy, compassion and benevolence speak to (and inspire) us as evaluators and methodologists’ (p. 9, brackets in original). In terms of its reputation as a discipline, though, evaluation is certainly not considered a ‘helping profession’, as Donaldson (2001) sets out:

I find that many relevant stakeholders who are nonevaluators view evaluation as an unpleasant and often threatening requirement typically forced upon them by outside influences . . . They often see evaluators as the uninformed enemy, using insensitive scientific methods to unfairly challenge their current claims, and to threaten the future of their efforts . . . (p. 356)

The body of literature on self-care is growing, ‘overwhelmingly aimed at those in the helping professions’ (Theidon, 2014, p. 2), in response to the particular stresses that come with daily emotional labour combined with a strong sense of social or civic responsibility. The situation for evaluators, where their practice comes with many of the helping professions’ challenges but without their recognition or esteem, can be ‘demoralising’ (Donaldson, 2001, p. 358). Evaluators are tasked with managing the disjunct between how they are seen by others and how they see themselves.

The burdens and opportunities of this task are irremediably ours . . . Telling the truth, especially to power, is a critically important function in a democratic society. It is why many of us came to evaluation in the first place, and it is the most meaningful and charismatic part of our work. (Chelimsky, 1998, p. 53)

Although relational practices in evaluation contexts are ‘mostly functional (versus personal)’ (Abma, 2006, p. 194, brackets in original), the personal impacts on evaluators can be significant. The toll of continually contending with threats, bribes, tears and appeals should not be underestimated. To do so while also pursuing positive and productive relationships with these stakeholders is particularly taxing. While specific references to the well-being impacts of evaluation practice are sparse, what is available affirms that evaluation practice is not comfortable or easy.

The evaluator’s role can be very stressful. The problems that we predictably encounter . . . impede evaluators’ own professional advancement and quality of life . . . what on earth possesses evaluators to believe they can handle these problems by themselves? (Leviton, 2001, p. 3)

Evaluators who share insights in this area often use rather evocative terms: for example, ‘too much to ask’ (McKegg, 2003, p. 234), ‘ripped to shreds’ (Podems, 2018, p. 211) and ‘being turned into a punch bag in a blame game’ (Njovu, 2020, p. 96). Navigating the social context of their role demands a lot. “It takes courage to refuse sponsors the answers they want to hear, to resist becoming a ‘member of the team,’ to challenge myths that everyone is comfortable with, to fight inappropriate intrusion…” (Chelimsky, 1997, p. 60, emphasis added).

In the wider context of the social sciences, there are calls for increased recognition of the impacts of research activities on researchers’ well-being (Dickson-Swift et al., 2007; Lee-Treweek & Linkogle, 2000; Theidon, 2014). Explorations into compassion fatigue and researchers’ need for self-care are increasingly common in fields like psychology, anthropology and health care (e.g., Johnson & Clarke, 2003; Stamm, 1999; Theidon, 2014). Dickson-Swift et al. (2007) interviewed 30 qualitative researchers in the health field about the stresses they encounter, finding that there were a number of negative impacts on their well-being, including difficulties with the level of rapport and feelings of guilt. The authors note that ethics processes are understandably more attentive to protecting participants than researchers, and that ‘the emotional nature of research work is undervalued within the university culture’ (Dickson-Swift et al., 2007, p. 345).

Although the body of research into the social and relational aspects of evaluation practice is growing, literature that is specifically focused on the personal impacts of evaluation practice is still unusual. The wider social research field, however, offers a range of strategies for preventing, recognising and responding to these challenges. Personal risk assessments, emotional regulation strategies, recovery time, deliberate pacing of research activities, motivation techniques and friend and family support networks are a few examples (Dickson-Swift et al., 2007; Lee-Treweek & Linkogle, 2000; Stamm, 1999; Theidon, 2014). Well represented in the evaluation context as well as in wider social research is the value of peer support networks with colleagues (Dickson-Swift et al., 2007; B. Johnson & Clarke, 2003; Leviton, 2001; Perrin, 2019 Podems, 2018). Leviton (2001) recommends peer sharing and mentoring to minimise negative impacts. Discussion about challenging experiences would mean experienced listeners could reassure an evaluator under stress:

‘No, you are not crazy or stupid, it is a bad situation, and here’s what you could do’ . . . all except the most experienced evaluators probably need this . . . The result [otherwise] can be self-blame, or the worst forms of cynicism about programmes (p. 10).

While it is generally agreed that novice evaluator training should focus more strongly on interpersonal skills (Davies & MacKay, 2014; Grob, 2018; Perrin, 2019; Skolits et al., 2009), this is unlikely to meet the demands of a career as a practising evaluator. King and Stevahn (2013) argue along these same lines, recommending the focused development of a range of skills in interactive evaluation practice. The complexities of evaluation work warrant ongoing training and development around social competencies. This, alongside purposeful maintenance of well-being, should be considered a necessity and a professional obligation. Even the most experienced and successful evaluators can be challenged by the ever-changing contexts of on-the-ground practice, as former AEA President Leviton (2014) attests:

Experts in evaluation, like experts in any field, work with context and draw from a larger repertoire of responses. They see different challenges and opportunities inherent in the situation. They can also smell trouble coming and sometimes can prevent it. That facility can only come with experience, and sometimes not even then. For example, I truly hope that I have seen all the varieties of evaluation context and can both anticipate and deal with them. My guess, however, is that there are still a few surprises in store for me. (p. 246)

Some recent studies have looked into the development and mastery of social competencies in practice: Warner (2020) examined the social skill-building of evaluation students, and Garcia and Stevahn (2020) analysed narratives and insights shared by experienced evaluators around interpersonal competence. A survey of the 2020 issues of The American Journal of Evaluation, the Evaluation Journal of Australasia and Evaluation Matters found very little discussion of social challenges (in nine of 57 papers, at least one sentence relating to social competency matters was evident), and just two brief mentions of the emotional or wellbeing impacts of these challenges on evaluators. Increased, focused study around the interpersonal realities and personal implications of evaluation-specific social competencies in practice would be useful for the field.

Conclusion

The awkward and challenging social context of evaluation means that social competencies are especially important – and especially taxing. This article has shown that there is increasing acknowledgement within the literature that evaluation interactions are both rewarding and risky.

Successful evaluation practice requires competencies in allaying fear and anxiety, establishing rapport, building and maintaining professional credibility, recognising tacit social dynamics, preventing and managing coercion attempts, and preventing and managing hostility. Practising evaluators should reflect on the unique demands of their role. Continual skill building in the six social competencies set out in this article is necessary, as is taking active steps to maintain well-being among the particular challenges that come with being a socially competent evaluator.