Abstract
Evaluation occurs in a context fraught with ethical issues. Evaluators are regularly faced with ethical tensions that are likely to influence the quality of their work. By developing an analytical model that categorizes evaluators along an altruistic-corporatist axis, we sought to understand which factors influence the ethical sensitivity of evaluators. To this end, utilizing a qualitative research design using vignettes, we exposed a dozen Canadian evaluators to ethically problematic situations to examine their ability to identify the ethical issues within the scenarios. The ensuing results allowed us to conclude that an evaluator’s ethical sensitivity is partially explained by the altruistic or corporatist nature of that evaluator; it also depends on other factors such as knowledge of the prescriptive norms in ethical matters, experience conducting evaluations, and the milieu and the working conditions within which the evaluators operate.
Introduction
Modern administrations are increasingly concerned with issues related to ethics and deontology (Goodyear, 2007). Part of this trend is marked by growing interest in evaluative procedures. Over recent years, professional associations (e.g. Canadian Evaluation Society, American Evaluation Association, African Evaluation Association, etc.) have not only made evaluation ethics a topic of discussion at annual conferences but have also drafted standards, guidelines and charters that are intended to shape evaluative practice in the field. Despite its roots in science, an evaluation process is never truly neutral (Eliadis, Furubo and Jacob, 2011). On the one hand, engaging in an evaluation may raise undue expectations from the parties involved. It may also bring elements to the fore that were not anticipated prior to the evaluation (making visible what was previously invisible), which can bring about negative consequences, or even have a perverse effect on the situation (Morell, 2005). Because an evaluative process can have considerable impact, it needs to be as credible, impartial, transparent and legitimate as possible. However, studies suggest that this process is not without its own ethical challenges (Newman and Brown, 1992; Sheinfeld and Lord, 1981). Several authors contend that the actions of evaluators are the greatest source of ethical problems during the evaluation process (Eastmond, 1998; Scheirer, 1998) as they are considered to have the greatest influence on evaluation practice and outcomes (Mbaïrewaye, 2011). Evaluators can easily modify evaluation design, manipulate data collection and even influence evaluation results so as to ensure that results please a third party or even support the evaluator’s own interests and beliefs (Smith, 2002: 201). Each step of the evaluation process (terms of reference elaboration, data collection, report drafting and results use) can potentially be prey to inappropriate behaviors (Church and Rogers, 2006; Mabry, 2004; Newman and Brown, 1996). In this context, ‘ethics is found in a preventive logic, i.e. it invites the subject to avoid conduct that might have negative consequences on himself or others’ (Boisvert et al., 2003: 43, trans.).
Despite growing interest in training and problem solving, and an overall increased awareness of the need to resolve ethical issues, structured and systematic research relating to ethical dilemmas faced by those involved in evaluation is quite rare. Of the few research projects addressing ethics in evaluation, two distinct project types have emerged. The first type focuses specifically on evaluator behavior when confronted with ethical questions (Newman and Brown, 1992; Sheinfeld and Lord, 1981). A second group of projects, which complement the first, essentially aims to highlight the evaluators’ ethical sensitivity by allowing them to express their views about the ethical questions they consider to be central to their profession or to their role in the evaluation process (Morris, 2008; Morris and Cohn, 1993; Morris and Jacobs, 2000). These two types of studies have helped advance knowledge about evaluators’ ethical behavior and judgment, mainly in the USA. Studies elsewhere continue to be sporadic even though evaluation has become commonplace in countries such as Canada (Jacob, 2006; Segsworth, 2002).
Our project follows in the footsteps of Morris and Cohn (1993) and Morris and Jacobs (2000), whose studies analyze how evaluators identify ethical dilemmas that they confront. An ethical dilemma can be defined as a situation in which the evaluator is faced with a moral choice: ‘Ethical problems deal with matters of moral accountability related to “doing the right (good) thing” or “doing the wrong (bad) thing”’ (Morris, 2008: 2). Studies in the field of evaluation ethics conclude that some evaluators easily recognize the existence of ethical dilemmas while others do not. Morris contends that the elements that may impact an evaluator’s ethical sensitivity are not well understood:
Unfortunately, our understanding of the factors that influence one’s tendency to view professional problems through an ethical lens is not well developed. All we can say with certainty at this point is that significant individual differences exist. Some might characterize these differences as variation in ‘moral sensitivity’, which is ‘the ability to recognise the ethical dimension of a situation’ (Welfel and Kitchener, 1992: 179). Framing the issue in terms of ability, however, suggests that the key dimension involved is analytical competence, when it may be the case that factors such as a value orientations and ideology play a much more prominent role. (2008: 3)
In fact, although evaluators may have similar objectives, the quality of the evaluation is influenced in part by the evaluator’s perception of his own role in the process. This perception can vary from one individual to the next. The first studies published on matters related to role of the evaluator date back to the 1970s, when Sheinfeld (1978) drew up a list of values (distributive justice, truth-seeking, human dignity, sharing, concern for quality of life and client loyalty) that were characteristically found among evaluators but that could vary depending on each individual’s personality. A quantitative study published soon afterwards identified protection of human rights, freedom from political interference and evaluator technical competency as the three core values of evaluators (Sheinfeld and Lord, 1981: 385). A few months later, Jenkins-Smith (1982), whose interest lay more with the policy analyst’s role, distinguished three broad categories that motivated individuals in this profession (issue advocate, objective technician and client’s advocate). As he analyzes the professional reasons that lead an evaluator to refuse a given mandate, Smith (1998) presents evaluation as an activity that ‘can promote either guild maintenance interests or societal improvement interests’ (p. 177).
Through considering the degree to which evaluators have either altruistic or corporatist objectives, we aimed to better understand how these attitudes impact their recognition of ethical considerations throughout the evaluation process.
This exploratory study had two distinct goals. First, we aimed to demonstrate that it is possible to link specific values to different types of evaluators. If this was in fact the case, our second objective was to build a model to correlate an evaluator’s reflective logic with their pursuit of either altruistic or corporatist objectives. Our research question was: ‘Do evaluators perceive the ethical issues with which they are confronted? If so, what factors influence ethical sensitivity?’ Ethical sensitivity can be defined as the evaluator’s ability to detect ethical issues that may surface within the course of their professional activities.
Methodology
Previously published works will guide our project. More precisely, we will combine the following:
Nick L Smith’s (1998) model, which details possible reasons that may lead an evaluator to accept or refuse to take on an assignment, highlights contradictory interests such as serving guild maintenance or contributing to societal improvement;
the analytical model proposed by Morris and Jacobs (2000) to determine if an evaluator’s ethical sensitivity is influenced by their own perception of their role during the evaluation process.
Thus, we hoped to show that ethical sensitivity (dependent variable) is influenced not only by evaluator type (independent variable) but also more importantly by an individual’s perception of the evaluator’s role (Jacob and Boisvert, 2010). The control variables are the evaluator’s education, experience (in years and in the number of evaluations completed), work environment, type of evaluation (internal or external), the evaluator’s field of expertise and their knowledge of the Canadian Evaluation Society’s ethics, evaluation norms and guidelines in matters of ethics.
We conducted a qualitative study with Canadian evaluators using the vignette method. This methodological approach has ‘long been used to study attitudes, perceptions, beliefs and norms within social science’ (Wilks, 2004: 80).
Vignettes are brief stories or scenarios that describe hypothetical characters or situations to which a respondent is asked to react. Because they portray hypothetical situations, they offer a less threatening way to explore sensitive subjects (Finch, 1987).
Their specificity allows contextual influences on judgments to be examined. To preserve realism, qualitative researchers may create vignettes based on actual situations reported to them. These are then used to stimulate open-ended discussions with respondents to explore their reasoning and judgments. (Martin, 2006: 2)
The six sample cases used were typical examples of ethical dilemmas often presented in literature on ethics in evaluation (Morris, 2008 and the ‘Ethical Challenges’ section of the American Journal of Evaluation, Government of Canada, 2003):
the sponsor exerts pressure to direct the report’s content (Vignette 1);
disclosure of information on a managerial problem unrelated to the evaluation (Vignette 2);
the withholding of evaluation objectives from participants to facilitate participation (Vignette 3);
clients only present those conclusions that are advantageous and suppress negative or less favourable data to protect their political agenda (Vignette 4);
during an interview, the evaluator uncovers information that would interest local authorities and prevent a potential crime (Vignette 5);
the sponsor, unsatisfied with the results of the evaluation, asks for new analysis (Vignette 6).
Participating evaluators were asked to rate the case on a 5-point Likert scale and were then asked to further clarify their opinion. For example, participants could be asked which elements stood out for them, what he would do if faced with such a situation and their opinion on the frequency of such a situation arising during an evaluation. We wanted to discuss each scenario in depth with the respondents in order to assess their understanding of each case scenario as well as their ability to detect ethical dilemmas.
The first author conducted semi-directed interviews with 11 respondents: eight face to face and three by telephone. We favoured the semi-directed interview because it allowed us delve deeper into the evaluator’s reality. This method allowed the informant to provide us with detailed nuanced responses about experiences, knowledge of the field and expectations (Gauthier, 2003: 299). A non-random sample aimed at representing different aspects of evaluation in Canada (government, evaluation firms, university, internal/external evaluation, etc.) allowed us to present different case scenarios to a wide sample of individuals involved in evaluation. We were then able to develop a list of elements, the identification of which allowed each respondent to demonstrate his or her ethical sensitivity. For example, the removal of one section of a report deprives the evaluation of significant conclusions and recommendations and risks leading to repercussions on civil society or damages the reputation of the evaluator community. This may lead to negative consequences for program beneficiaries.
Before the interview began, the participating evaluator was asked to complete a two-part closed-question survey. The first section allowed for the collection of data related to the study’s control variables. The second section contained questions about the respondent’s perception of the evaluation process and revolved around a series of general assumptions or themes. For example, it might be assumed that: an evaluation must always contribute to the well-being of society; it is possible for a good evaluation to damage the reputation of evaluators; a good evaluation needs to address the objectives of the sponsor; all evaluations need to be profitable for the evaluator; the evaluator is an active political actor, etc. Questions were based on Smith’s list of elements that may influence an evaluator’s acceptance or refusal of an evaluation assignment. The remainder of the meeting consisted of a discussion about the six hypothetical vignettes during which the participant was asked to identify the ethical dilemmas present within each scenario. They were also asked to express their views on the fictional evaluator’s behavior.
Data analysis was based on the operationalization of the two principal concepts at the heart of this project. Evaluator type was divided into two categories: altruistic and corporatist. An altruistic evaluator places a great importance on evaluations that contribute to the greater welfare of society; favours an evaluation that reflects society’s values and believes that his or her role as a political agent is to be the voice of the general population. A corporatist evaluator places greater importance on the profitability of the evaluation; considers that such a process should not harm the reputation of the evaluator or the profession as a whole; and places great importance on the evaluation’s ability to address the sponsor’s objectives. Corporatist evaluators also believe that the evaluation should provide the sponsor with arguments so that they might defend their point of view on the evaluated program. These indicators allow us to classify each respondent on a scale that ranges from ‘very altruistic’ to ‘very corporatist.’ The study’s second central element relates to whether the subjects’ ethical sensitivity may be described as weak, average or strong. Ethical sensitivity increases in relation to the ability of the evaluator to detect the presence of ethical dilemmas.
To measure the respondents’ ethical sensitivity, we compared their reaction to each scenario with the responses we were expecting, derived from a list of factors that could influence ethical tensions in each scenario. These responses were coded to measure each respondent’s ethical sensitivity for each vignette in relation to the other participants. When a participant demonstrated a weak ethical sensitivity, they were given 0 points for the scenario in question; for average sensitivity, 1 point; and for strong ethical sensitivity, they received 2 points. The maximum score an evaluator could obtain was 12. On this basis, we concluded that the evaluator who received a score of 0 to 4 points has a weak ethical sensitivity, the evaluator who obtained a score of 5 to 8 has an average ethical sensitivity, and those who obtained 9 points or more had strong ethical sensitivity.
Results and discussion
Before presenting the study’s results, we would like to draw a global portrait of the individuals who took part in our project. The study’s participants were Canadian evaluators who were either employed by the public sector (federal or provincial governments) or by the private sector (consulting firms or universities). They were primarily external evaluators who conduct their evaluation as part of a team. All interviewees were university-educated (one BA, seven MA’s and three Phd’s). As a group, their work experiences were varied when it came to conducting and supervising evaluation mandates (four participants have more than 21 years of experience, two had between 16 and 20 years, one had six to 10 years, two had 11 to 15 years, and two had one to five years experience). The majority of evaluators had conducted more than five evaluations over the course of their career. Despite this, a much smaller number (27%) stated they were very knowledgeable about the Canadian Evaluation Society’s ethical norms and guidelines and 36 percent stated they were not aware of these at all. Data collected through the short survey completed by the respondents prior to the interview seems to indicate that not all Canadian evaluators prioritize the same values when they engage in the evaluation process. As demonstrated in Table 1, a small majority of evaluators who took part in the study had stronger corporatist-type values and one participant’s values fell somewhere between the two defined categories.
Evaluator characteristics (N = 11).
The participating evaluators’ considerations of the problematic nature of the presented situations (Table 2) and ethical sensitivity (Table 3) varied depending on the case scenarios they faced. On the whole, most evaluators demonstrated a greater ethical sensitivity when they discussed the situation in detail and were asked to identify the more contentious elements that could adversely impact the evaluator or the profession at large, the sponsor, those who could benefit from the program’s evaluation and society in general. By confronting the participants with different scenarios, then analyzing their views on these vignettes in relation to their evaluator profile, we were able to assess how personal values can influence detection of ethical dilemmas, as demonstrated in Table 4.
Considerations of the problematic nature of presented situations (N = 11).
Ethical sensitivity towards presented situations (N = 11).
Ethical sensitivity according to evaluator type (N = 11).
Further analysis of this data allows us to see that an individual’s nature or evaluator type affects ethical sensitivity. As we reviewed the contents of the interviews, we noted a tendency for more altruistic evaluators to identify ethical dilemmas more readily.
As different scenarios were discussed, comments from those respondents who were more altruistic referred more frequently to principles such as equity, justice and respect of ethical guidelines. For example, when commenting on the second vignette, which deals with a general management issue uncovered in the course of the evaluation, an altruistic evaluator explains why he would choose to mention inappropriate behavior to the sponsors: ‘In this case, it’s an important issue because the individual’s behavior can negatively affect program quality. Despite this, one has to be very careful because there’s a real person behind that problem. One has to make sure that this individual cannot be identified.’ In contrast, an evaluator with stronger corporatist tendencies would have a reaction similar to this respondent’s: ‘I would talk about it openly in the report and I would make sure that the program leaders know which individual’s behavior was problematic. I don’t have a choice. I was hired to find problems and this is a major one.’ In this case, one can clearly see how altruistic and corporatist logics can influence an evaluator’s ethical sensitivity. Perceiving evaluation as a way to contribute to society’s well-being seems to lead evaluators to consider a greater number of perspectives. Interestingly, those evaluators who were more altruistic found it harder to judge the fictional evaluator’s behavior when they first read the vignette. Altruistic evaluators would often justify their answers by stating that each situation needed to be assessed on a case by case basis because of this or that factor. They would regularly ask for more details about the evaluation’s initial mandate to better understand the proposed scenario’s context and the issues surrounding it. For example, in the case scenario where the evaluation’s actual aim is withheld from the participants (Vignette 3), a typical answer from an altruistic evaluator could be as follows: ‘This type of situation arises from time to time. It’s a case by case situation. Essentially, we need not be afraid to talk to the sponsors so we can find an alternative solution to this problem because it’s never permissible to lie to participants.’ Altruistic evaluators make no compromises that can affect a participant’s integrity. For them, there is no other avenue other than modifying the research method or withdrawing from the evaluation process.
On the other hand, evaluators with corporatist tendencies prioritize evaluation results by occasionally forgoing or ignoring ethical standards and guidelines for the sake of obtaining results. A typical response from a corporatist-minded respondent in reaction to the aforementioned vignette, where information is withheld from participants, would be as follows: ‘Sometimes we have no other choice but to hide our intentions to reach our goals. However, the expression “masking the objectives” is a little strong. I’d say that we don’t put any emphasis on the evaluation’s subject and try to adjust the survey to get answers to the questions we need answered, without alarming those involved. You’ve got to understand – we’re hired to find answers.’ Thus, despite insisting that they would not openly deceive participants, an evaluator might believe, it was inconsistent with the evaluators’ role to disclose too much information in order to complete the mandate. This corporatist evaluator, who is primarily results-oriented, does not seem concerned that not respecting a participant’s right to informed consent contravenes a fundamental ethics principle. This example illustrates how a corporatist perspective on evaluation can lead to neglect of ethical concerns during the evaluative process. In comparison, though altruistic evaluators do also feel the need to comply with the sponsor’s expectations, they seem to prioritize the impact of the process on participants and on society as a whole. Incidentally, discussions with altruistic-type respondents were generally longer as they tended to identify a greater number of elements to consider in each case scenario. They had a net propensity to ask numerous questions about each case. This type of evaluator was more inclined towards a dialogical logic of ethics (Arnett et al., 2009) as they tried to fully comprehend the points of view and perspectives of those involves in the evaluation process and to consider the evaluation’s general effect before making a decision. This attitude corresponds to Smith’s (1998: 184–5) description of altruistic evaluators, whose main concerns include not only the evaluation’s impact on their profession but also the impact of the evaluation on society at large. One of the altruistic respondents addresses the issue by maintaining that, during the evaluation process, the evaluator has two commitments to fulfill. The first commitment is with the evaluation process’s sponsor. In this case, the evaluator must commit to complete the best possible evaluation, taking into account all mutually agreed quality standards. The second commitment is with society at large. It consists in conducting an optimal evaluation that considers, throughout the process, all possible consequences it may have on society’s well-being.
Adversely, giving priority to corporate values within the evaluation process can sometimes lead to undetected ethical issues. The corporatist-type respondents involved in our study were predominantly concerned with issues related to delivery of an evaluation that was in keeping with the sponsor’s expectations. Their priorities tended to be linked to the client’s goals and objectives, an attitude that is not quite the same as with altruistic evaluators. For examples, in the case scenarios where the sponsor demands a section of the evaluation report be excluded (Vignette 1), altruistic evaluators were categorical: ‘This is a serious issue. I cannot not report this. I have to mention it in my report!’ and he goes on to say: ‘This is exactly the type of situation that can lead to confrontation or even to threats. You’ve got to be careful. I was faced with this type of situation and you’re faced with an incredible amount of pressure. But it’s our duty to take a stand.’ One can see here how the evaluator’s priorities can affect ethical sensitivities. In this particular case, an evaluator who has little concern for the evaluation’s impact could have easily removed the controversial section from the report without considering this decision’s influence on the results. A corporatist evaluator, despite raising concerns about the situation, makes the following statement: ‘First and foremost, I’d suggest a re-writing of the report’s conclusions. Pretty often, it’s possible to find common ground. Ultimately, I have to be there for my clients.’ From a corporatist perspective, pressure from the sponsor will weigh more in the balance than the risk of producing erroneous findings. Similarly, with the scenario dealing with a secret arms cache (Vignette 5), the corporatist evaluators were generally the ones who would consider the sponsor’s point of view before that of society as a whole. One particular corporatist respondent was extremely sensitive about taking risks with his reputation when he transmitted findings: ‘It’s similar to a journalist revealing his sources. If word gets out that you’re willing to throw your sponsor under the bus, you’ll never have access to the field again. You won’t be considered trustworthy anymore.’
As a general rule, altruistic-type respondents tried harder to avoid conflict than those who were corporatists. In some cases, when it was evident that a troublesome situation could not be resolved and would probably have a negative effect on the evaluation process, altruistic evaluators chose to stop the process while the corporatist respondents preferred to seek results despite the risks involved. This reaction could partly explain why these evaluators had a propensity to detect only some of the ethical issues present in the case scenarios. They regularly favoured the sponsor’s position or preferred to continue the evaluation, all the while not recognizing that these decisions may have an impact on society.
Alternative factors
The results suggested that ethical sensitivity can only be partially explained by a respondent’s altruistic or corporatist tendencies. Therefore, the study also sought to identify other factors that may influence ethical sensitivity. The following elements appear important to consider: knowledge of evaluation norms and guidelines, the environment in which the evaluation is being conducted, the evaluator’s professional affiliation and past experiences in the field.
Knowledge of prescriptive norms
Despite the fact that study participants rarely referred to prescriptive norms (e.g. the Canadian Evaluation Society’s evaluation norms and guidelines) to justify the appropriateness of their choices, those evaluators who declared knowledge of these norms and guidelines identified a greater number of ethical issues than those who had little or no knowledge of them. On the contrary, evaluators who did not have working knowledge of prescriptive norms more often failed to recognize certain ethical tensions in the discussed scenarios. They tended to consider that the evaluator’s role had limited scope, giving precedence to the sponsor’s interest and disregarding their relationship with other parties involved.
Generally, evaluators with a partial or perfect knowledge of the prescriptive norms put forward a more detailed analysis of the cases and were able to identify more elements that could influence an evaluator’s role. In particular, the third and fifth vignettes, where issues of consent and confidentiality were brought up, those evaluators with knowledge of ethical guidelines were more likely to identify situations with ethical tensions in both of the aforementioned scenarios. One of the study participants described informed consent in these terms: ‘There are ethical rules that have to be followed. You can’t be allowed to hide your intervention’s goals. Anyway, sooner or later, people are going to find out and you’re going to get burned.’ However, despite this better understanding of the norms, not all evaluators identified the same factors to consider when adopting ethical behavior. This conclusion is corroborated by Mabry’s (1999, 2004) findings; these demonstrate that the existence and prior knowledge of evaluation norms do not automatically lead to the ability to identify or to foresee all possible ethical issues that arise during the evaluation process, nor do they steer evaluators toward appropriate ethical behaviors. Ultimately, ‘in the subjective courts of our own minds, we decide individually how to conduct ourselves, how and how much to draw from the Standards and Principles’ (Mabry, 1999: 201). Though prescriptive principles are not a panacea, they can act as markers that may help prevent ethical dilemmas and offer guidance as ethical issues surface throughout the evaluation process (Datta, 2002; Smith, 2002). One evaluator thought it useful to discuss these ethical norms with the sponsor before the mandate began to make sure the process was well understood: ‘Before each evaluation, one thing I make sure I do is explain evaluation standards to clients. That way, they know exactly how I’m going to do things and the norms that I have to comply with. It prevents a lot of conflicts.’ This approach can avert misunderstandings as well as defend against the sponsor challenging the evaluator’s work in the midst of the evaluation process.
Private versus public / internal versus external evaluation
The second factor that may lead to a better understanding of a respondent’s ethical sensitivity is the evaluator’s professional affiliation (i.e. consultant firm/external or public administration/internal). It is generally assumed that private consultants tend to be corporatists and, thus, have reduced ethical sensitivity because their working environment is strongly influenced by market forces (Morris and Jacobs, 2000). Evaluators employed by consulting firms are aware of market forces and their impact on their work. Nevertheless, these same evaluators consider that these market forces compel them to exercise greater vigilance throughout the evaluation process. They maintain that they cannot allow themselves to ignore ethical issues if they hope to remain credible and, thus, must make sure that they perform a high-quality evaluation that follows ethical norms and that they act in accordance with the highest possible ethical standards. This observation goes against Hosmer’s (1994) views on market ethics. According to him, despite the fact good ethical behavior is potentially profitable in the long term, it occasionally becomes necessary, within a market logic, to shy away from, or make concessions in relation to, ethics so as to obtain short-term profitability. What distinguishes this theory from the perceptions of the private-firm respondents in our study is that short-term objectives do not impact their ethical behavior. Private consultants are thus willing to accept short-term losses for the prospect of long-term profit. This private-sector evaluator voiced it in the following terms: ‘I can’t allow myself to cut corners to please a sponsor. I’ll just discredit myself. If I chose to do it, I’d probably be able to complete the mandate I was hired for but it may be my last.’ This point of view is also shared by another private-sector consultant: ‘I’d rather lose a contract than to see my firm lose all credibility when tainted evaluation results are published.’ According to these two respondents, good work ethics have greater value than pleasing the sponsor because it increases their overall credibility. Looking to fulfil the client’s needs by taking ethically questionable shortcuts is frowned upon in their line of work. They seem to believe they are better off refusing or putting an end to an assignment if it is not possible to undertake it in accordance with their quality standards.
Though not all private-sector study participants were aware of all the prescriptive norms established by the CES, they emphasized that they respected those norms and standards put forth by the consulting firm that employed them. Generally speaking, though sponsors are very important for private consultants, they are able to distance themselves from them once the assignment is underway. Thereby, once terms are agreed, the consultant is able to handle the evaluation according to the set agreement and is less likely to serve the sponsor’s cause alone. This perspective is also shared by the university-based external evaluator. When initially provided with the first vignette, in which the scenario includes the removal of a section of the report, the university-based evaluator explained that, in his case, things were negotiated before the contract was signed: ‘If the contract stipulates that I am to investigate this case, it is then clear that I cannot allow this section to be withdrawn if the results are pertinent to the case, even if it may ruffle the sponsor’s feathers.’
Additionally, external evaluators identified the fictional evaluator’s transgression as well as ethical issues with greater ease than internal evaluators, particularly those issues linked to the correct attitude to have when dealing with sponsors’ expectations. This view corresponds to Morris and Cohn’s (1993) conclusion that external evaluators are sometimes more likely to identify ethical dilemmas throughout the evaluation process. In this case, the results can be explained predominantly by the structure of the internal-evaluation process in governmental settings. The federal-government evaluator is not necessarily the one that directly accomplished the evaluation but rather is responsible for ordering and steering the process. Sometimes, the federal evaluator co-signs the evaluation report or sometimes even removes the private firm’s name from the report and replaces it with the department’s contact information. Thus, because they steer the evaluation process and they are accountable for the data, all evaluators who are also civil servants showed greater concern for the sponsor’s need, their employer, and ministerial management than they did for their work’s impact on society in general. Internal evaluators insisted that they accomplished their work in a distinctive political context, as described by a respondent when he was confronted with the scenario in which he was asked to remove a section of the report (Vignette 1): ‘This is neither a good or a bad thing; it’s typical of how things works in government. Some information can be misinterpreted from time to time and it’s best that it be removed from the report so not to create more confusion.’ Internal federal evaluators assumed that the evaluation could serve political ends and, consequently, they accepted that they might need to adjust evaluation to conclusions to be more in line with sponsor expectations. The internal evaluator’s primary goal when conducting an evaluation is to optimize the program’s operations. This statement is supported by another evaluator when speaking of the need for more research (Vignette 6): ‘As an evaluator, you’re mandated to conduct research. Your role isn’t to question what type of research is commissioned. As long as you’re paid to do that extra work, refusing to consider a manager’s demands wouldn’t be well received.’ Their ministries had goals to reach and internal evaluators needed to do whatever was necessary to achieve results. Consequently, their ethical sensitivity is aimed at dealing with internal evaluation issues, which translates into an increased loyalty towards sponsors and program managers. All the while, most fail to detect global ethical issues and ignore those ethical dilemmas that relate to the sponsor’s relationship as well as those concerning the impact that their own behavior may have on the greater good. This position is paradoxical and contradicts the Values and Ethics Code for Public Service, which stipulates:
Public servants, in fulfilling their official duties and responsibilities, shall make decisions in the public interest. If a conflict should arise between the private interests and the official duties of a public servant, the conflict shall be resolved in favour of the public interest. (Government of Canada, 2003: 9)
Nevertheless, these results confirm the assumption that internal evaluators are more likely to be pressured by the organization for which they work. On occasion, ‘organizations work against self-reflection and self-criticism and thus internal evaluators must often go against the organizational zeitgeist’ (Mathison, 1991: 177). Hence, internal evaluators must juggle the need to produce a quality evaluation with the pressure from employer as well as the evaluation’s impact on their own, and colleagues’, careers (Lovell, 1995). In such circumstances, the internal evaluator is often encouraged to extract only the data that show the program in a positive light (Adams, 1985).
Another interesting difference between internal and external evaluators relates to the evaluator’s proximity to the program being evaluated and the management of confidential data. From the discussions we had with the respondents, there is a tendency for internal evaluators to be close to individuals involved in program development and implementation. They are familiar with day-to-day challenges faced by these people and know which employees are considered problematic. Despite the fact that the evaluators may be aware of the dangers related to the conflict of interests that can be caused by such a proximity (Yang and Shen, 2006), they seemed generally less concerned with respecting principles of confidentiality than did their external counterparts. Moreover, these internal evaluators clearly recognized that they are open to discussing problems identified in the course of the evaluation with program administrators. Some go so far as to say that they sometimes have no other choice but to forgo confidentiality agreements so that a problem may be solved, since program administrators are very much involved in the evaluation process.
The evaluator’s experience
The interviews have also led us to better understand how previous evaluation experience affects ethical sensitivity. First, it was noted that ethical sensitivity and previous evaluation experience did not necessarily go hand in hand. This is confirmed by previously published studies that show that evaluation experience does not guarantee greater sensitivity to ethical issues (Morris and Cohn, 1993; Morris and Jacobs, 2000; Newman, 1995).
An inexperienced evaluator expresses his perspective on a case involving the disclosure of information that could lead to crime prevention (Vignette 5): ‘I would check with my boss, I’m not sure. If the confidentiality agreement needs to be respected, then I wouldn’t let the police know. This is the type of case that would give me nightmares.’ A second evaluator with little experience expressed himself in these terms: ‘It’s not that obvious. You’ve got to juggle with concepts like security and confidentiality. I’d probably speak to coworkers or to lawyers about this.’ These examples show that despite their ability to properly identify ethical issues, inexperienced evaluators are not comfortable addressing the issue. This can be partially explained by Barrington’s (1999) thesis that the only way to be at ease with ethical issues is to be repeatedly confronted by them. On the other hand, some more experienced evaluators were not able to identify all of the ethical dilemmas present in a given scenario and would rather proceed with the evaluation than solve the problem, stating that their reputation hangs in the balance. However, one long-time evaluator was quick to respond to the arms-cache vignette: ‘In a case like this one, you don’t even stop and think. There’s a risk to the population so you’ve got to speak up!’ This example demonstrates that experience can have a negative influence on ethical sensitivity without adversely affecting ethical judgment. The most experienced evaluators seem to deal with ethical dilemmas with more ease.
Conclusion
Based on the study results, we can state confidently that multiple factors affect an evaluator’s ethical sensitivity, notably in relation to values indentified as significant to an evaluator’s role. This is supported by Mabry (2004) whose research findings demonstrate that different attitudes towards evaluation translate into systematically different ethical priorities (p. 386). These are especially found in cases where decisions have repercussions for society as a whole; and according to our findings the ethical sensitivity of these cases is identified more often by altruistic rather than corporatist evaluators. Regularly, corporatist-type evaluators give precedence to the sponsor’s demands or to the respect of norms related to confidentiality. On the other hand, the altruistic-type evaluator has a more humanistic and collective perspective when they analyze ethically challenging situations, regardless of the fact that they consider both considerations put forward by the corporatist evaluators to be significant.
At the same time, we noticed that knowledge of evaluation norms and of the Canadian Evaluation Society’s ethical principles influenced the respondents’ capacity to identify ethical issues. Those who were familiar with these principles identified a greater number of ethical elements to consider before making a fair and just assessment of the scenario. This observation is in agreement with Smith’s conclusions which suggest that evaluation norms steer evaluators to recognize potential ethical issues that may arise during the evaluation process while at the same time, these same principles may provide possible solutions to these dilemmas (Smith, 2002: 201). To a lesser extent, conducting internal evaluations within the government tends to reduce the ethical sensitivity of evaluators, as does seniority in the profession. However, an increase in the number of evaluations conducted by a particular evaluator increases ethical sensitivity.
There are limitations to this study that should be noted. Given the fact that we used a qualitative research design based on semi-directed interviews to discuss fictitious scenarios, we needed to limit the number of observations. These results are not representative of all evaluators or of evaluation in Canada. However, this design allowed us to take an in-depth look at the thought process of a selected group of evaluators to observe how they approached ethical dilemmas. We were also able to gather precious information that would have been impossible to obtain using a questionnaire. For example, the order in which the answers were given or the length of time of the discussions for each scenario helped us to identify what the participants prioritized and considered to be the inherent issues to their profession. Furthermore, the face-to-face relationship of the interview also carried a disadvantage and could encourage certain respondents to stick to a point of view that they thought was in line with researcher expectations. They may have been more transparent and honest if they had answered more anonymously. However, in encouraging the evaluators to detail and explain their answers and the reasons that led them to consider a situation to be ethically problematic, we were in a position to partially overcome the respondent’s problem of social desirability. Furthermore, it can be argued that this effect is limited thanks to the use of vignettes that allows the respondents more freedom in their responses since they are commenting on a hypothetical situation rather than the reality of their own practice (Gould, 1996; Wilks, 2004).
Other research should be conducted to further explore this topic. For example, a research project aimed at observing an evaluator’s daily practice could allow for a better understanding of the ethical sensitivity of evaluators, the identification of ethical dilemmas and resolution or avoidance strategies that they adopt.
Footnotes
Acknowledgements
Financial support from the Social Sciences and Humanities Research Council of Canada (SSHRC) for the project entitled ‘Ethics and Evaluation of Public Policy: Links to (re)Discover’ is gratefully acknowledged. We would like to thank the Canadian evaluators who participated in this research for their availability. The information they offered us via interviews was invaluable to this research. So as to respect the confidentiality agreement we have with the interviewees, we have omitted their names and have used gender-neutral pronouns throughout the text. We also would like to thank participants at the seminar organized by the Working Group on Quality and Deontology of the French Evaluation Society (especially Jacques Toulemonde, the group’s coordinator) for their comments on a preliminary version of this text. This article was written while Steve Jacob was a visiting scholar at the School of Behavioural and Organizational Sciences at Claremont Graduate University.
Funding
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.