Impartiality and Hierarchical Evaluations in the Japanese Development Aid Community

Abstract

This article presents a discourse analytic study of how the concept of impartiality is socially constructed by members of the development aid community through an examination of linguistic traits and patterns within (a) inter- and intraorganizational interactions and (b) relevant aid evaluation policy documents. A qualitative analysis of unstructured and semistructured interviews with development professionals in Japan and a content analysis of relevant evaluation policies and documents have revealed observable evidence of distinct institutional practices of impartiality via project evaluations. Most notably, in contrast to a notion of evaluator impartiality that is strengthened through the principle of independence, development professionals in Japan perceive evaluations (‘hyouka’) to be a fundamentally hierarchical construct in which evaluator impartiality can be strengthened or legitimized through authoritative or hierarchical means.

Keywords

cross-cultural evaluation international development evaluation evaluation policy content analysis Japan

Introduction: Evaluating Development Aid

For organizations involved in international development aid, evaluations enable stakeholders to assess the performance and quality of aid initiatives. Project and program evaluations are a key component of the development aid process to ensure that aid delivery continues to improve for beneficiaries in developing regions as well as to enforce accountability to stakeholders involved in the development aid process. According to the Development Assistance Committee (DAC) of the Organization for Economic Cooperation and Development (OECD), development organizations (i.e., bilateral and multilateral aid organizations) among OECD member countries spent an average of US$5.1 million on evaluations in 2010 (OECD, 2010c). Although average annual spending in by bilateral agencies was lower at US$2.4 million, institutions primarily responsible for planning and implementing bilateral Official Development Assistance (ODA) in Japan—namely, the Ministry of Foreign Affairs (MOFA) Japan and the Japan International Cooperation Agency (JICA)—spent a total of US$14.8 million on evaluations in 2010 (OECD, 2010c). Development evaluation efforts in Japan thus cost 6 times the average spending rates of bilateral agencies in the OECD during this time (OECD, 2010c). Japanese aid organizations also conducted a whopping 591 evaluations in 2012, compared to OECD members’ reported average of 24 evaluations a year per evaluation unit in 2010 (OECD, 2010c).

Such wide variations in institutional practices reveal how evaluation schemes and practices can take on a multitude of forms among development agencies. Each approach is dependent upon the purposes, objectives, mandates, characteristics, and scale of any given evaluation effort. Evaluations of development aid may target individual projects, sector-based or regional programs, or overarching national policies and global strategies. Difficulties associated with the complex nature of such tasks can create numerous issues for publicly funded development organizations in their efforts to implement credible and useful evaluations.

Impartiality in evaluations is a key issue that applies to addressing the quality and credibility of such evaluation findings. The ability to maintain an impartial stance, or practice impartial judgment, is a central issue in various domains, such as the rule of law (e.g., the United Nations [UN] Office of the High Commissioner for Human Rights, in collaboration with the International Bar Association, 2003), moral judgment (Barry, 1995; Friedman, 1989), humanitarian aid and peacekeeping (International Federation of Red Cross and Red Crescent Societies, 2014; UN, 2014), and journalism (Sambrook, 2012), among many others. Within these domains, impartiality is largely conceptualized as a means of attaining “better” or more appropriate results (since impartial standards can theoretically be exercised to attain morally “bad” or undesirable outcomes). For evaluators, this amounts to the challenge of “promoting the fairness, balance and justice” of evaluations (Datta, 2000, p. 87).

In acknowledging the complexity associated with evaluating development aid projects, this research focuses on the evaluation practices of development professionals in Japan. By focusing on the evaluation practices of the Japanese international development community, the discussion provided in this article highlights differences in evaluation practices across institutional and cultural contexts.

Objectives and Methodological Framework

The objective of this study is to provide a descriptive analysis of how the concept of impartiality is socially constructed by members of the development aid community through an examination of linguistic traits and patterns within (a) inter- and intraorganizational interactions and (b) relevant aid evaluation policies. The case examined is drawn from a quasi-public bilateral aid agency in Japan, namely, the JICA, as well as from development consultants in the private sector that take on contractual work for JICA’s ODA projects. The primary data of this study are composed of recordings, transcripts, and notes of unstructured and semistructured interviews with 14 subjects (Table 1), as well as a facilitated dialogue with professionals in these respective organizations. With limitations of restricted access, a snowball sample was obtained in Japan over the course of several months in 2013. Where access was successfully gained, subjects completed semistructured interviews and participated in discussions about impartiality by sharing experiences about the evaluation procedures of international development projects.

Table 1.

Experience and Affiliation of Interview Subjects.

Affiliation	Years of Experience
Affiliation	3–5	6–9	10–15	20+	Total
JICA official	5	1	—	—	6
Consultant	3	1	3	1	8
Total	8	2	3	1	14

Note. JICA = Japan International Cooperation Agency.

Preliminary interviews were structured around a central problem statement: “What is the role of impartiality in Japan’s development community?” These initial interviews with development professionals revealed that project and program evaluations are perceived to be the most prominent example of “impartial” organizational procedure. Inspired by a grounded theory approach (cf. Glaser & Strauss, 1967), interviews evolved progressively to focus on evaluation procedures as key themes emerged from interview testimonies. Subsequent topics and questions covered throughout each interview included: How do you (impartially) determine and evaluate the value of a project? What do you think most determines the value of an aid project? Have you ever experienced any disagreements about the value of a project, or its evaluation outcome? and How do you think current evaluation methods/processes can be improved and why?

Upon identifying the perception of evaluations as an impartial institutional procedure among development professionals, the study organized a facilitated dialogue between two JICA officials and two development consultants to allow subjects from respective stakeholder groups to share perceptions and discuss conflicts associated with the evaluation process. Dialogue refers to the multifaceted process of exchanging and exploring individual perceptions (Bohm, 1996), while the term facilitated dialogue was developed and is used by conflict resolution practitioners to refer to the process in which a facilitator attempts to aid “parties [to] engage in deep and meaningful conversations with their opponents, not for the purpose of resolving a dispute, but rather for the purpose of developing a better understanding” (Burgess, Burgess, Glaser, & Yvsyukova, 1997).

A total of 15 hr of interview material and facilitated conversations have been audio-recorded, transcribed, examined, and coded in consultation with interview notes. Theoretical concepts from cognitive linguistics (Lakoff & Johnson, 1980; Talmy, 1983) enabled the analysis to reveal distinct features within the primary data (these concepts will be outlined and discussed in detail with the main findings).

A content analysis of major evaluation policy documents and international guidelines comprises the secondary data analysis of this study. Excluding the (1) evaluation policies of other OECD member countries and (2) OECD peer reviews, documents within the secondary data that are cited as examples of Japanese and international “evaluation discourse” are only those that were also mentioned by interview subjects in discussions of evaluation policy and literature. The bilateral ODA evaluation policies of other OECD member countries and OECD peer reviews have been utilized for comparative purposes and were not directly referred to in interviews with Japanese development professionals (see Table 2 for a list of documents included in this analysis). This secondary analysis has been utilized to triangulate observations made of the primary data. A discourse analytic approach has been employed to the multisourced and multistructured data to create a qualitative and situated discussion of evaluations within the development community in Japan.

Table 2.

Documents Utilized for Secondary Analysis.

Stakeholder	Title of Document	Year
JICA	Guidelines for Operations Evaluations Guideline for Project Evaluation Annual Evaluation Report	2014 2004 2013
MOFA	ODA Evaluation guidelines	2013
Top 5 ODA Spenders in OECD	USA—Evaluation at USAID UK—DFID International Development Evaluation Policy Germany—BMZ Evaluation Criteria for German Bilateral Development Cooperation France—AFD Evaluation Policy	2013 2013 2006 2013
OECD	OECD DAC Principles for the Evaluation of Development Assistance Evaluation in Development Agencies Evaluating Development Co-operation: Summary of key norms and standards Quality Standards for Development Evaluation 45 OECD Peer Reviews	1991 2010 2010 2010 —

Note. OECD = Organization for Economic Cooperation and Development; DAC = Development Assistance Committee; AFD = Agence Française de Développement; DFID = Department for International Development of the United Kingdom; ODA = Official Development Assistance; USAID = U.S. Agency for International Development; BMZ = Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung; MOFA = Ministry of Foreign Affairs; JICA = Japan International Cooperation Agency.

The discussion begins with a description of how Japanese evaluation practices fit within the efforts of the international development aid community, and a brief explanation of the central role of impartiality in evaluations. This is followed by a review of actual evaluation practices in Japan. It is then proposed that a cultural model of interdependence shapes, guides, and influences institutional discourse in the Japanese development community. This leads to the main findings of the research, in which distinct linguistic characteristics identified in multiple sources of data are presented. The article concludes with a brief discussion and summary of the paper.

Evaluation Practices and Discourse in Development Aid (The OECD-DAC)

We will now describe the evaluation discourse of the international community, via the OECD’s DAC, to situate Japan’s aid evaluation practices as part of a larger community of aid institutions. The significance of the OECD-DAC as a policy-setting entity is described, and their references to principles of impartiality and independence are presented. The most prominent example of international evaluation standards and policy in the field of development aid is arguably represented by policy work from the OECD-DAC. The OECD has a long and well-established history of providing influential policy for economic development, originating from its initial mandate to run the Marshall Plan in 1948 as the Organization for European Economic Cooperation. DAC continues to provide a wide range of research and policy advice concerning the understanding, implementation, and improvement of development policy. These include the widely adopted framework outlined in the “DAC Quality Standards for Development Evaluation” (2010a). DAC policy language will be used to illustrate arguments of evaluation discourse primarily because during interviews with development professionals in Japan who were asked to explain the evaluation process (How are projects or programs evaluated?), all 14 subjects replied with a reference to the DAC standards of effective aid. Originally drafted in 1991, the DAC principles of effective aid outline five specific objectives for evaluations to ascertain and assess the (1) results, (2) effectiveness, (3) efficiency, (4) relevance, and (5) sustainability of specific development interventions (OECD, 2010a). An outline of the concept of impartiality within the DAC principles of aid evaluations is provided next.

Independence and Impartiality as Key Principles in the International Development Aid Community

Through their attempts to measure the five DAC standards for evaluation, aid agencies create various procedures and schemes to suit organizational goals and needs. However, additional principles featured prominently within policy discourse also guide evaluation processes overall. As mentioned above, one of the central concepts that guide evaluation discourse is impartiality. Impartiality refers to a profound fairness principle (cf. Barry, 1995) that embodies the notion that one can exercise judgment without being affected by personal biases or prejudice. The ability to exercise impartial judgment intuitively becomes a central issue when designing and implementing evaluation procedures. These intuitions are reflected explicitly in evaluation standards from DAC, which explicate clearly that evaluation functions must be impartial and independent from the policy making and operational functions of development assistance. For example, DAC’s evaluation principles (1991) specifically stipulate that impartiality is necessary to avoid bias and that independence reduces the potential for conflicts of interest that can arise from evaluating one’s own activities:

The evaluation process should be impartial and independent from the process concerned with policy-making, and the delivery and management of development assistance. (OECD, 1991, p. 6)

The document goes on to state that evaluation results become credible through procedural independence, transparency, and expertise (OECD, 1991). The emphasis on independence implies that failures should be reported on as well as successes with a sincere intent to incorporate critical input in order to support organizational learning. Such policy language implies strongly that independence is a condition that contributes to the impartiality of evaluators. In this manner, international evaluation standards and policies advocate strongly for the principles of independence and impartiality for all institutions that claim membership to the “international community.”

The policy excerpt from the OECD cited above links the concepts of impartiality and independence directly (impartial and independent) and goes on to further qualify specific independent practices and relate them to “impartiality” with the following language:

Impartiality and independence will be best achieved by separating the evaluation function from the line management responsible for planning and managing development assistance. (OECD, 1991, p. 6)

The strong emphasis on independence can be interpreted as a reflection of organizational directives that prioritize accountability, which has “long been a condition of survival” for international organizations such as development agencies (Chelimsky, 1995, p. 9). Evaluators have argued that by maintaining independence, evaluators are able to preserve impartiality by avoiding conflicts of interest and “deliberate” attempts at manipulating evaluation processes (Datta, 1999, p. 85).

However, issues arise due to the fact that it is entirely possible for nonindependent agents to act impartially and for independent agents to be subject to partial bias. The emphasis on independence as a condition of impartiality in evaluations stands opposed to participatory, inclusive, and constructivist evaluation approaches which argue that useful and effective feedback requires the active participation and ownership of the stakeholders involved (e.g., Cousins & Earl, 1992; Guba & Lincoln, 1989; Mertens, 1999; Patton, 1994). These divergent positions among evaluation scholars and development agencies highlight the fact that impartiality is neither equivalent nor synonymous with independence. It is the relationship between the two concepts that must be explored in more depth. In the context of evaluations, independence refers to the state of an agent (i.e., the evaluator) in relation to a subject (e.g., a project or program). Impartiality, on the other hand, refers to the characteristics or state of the agent itself and is infused with notions of fairness and justice. While most conceptual deliberations on the notion of impartiality itself have therefore taken place at the philosophical level (Barry, 1995; Friedman, 1989; Sen, 2002), literature on appropriate evaluator characteristics have long been invigorated by debates about the effectiveness and limitations of detached and objective evaluators (Chelimsky, 1997; Scriven, 1997), constructivist paradigms (Fishman, 1992; Guba & Lincoln, 1989; O’Neill, 1995), participatory evaluations (Cousins & Earl, 1992; Cullen, Coryn, & Rugh, 2011; Mertens, 1999), or advocative approaches (Greene, 1997). As the term impartiality relates to an enormous field of philosophical deliberation concerning fundamental issues of moral judgment, discussions can necessarily become quite abstract. A lack of philosophical consensus and abstraction leads naturally to inconsistent understandings or manifestations of the concept among practitioners. Although the term “neutral” is often used as an equivalent term, the limitations of neutrality (as opposed to impartiality) in cultivating stakeholder inclusion have been discussed at length (Mayer, 2004), and evaluators have specifically considered the importance of distinguishing and defining the relationships between impartiality and concepts such as neutrality and nonpartisanship (Datta, 1999, 2000).

With this active theoretical and practical debate as its backdrop, the development community and its agencies have responded to (a) the establishment of participatory approaches to evaluation and (b) increases in decentralized partnership-based aid by shifting the ownership of aid and its processes toward aid recipients (OECD, 2005; Piccioto, 2007). Joint donor evaluations—of which Japan reports an above average proportion within the evaluation efforts of OECD countries (OECD, 2010c)—have also increased in development aid institutions to “ease pressure on country programmes and partner governments, give more credibility to evaluation, and act as a lever on other agencies” (Foresti, Archer, O’Neil, & Longhurst, 2007, p. 21).

This rapidly evolving field of development aid and evaluations has created a complex mass of evaluation objectives, methods, procedures, and standards. As evaluations take myriad forms dependent upon their purpose and objectives, these complexities can be compounded within the field of international development with the diplomatic, political (domestic and international), and contractual elements that come into play. Despite efforts to produce clear and consistent development policies across the international community (Picciotto, 2005), the development community is still in lively debate as to what types and what processes of evaluations are most appropriate for the development community (Guijt & Roche, 2014). Whether this means that impact evaluations must continue evolving to fit a larger range of aid initiatives (Stern et al., 2012) or that the use of randomized controlled trials (Camfield & Duvendack, 2014) and attribution analysis (White, 2013) must be expanded to evaluate projects and programs rigorously, development evaluators continue to refine methodological approaches in various respective fields. Evaluations in the field of development remain as diverse and complex as the policy goals and visions that lie underneath the entire domain of aid.

The next section describes the current state of evaluation projects in the Japanese development community and how evaluations in this community produce the familiar struggle that exists between the importance of maintaining evaluator impartiality through independence (as espoused by the OECD evaluation principles discussed above) and the implications this can have on fostering ownership and producing useful evaluation results (cf. Patton & Gornick, 2011).

Evaluation Policy and Practice in Japan

Excerpts of Japanese evaluation policy language reveal consistency with the principles of impartiality advocated for in international policy by emphasizing the recognition of these principles as “important” directly in organizational guidelines:

JICA recognizes that the most important elements in … evaluation are the unbiased impartial attitude and ethical awareness of the evaluators, and performs its evaluation to ensure them. (JICA, 2014, p. 2)

First note that this passage (and JICA’s guidelines for operations evaluations in its entirety) does not refer to independence. While policy and program evaluations (by Japan’s MOFA) are mainly conducted by external third parties or jointly with institutional partners, JICA evaluations focus on individual projects or themes (MOFA Japan, 2013). Furthermore, a look into the number and type of JICA evaluations reveal a significant number of nonindependent processes. Table 3 outlines the total number and type of JICA project evaluations conducted in fiscal year 2012. The bolded numbers indicate evaluations conducted through internal procedures. Internal evaluations in JICA refer to evaluations that place a JICA official as head (evaluation chief) of an evaluation mission.

Table 3.

Number and Type of Evaluations Conducted by JICA in FY 2012 (JICA, 2013).

Project Type	Evaluation Type
Project Type	Ex-Ante Evaluation	Mid-Term Evaluation	Terminal Evaluation	Ex-Post Evaluation	Total
Technical coop	103	71	88	68 (48)	330
ODA loan	55	—	—	50	105
Grant aid	96	—	—	60 (34)	156
Total	254	71	88	178	591

Note. Internal evaluations are in bold. JICA = Japan International Cooperation Agency; ODA = Official Development Assistance; FY = fiscal year.

By appointing a JICA official as the evaluation chief, formal organizational protocol for internal evaluations allows JICA staff to authorize the content of these evaluations. Some internal evaluation procedures do require external evaluation consultants to collaborate with implementing stakeholders. However, the ability to authorize evaluation content remains in the hands of JICA. This is in contrast to the separate unit model of many other agencies, where evaluation units are not subordinate to management or operational departments (Foresti, Archer, O’Neil, & Longhurst, 2007). JICA’s annual evaluation report reveals that the large majority of evaluations are conducted in this manner (495 of 591, or 84% of evaluations in 2012). In a study that compares evaluation practices among development agencies, Foresti, Archer, O’Neil, and Longhurst (2007) claim that “most Evaluation Units [in the development community] are either phasing out or significantly reducing their direct involvement in project evaluations” (p. 29) and are instead focusing on independent evaluations (country, policy, thematic, or strategic). We find that this is not the case for JICA, as a large proportion of evaluations are headed and approved by their own staff.

The current mix of internal and external evaluations utilized by JICA can be a source of confusion for the stakeholders involved, as existing literature has revealed how organizations often assume only one approach to be appropriate (Conley-Tyler, 2005). The differences in levels of accountability, ownership, collaboration, and organizational learning that are supported through evaluation schemes are indicative of Foresti et al.’s claim that there is an apparent “disconnect between the rhetoric … of development evaluation, and evaluation practice in development agencies” (2007, p. 9) as well as the “tension [that] exists between the independence of evaluation and its integration with other functions” (2007, p. 19).

In any case, the implementation of such a large number of internal evaluations gives rise to the question of whether or not JICA is sincerely incentivized to report on failures as well as on successful aid initiatives, and whether (or how) impartiality is maintained through internal evaluations. Sen (2002) has argued strongly against such a system of “closed impartiality,” stating that such practices are inherently parochial, as no entity or mechanism is in place to monitor the biases of the group itself. Although external evaluation experts and consultants are contracted by JICA to provide technical advice and assist the evaluation process, scholars have long argued that psychological barriers will even prevent independent auditors (let alone contracted consultants) from exercising true independence (Bazerman, Morgan, & Loewenstein, 1997), as a consequence of competitive interests in securing future employment.

In contrast to such theoretical concerns that argue for the need of independence in maintaining impartiality, a deeper look into higher level national policy language again reveals organizational policies that place substantial control of internal evaluations into the hands of the government agencies regardless of external and independent input into the process. The following passage from the MOFA ODA Evaluation Division recommends the appropriate course of action if stakeholders were to come to a disagreement about evaluation results:

In case where an agreement cannot be made between the evaluation teams and other relevant parties because of their differences in opinion, it is reasonable that the evaluation team, which is responsible for writing the report, makes a final decision considering the benefit of a third party evaluation. However, in order to have practical feedbacks on the policies and implementations, it is important to make an objective and well-balanced evaluation report, reflecting the opinions of MOFA and JICA, which are the receivers of the report, up to some level. (MOFA Japan, 2013, p. 27, emphasis added)

This policy language implies and assumes strongly that for “practical” purposes, an “objective” and “well-balanced” report is one that reflects “the opinions of MOFA and JICA.” The characteristic assumptions that describe Japanese planning and operational units in the event of disagreements over evaluation contents appear to stand in contrast to the “golden rule” of evaluations as understood by development professionals: “that it should be protected from capture by program managers and donors” (Picciotto, 2007, p. 515).

The discussion now aims to help readers better understand how Japanese development professionals perceive these internal procedures through a proposed cultural model. The discussion utilizes an existing discourse analytic definition of cultural models as “every day ‘theories’ … about the world that tell people what is ‘typical’ or ‘normal’ … from the perspective of a particular Discourse” (Gee, 2004, p. 40). Characteristics of the aid evaluation process in Japan that can be categorized as interdependent are presented in the following section through a description of evaluation practices and discourse as described by professionals within the Japanese international development community.

Internal Evaluation Procedures: Evaluating Within a Culture of Interdependence

As successive interviews with Japanese development professionals evolved to focus specifically on the evaluation process, interview questions were refined in an attempt to address the fundamental issues concerning impartial evaluations in the context of the development aid industry. Some of these included questions such as:

JICA takes the lead on several evaluations, which essentially means JICA is evaluating itself. Is JICA truly able to evaluate its own activities impartially?

JICA hires consultants to conduct or provide input for evaluations. Can consultants (companies) be critical of their clients?

What kind of pressures do development consultants and JICA staff face through evaluations? How do these pressures affect the evaluation process?

When asked to explain the legitimacy of internal evaluation mechanisms, evaluation experts and JICA officials explained that the stringent nature of evaluation procedures and methods ensured that evaluation results were protected from individual biases. Subjects argued that the underlying premise of the impartiality of evaluations was that “if evaluation methodology is sound, it does not necessarily matter who does the evaluations” (JICA Evaluation Department staff).

Furthermore, subjects described internal evaluations as a team-based initiative, in which an evaluation team made up of several individuals is tasked to evaluate a project. Respondents claimed that the team-based mechanism protected evaluation results from individual bias by incorporating the input of multiple perspectives. Interview testimonies described a (time-consuming) process by which tentative evaluation results are passed around relevant project stakeholders in an attempt to generate a consensus about the final language that would go into an evaluation report. Subjects felt that the “ideal” evaluation was thus a report that reflected the common understanding and consensus of all members of this interdependent network of stakeholders. As JICA’s previous evaluation guidelines stated very clearly: “It is also very important to follow through with data interpretation … by coming to a consensus about them with the stakeholders” (2004, p. 84). The importance of consensus is emphasized strongly, with the 2004 JICA evaluation guidelines mentioning the term “consensus” a total of 18 times. This emphasis on consensus is more noticeable when compared with evaluation policies and guidelines of other major aid agencies and organizations. For example, the other countries that make up the Top 5 bilateral aid donors along with Japan (i.e., the U.S. Agency for International Development, 2013; U.K. Department for International Development, 2013; Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung [BMZ], 2006; Agence Française de Développement, 2013) all do not mention the term within their evaluation guidelines. The DAC principles of evaluation of Evaluating Development Assistance (OECD, 1991) also does not mention consensus, while the OECD summary of key terms for evaluations does mention consensus once, but in the following context: “… building consensus between the agencies and maintaining effective co-ordination processes can be costly and time consuming” (OECD, 2010b, p. 32).

Interdependency in the Japanese development aid community

This article proposes that the emphasis on consensus which frames internal evaluation processes are remnants of an institutional culture that is perceived as highly interdependent. Indeed, interview subjects from both JICA and development consultants claimed that interorganizational relationships and managing relational tensions represent a major concern throughout the aid process and its evaluation. Subjects from JICA explained how each stakeholder held a vital role in ensuring the success of a project, and consultants described how positive relations with JICA could help ensure future employment. For evaluation scholars, an interdependent culture can be considered a defining (albeit simplified) label to represent the cultural context that complements participatory and inclusive evaluation approaches. In discussing the issues of independence in evaluations, subjects appeared to hold concerns that reflected the philosophies behind the inclusive, participatory, and constructive schools of evaluation (Cousins & Earl, 1992; Guba & Lincoln, 1989; Mertens, 1999). Subjects claimed that when implementing technical cooperation projects, a “completely” external stakeholder, or an “extremely” independent agent, would not necessarily be recognized as a legitimate evaluator but could in fact be perceived as an irrelevant stakeholder intervening in ongoing efforts. Similarly, in discussing the operations of independent auditors, subjects jokingly referred to external authorities as “enemies” and the nature of their work and accompanying work styles were reportedly perceived as significantly hostile. Such experiences shared by the respondents are again illustrative examples of the dialectic tension that exists between an evaluation unit’s need for independent and external evaluations at the threat of being isolated from operational units (Foresti, 2007).

Leif has argued that “accountability is … the opposite of trust: making an agent accountable can signal to that agent that outsiders distrust the agent to fulfil its responsibilities” (2006). The evaluation-related difficulties shared by interview subjects can be described as situations in which the intrinsically critical and “distrustful” nature of an external and independent evaluator would clash with existing group dynamics of a highly collaborative and interdependent network. Within the experiences shared by interviewees, the (perceived) sentiment of distrust that lay behind certain actions by evaluators, or during evaluations, often became the source of various conflicts among project stakeholders. Several respondents shared troubling experiences of implementing parties (development consultants, project managers, or counterpart officials) becoming highly offended at suggestions for improvement. Interviewees (from both JICA and development consultancies) claimed that the act of critically examining a development project’s impacts can often be perceived as a reproach on the actions and efforts of the implementing stakeholders themselves. Development consultants emphasized the need to maintain positive relations with JICA staff in describing past experiences of disagreements over project evaluations (supporting Bazerman et al.’s, 1997, argument mentioned earlier concerning the “impossibility” of independence in a client–seller relationship). The consultants interviewed felt that being too “critical” or “uncooperative” with JICA officials would negatively affect their company’s ability to win future contracts. JICA officials on the other hand described the difficulty of managing interorganizational tensions that occurred between, for example, local beneficiaries, government officials and bureaucrats, development consultants, JICA field offices, and headquarter staff in their attempts to produce a consensus-based evaluation report.

In this manner, interview testimony describing evaluation difficulties were made up of stories discussing the need to maintain positive interorganizational relationships between multiple stakeholders. This can be interpreted as the perceived existence of a highly interdependent network of collaborating stakeholders who are involved in development aid processes. A participatory evaluation approach would appear to demonstrate a suitable fit for such an institutional context, which again raises questions about how impartiality can be strengthened or maintained within such procedures. Just as independent and detached approaches to evaluations diverge with inclusive and participatory approaches, a model of interdependence theoretically clashes with a construct of impartiality that is strengthened through the principle of independence. At the semantic level, independence and interdependence are fundamentally contrary to one another; the dictionary definitions of the two words are literally antonyms. Taking these dialectic definitions into consideration, the existence or acknowledgment of a legitimate independent entity can implicitly reveal questions concerning the legitimacy of an interdependent network of stakeholders. Similarly, the existence of an impartial third party can imply that existing stakeholders are biased or partial to their self-interests. It is thus very plausible that such a dialectic clash of constructs affects the institutional discourse and practices that are meant to reflect these principles. The following section argues that this clash of constructs between evaluation principles and processes in the Japanese development community results in a situated context that contributes to the production of distinct linguistic and institutional evaluation practices. That is, in consistency with a cultural model that normalizes an inclusive and interdependent network of stakeholders, it is proposed that the impartiality of an evaluator among Japanese development professionals can be emphasized through a vertical hierarchy, rather than through principles of independence.

Accommodating for Interdependence: Hierarchical Language Within Evaluation Discourse

The claim that the impartiality of evaluators is legitimized through a vertical hierarchy is supported by the identification of hierarchical characteristics within the language at multiple levels of evaluation discourse. These levels include the individual language user, domestic policy discourse, and international policy discourse. The most notable examples have been found within the utterances of individual subjects, while the observations from the other two levels are presented to triangulate and demonstrate consistency with the observations made at the level of the individual language user. Discourse analysts argue that discourse is action oriented, situated, constructed, and constructive. “Constructive” in that they build versions of the psychological worlds, and “constructed” in that language resources are used to do the building (Potter, 2012). The current article builds on these claims that discourse helps produce institutions as well as “self-regulating mechanisms that shape individual behaviour” (Phillips, Hardy, & Lawrence, 2004, p. 635) by presenting observable evidence of individual utterances and language use that influences and is influenced by the larger institutional context.

Evaluations as a hierarchical cognitive-linguistic category in Japanese

Theoretical concepts from cognitive linguistics allow us to identify the use of vertical schemas in the language of Japanese development professionals when they conceptualize project evaluations. Cognitive linguists argue that a universal characteristic of language use lies in the utilization of schemas to organize the knowledge of concepts (Talmy, 1983). These schemas can be used to structure spatial relations and basic experiences (Lakoff & Johnson, 1980) and to construct cultural models, such as a model of interdependence. Discourse analysts provide definitions that enable us to link cognitive-linguistic schemas to the existence of a cultural model. The utterances that will now be presented contain the use of schemas, which have been defined as a component of the previously mentioned definition of “cultural models” in the following manner:

Cultural models are every day ‘theories’ (i.e., storylines, images, schemas, metaphors, and models) about the world that tell people what is ‘typical’ or ‘normal’, not universally, but from the perspective of a particular Discourse. (Gee, 2004, p. 40; emphasis added)

As the analytical framework describing the link between the hierarchical schemas and the cultural models has been outlined, we will now go through several natural phrases produced by Japanese development professionals that reveal distinct language properties. The following utterances contain the use of hierarchical schemas and metaphors being applied directly to the concept of evaluations among Japanese development professionals:

情報をあげる (jouhou-wo-ageru: “Pass the information up[wards]”).

情報をあげてもらう (jouhou-wo-agete-morau: “Have the information passed up[wards]”).

These two phrases literally translate to the act of “passing information upwards” and were used by subjects to describe the process of information exchange during evaluation processes. Evaluation processes are described in which project members located on-site or in the field are tasked to pass information “upward” to the other agencies so that evaluations can be conducted. In a similar manner, the following phrase has also been identified:

情報を汲みとる (jouhou-wo-kumitoru: “Skim the information”).

This phrase metaphorically describes the act of collecting information as a transfer of liquids. This dead metaphor conceptualizes the complete mass of information required for evaluations as a body of water. Relevant information is conceptualized as being skimmed off the top of the body of information, again spatially placing information gatherers (evaluators) above the source of information (project sites, consultants, counterpart agencies, etc.), while also placing relevant and useful information near the “top” or “surface” of the mass of complete information. The examples illustrate how subjects conceptualize and naturally assume a spatial relation between evaluators and evaluation subjects, in which former is “above” the latter.

Apart from the metaphorical references to spatial hierarchy, one specific utterance reveals a fundamental cognitive–semantic difference in how evaluations are conceptualized by Japanese development professionals (in comparison to the English context). The phrase is:

評価を下す (hyouka-wo-kudasu: “*To hand down an evaluation”; “To evaluate [*downward]”).

This directional verb phrase, 評価を下す (hyouka-wo-kudasu: “To evaluate [downward]), places the Japanese word for evaluations, 評価 (hyouka: “evaluation”), within a distinct cognitive-linguistic category from that of the English equivalent term. The phrase infuses a dimension of hierarchy, authority, and privilege within the act of evaluations, placing it in the same category as verb acts such as 判決 (hanketsu: “judge”), 決定 (kettei: “decide”), or 命令 (meirei: “order, demand”). Utilizing these other verbs, the authoritative or privileged status required of one that can be the agent of hyouka can be illustrated through the principle of substitutability. Substitutability demonstrates how utterances remain grammatical or comprehensible when replaced with other words in the same category (Rosch, 1978). Through substitution, Japanese utterances allow for the phrases hanketsu-wo-kudasu (to hand down a judgment/to pass a judgment), kettei-wo-kudasu (to hand down a decision/to pass a decision), and meirei-wo-kudasu (to hand down an order/to pass an order).

The logic of substitutability reveals that the English word for evaluations cannot be used in the same manner and context as the Japanese hyouka. In English, although one can “hand down a judgment,” “hand down a decision,” or “hand down an order,” an evaluation cannot be “handed down” in the same way that judgments, decisions, or orders are produced from a position of authority. Examples of the substitutability of hyouka can also be demonstrated by observing uses of the action verb “to pass,” which also requires the authority or privilege of the agent. The English phrases “pass a judgment,” “pass a decision,” and “pass an order” place each act of judging, deciding, and ordering within the same category of actions that require a degree of authority. These actions are understood from the agent perspectives of the “judge,” “decider,” and “orderer,” respectively. However, the phrase “to pass an evaluation” forces English speakers to comprehend the sentence as a task- and subject-oriented statement (from the perspective of the “evaluated”), in which evaluations represent an assignment that requires completion. Thus, the sentence to pass an evaluation is interpreted as “to successfully complete an evaluation.”

These actions, which fall under the category of a “verb of judging” (Fillenbaum & Rapoport, 1974; Fillmore, 1969), contain within them tacit assumptions as to the appropriate agent of such acts. The cognitive-linguistic category of the verb hyouka assumes that the appropriate characteristics of an evaluator comprise an entity that is vertically or spatially above the object being evaluated. The fundamental structural differences in the concept of Japanese hyouka and English evaluations reveal how definitions are constructed and accepted to link social cognition with different realities (Semin & Fiedler, 1988).

Vertical hierarchy in domestic and international evaluation policy discourse

While distinct hierarchical cognitive-linguistic traits can be found within the utterances of individual Japanese development professionals, an investigation of secondary data reveals that these hierarchical structures can also be observed in higher level discourse and policy at the domestic and international levels. At the domestic level, the following excerpt can be found in MOFA’s guidelines for ODA evaluations:

Conventionally, the International Cooperation Bureau was in charge of ODA evaluations … however … the ODA Evaluation Division was relocated to the Minister’s Secretariat (which is structurally positioned at higher rank than the International Cooperation Bureau) from FY 2011, in order to maintain the independency of implementation of evaluations. (MOFA Japan, 2013, p. 11; emphasis added)

This passage from MOFA policy describes recent changes to the organizational structure of departments involved in ODA evaluations. Fundamental assumptions supporting the phrase are revealing; if one deconstructs the logic of the above statement, the proposed claim is that the ODA Evaluation Division was given a class higher than another Bureau in order to maintain its independent implementation. In other words, this particular policy has been formed based on a logical assumption that a higher class is required to maintain independency (and thus, act impartially). Such an assumption clearly does not exist in many other contexts, and this policy language represents a distinct (and perhaps amplified) example of claims by modernist-constructivist evaluators that political structures and authorities depend on knowledgeable experts being given special powers in evaluative decision-making processes (Guba & Lincoln, 1989). The organizational structure of the ODA Evaluation Division presented in this domestic policy language is consistent with the notion that hyouka requires evaluators who are spatially or hierarchically located “above” their subjects.

Moving beyond the policy language of the Japanese government, and into the international domain, a content analysis of OECD peer reviews also confirmed a tendency for Japanese international development agencies and their evaluations to be described as relatively more hierarchical or vertical in comparison with other aid agencies. The following description of Japanese aid evaluations was published by peer examiners from the United States and the European Commission:

MOFA focuses on policy and programme evaluation, and JICA and JBIC place their primary emphasis on project-level evaluation. While this information tends to rise vertically within the framework of each of the ODA organisations …. (OECD, 2004, p. 53; emphasis added)

Observers from the United States and Europe therefore also suggest that the flow of information during evaluations is vertical in nature, with the information “rising” upward toward the evaluators. From the perspective of an organizational theorist, it is very plausible that descriptions of “vertical” organizational structures are a common pattern and hardly unique to the Japanese context. However, a term-frequency analysis of the terms vertical or “vertically” conducted against 45 other DAC peer reviews of 24 other OECD aid agencies reveals that the terms occur only 23 times (of approximately 1,172,670 words with three or more characters). Furthermore, only 13 references to vertical organizational structure remain after irrelevant contextual uses of the term vertical are excluded, such as the mention of “vertical funds,” or descriptions of organizations other than the aid agency being reviewed. Among the remaining 13 references, the largest proportion of references to vertical organizational structure is found in the reviews of Japan (8). Examples include descriptions of “vertical divisions of labor” and “vertical organizations” among the Japanese agencies. The remaining five examples are found in peer reviews of Germany (three), Switzerland (one), and the European Community (one). Thus, the ODA practices of all 20 other OECD members who have been reviewed by their peers are not described as vertical in the reports. In consistency with a hierarchically schematized concept of evaluations by Japanese development professionals, OECD peer reviews suggest that ODA in Japan (along with Germany, Switzerland, and the European Community), as perceived by the reviewers, are relatively more vertical than the practices of other OECD member countries.

These general findings of the OECD peer reviews at the international level are thus consistent with the logic that can be observed in domestic policies defining the hierarchical structure of Japan’s ODA Evaluation Division. The findings are also consistent with the fundamentally different cognitive–semantic structure of the Japanese concept of hyouka in contrast to the English concept of evaluations. The characteristics and trends identified in international and domestic evaluation discourse thus appear consistent with the cognitive–semantic characteristics of hyouka as a hierarchically schematized construct that requires evaluators to possess authority or privilege.

The distinct hierarchical structural characteristics of evaluations that have been identified within the language of Japanese development professionals can contribute to existing debates in the evaluation literature: whether evaluations need to utilize independent and detached evaluators for the sake of impartial, objective, and credible evaluations or whether the direct inclusion of stakeholders and participant ownership of the evaluation process is required to actualize true institutional learning through the production of practical evaluation results. The analysis finds that in contrast to a notion of evaluator impartiality that is strengthened and maintained simply through principles of independence, Japanese development professionals perceive evaluations (hyouka) to be a hierarchical construct in which evaluator impartiality can be strengthened or legitimized through authoritative or hierarchical means.

Conclusion

This article has examined the evaluation discourse of the Japanese development community, a relatively neglected institutional context within the evaluation literature, to illustrate how conceptual understandings of impartiality can be socially constructed. Evidence has been presented from three levels of evaluation discourse: the individual language user, Japan’s domestic evaluation policy language, and international (OECD) aid reports. Observations of the natural language use of Japanese development professionals reveal the use of hierarchical schemas to conceptualize hyouka as a vertically structured cognitive-linguistic category. Consistent hierarchical patterns and linguistic characteristics have also been presented from domestic evaluation policy and international reviews of evaluation processes. Evaluation discourse that formally conceptualizes the “higher” position of the evaluator (relative to the subject being evaluated) represents observable evidence of a distinct social construct of impartiality among development professionals and evaluators in Japan. The discussion argues that the hierarchical cognitive-linguistic constructs and linguistic tendencies affecting evaluation discourse can be interpreted through a cultural model of interdependence prevalent in the current case. Within a cultural model of interdependence, a purely independent evaluator will not necessarily be perceived as impartial and may lead to evaluators being treated as “outsiders.” To accommodate for the threat of isolation, it is proposed that where independence can be sacrificed in participatory, inclusive, and joint evaluations among interdependent stakeholders, development professionals in Japan can utilize hierarchical schemas to strengthen evaluator impartiality by leveraging hierarchical social values such as authority, seniority, or privilege.

This article has examined evaluations as the institutional practice that is designed and perceived to be impartial by development professionals. The findings contribute to a discussion on the concept of impartiality itself as well as to an understanding of how impartiality can be socially constructed through organizational processes. The analysis also produces systematic findings to support the intuitive claims of experienced evaluators that observe how cultural differences can affect communication norms during evaluations (e.g., Patton, in King & Greenseid, 2007). As the current article tries to demonstrate, these differences can be a product of perceptual gaps about the nature of institutional practices as well as a fundamentally different structural understanding of the concept of evaluations (vs. hyouka) itself. The assumed hierarchies between evaluators and the project or program stakeholders that exist among Japanese development professionals have significant implications for the communicative styles and strategies for stakeholders and evaluators in Japan’s international development community.

For the development aid community at large—an inherently interorganizational and international domain—deep-rooted challenges indeed await evaluators to produce impartial, unbiased, and practical evaluation results in a multitude of cultural contexts. Even with strongly defined international policy language that link principles of impartiality with independence, as this study reveals, fundamental differences in how these principles are framed, constructed, and understood across institutional cultures can and do exist. Continued efforts to expose and build a deeper understanding of the differing knowledge structures of key evaluation concepts, and the tacit assumptions they may generate, remain an important and essential area of exploration for evaluators in the development community and in other fields.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Agence Française de Développement. (2013). AFD Evaluation Policy, 2013. Retrieved from http://www.afd.fr/webdav/site/afd/shared/RECHERCHE/Evaluation/document/AFD_evaluation_policy.pdf

Barry

(1995). Justice as impartiality. New York, NY: Oxford University Press.

Bazerman

M. H.

Morgan

K. P.

Loewenstein

G. F.

(1997). Opinion: The impossibility of auditor independence. Sloan Management Review, 38, 89–94.

Bohm

(1996). On dialogue. In Nichol

(Ed.). London, England: Routledge.

Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung (BMZ). (2006) Evaluierung der Entwicklungszusammenarbeit. “Evaluierungskriterien” (Evaluation Criteria). Retrieved from http://www.bmz.de/de/zentrales_downloadarchiv/erfolg_und_kontrolle/evaluierungskriterien.pdf

Burgess

Glaser

Yvsyukova

(1997). Transformative approaches to conflict. Retrieved from http://www.colorado.edu/conflict/transform/

Camfield

Duvendack

(2014). Impact evaluation—Are we ‘off the gold standard?’ European Journal of Development Research, 26, 1–11.

Chelimsky

(1995). Where we stand today in the practice of evaluation: Some reflections. Knowledge and Policy, 8, 8–19.

Chelimsky

(1997). The political environment of evaluation and what it means for the development of the field. In Chelimsky

Shadish

(Eds.), Evaluation for the 21st century: A handbook (pp. 53–68). Thousand Oaks, CA: Sage.

10.

Conley-Tyler

(2005). A fundamental choice: Internal or external evaluation? Evaluation Journal of Australasia, 4, 3.

11.

Cousins

J. B.

Earl

L. M.

(1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397–418.

12.

Cullen

A. E.

Coryn

C. L.

Rugh

(2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32, 345–361.

13.

Datta

L. E.

(1999). The ethics of evaluation neutrality and advocacy. New Directions for Evaluation, 1999, 77–88.

14.

Datta

L. E.

(2000). Seriously seeking fairness: Strategies for crafting non-partisan evaluations in a partisan world. American Journal of Evaluation, 21, 1–14.

15.

Department for International Development of the United Kingdom. (2013, 5). International Development Evaluation Policy. Retrieved from https://www.gov.uk/government/publications/dfid-evaluation-policy-2013

16.

Fillenbaum

Rapoport

(1974). Verbs of judging, judged: A case study. Journal of Verbal Learning and Verbal Behavior, 13, 54–62.

17.

Fillmore

C. J.

(1969). Verbs of judging: An exercise in semantic description. Research on Language & Social Interaction, 1, 91–117.

18.

Fishman

D. B.

(1992). Postmodernism comes to program evaluation: A critical review of Guba and Lincoln’s fourth generation evaluation. Evaluation and Program Planning, 15, 263–270.

19.

Foresti

Archer

O’Neil

Longhurst

(2007). A comparative study of evaluation policies and practices in development agencies. Report for AFD Evaluation Department. London, England: Overseas Development Institute and Agence Française de Développement.

20.

Friedman

(1989). The impracticality of impartiality ( Eighty-Sixth Annual Meeting American Philosophical Association, Eastern Division ). The Journal of Philosophy, 86, 645–656.

21.

Gee

J. P.

(2004). Discourse analysis: What makes it critical? In Rogers

(Ed.), An introduction to critical discourse analysis in education (pp. 19–50). Mahwah, NJ: Lawrence Erlbaum.

22.

Glaser

Strauss

(1967). The discovery of grounded theory. New York, NY: Aldin.

23.

Greene

J. C.

(1997). Evaluation as advocacy. American Journal of Evaluation, 18, 25–35.

24.

Guba

E. G.

Lincoln

Y. S.

(1989). Fourth generation evaluation. Beverly Hills: Sage.

25.

Guijt

Roche

(2014). Does impact evaluation in development matter? Well, it depends what it’s for! European Journal of Development Research, 26, 46–54.

26.

International Federation of Red Cross and Crescent Societies. 2014. The Seven Fundamental Principles. Retrieved from http://www.ifrc.org/who-we-are/vision-and-mission/the-seven-fundamental-principles/

27.

Japan International Cooperation Agency. (2004). JICA Guideline for Project Evaluation, Practical Methods for Project Evaluation. Office of Evaluation, Planning and Coordination Department. Retrieved from http://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/guides/pdf/guideline01-01.pdf

28.

Japan International Cooperation Agency. (2013). Annual Evaluation Report 2013: Inclusive and Dynamic Development. Evaluation Planning Division. Retrieved from http://www.jica.go.jp/english/our_work/evaluation/reports/2013/index.html

29.

Japan International Cooperation Agency. (2014, 5). Guidelines for operations evaluations. JICA Evaluation Department. Retrieved from http://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/guides/

30.

King

Greenseid

(2007). The oral history of evaluation, Part 5: An interview with Michael Quinn Patton. American Journal of Evaluation, 28, 102–114.

31.

Lakoff

Johnson

(1980). Metaphors we live by. Chicago, IL: University of Chicago Press.

32.

Leif

(2006). Accountability in international development aid. Ethics and International Affairs, 20, 1–23.

33.

Mayer

(2004). Beyond neutrality: Confronting the crisis in conflict resolution. San Francisco, CA: Jossey-Bass.

34.

Mertens

D. M.

(1999). Inclusive evaluation: Implications of transformative theory for evaluation. American Journal of Evaluation, 20, 1–14.

35.

Ministry of Foreign Affairs Japan. (2013). ODA Evaluation Guidelines. ODA Evaluation Division, Minister’s Secretariat. Retrieved from http://www.mofa.go.jp/policy/oda/evaluation/basic_documents/index.html

36.

O’Neill

(1995). Implementation frailties of Guba and Lincoln’s fourth generation evaluation theory. Studies in Educational Evaluation, 21, 5–21.

37.

Organisation for Economic Co-operation and Development. (1991). OECD DAC principles for the evaluation of development assistance. Paris, France: Author. Retrieved from http://www.oecd.org/development/evaluation/2755284.pdf

38.

Organisation for Economic Co-operation and Development. (2004). Peer review of Japan. OECD-DAC report. Paris, France: Author. Retrieved from http://www.mofa.go.jp/policy/oda/category/coordinate/oecd2004.pdf

39.

Organisation for Economic Co-operation and Development. (2005). The Paris Declaration on Aid Effectiveness. Paris, France: Author. Retrieved from http://www.oecd.org/dac/effectiveness/34428351.pdf

40.

Organisation for Economic Co-operation and Development. (2010a, 2). Quality standards for development evaluation. DAC guidelines and reference series. Paris, France: Author. Retrieved from http://www.oecd.org/development/evaluation/qualitystandards.pdf

41.

Organisation for Economic Co-operation and Development. (2010b, 6). Evaluating development co-operation: Summary of key norms and standards. Paris, France: Author. Retrieved from http://www.oecd.org/dac/evaluation/dcdndep/41612905.pdf

42.

Organisation for Economic Co-operation and Development. (2010c, 12). Evaluation in development agencies—Better aid. Paris, France: Author. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.1787/9789264094857-en

43.

Patton

M. Q.

(1994). Developmental evaluation. Evaluation Practice, 15, 311–319.

44.

Patton

M. Q.

Gornick

(2011). Evaluation of the Phase 2 evaluation of the Paris declaration: An independent review of strengths, weaknesses, and lessons. Saint Paul, MN: Utilization-Focused Evaluation. Retrieved from http://www.oecd.org/dac/evaluation/dcdndep/48620425.pdf

45.

Phillips

Hardy

Lawrence

(2004). Discourse and institutions. Academy of Management Review, 29, 635–652.

46.

Picciotto

(2005). The evaluation of policy coherence for development. Evaluation, 11, 311–330.

47.

Picciotto

(2007). The new environment for development evaluation. American Journal of Evaluation, 28, 509–521.

48.

Potter

(2012). Discursive psychology and discourse analysis. In Gee

J. P.

Handford

(Eds.), The Routledge handbook of discourse analysis (pp. 104–119). New York, NY: Routledge.

49.

Rosch

(1978). Principles of categorization. In Roach

Lloyd

(Eds.), Cognition and categorization (pp. 27–48). Hillsdale, NJ: Lawrence Erlbaum.

50.

Sambrook

(2012). Delivering trust: Impartiality and objectivity in the digital age. Oxford, UK: Reuters Institute for the Study of Journalism. Retrieved from http://reutersinstitute.politics.ox.ac.uk/publication/delivering-trust-impartiality-and-objectivity-digital-age

51.

Scriven

(1997). Truth and objectivity in evaluation. In Chelimsky

Shadish

(Eds.), Evaluation for the 21st Century: A handbook (pp. 477–500). Thousand Oaks, CA: Sage.

52.

Semin

G. R.

Fiedler

(1988). The cognitive functions of linguistic categories in describing persons: Social cognition and language. Journal of Personality and Social Psychology, 54, 558–568.

53.

Sen

(2002). Open and closed impartiality. The Journal of Philosophy, 99, 445–469.

54.

Stern

Stame

Mayne

Forss

Davies

Befani

(2012). Broadening the range of designs and methods for impact evaluations. London, UK: DFID. Retrieved from http://r4d.dfid.gov.uk/pdf/outputs/misc_infocomm/DFIDWorkingPaper38.pdf

55.

Talmy

(1983). How language structures space. In Pick

H. L.

Acredelo

L. P.

(Eds.), Spatial orientation—Theory, research, and application (pp. 225–282). New York, NY: Plenum Press.

56.

United Nations. (2014). Principles of UN peacekeeping. Retrieved from http://www.un.org/en/peacekeeping/operations/principles.shtml

57.

U.S. Agency for International Development. (2013). Evaluation at USAID. November 2013. Washington, DC: Office of Learning, Evaluation and Research, USAID Bureau for Policy, Planning and Learning.

58.

United Nations Office of the High Commissioner for Human Rights. (2003). Human Rights in the Administration of Justice: A Manual on Human Rights for Judges, Prosecutors and Lawyers. Professional Training Series No. 9, In Cooperation with the International Bar Association. New York, NY: United Nations Publication.

59.

White

(2013). An introduction to the use of randomised control trials to evaluate development interventions. Journal of Development Effectiveness, 5, 30–49.