Evaluative thinking

Abstract

Evaluative thinking is a core skill used by evaluators. It brings evaluation practice to a higher level of sophistication and utility and helps to ensure that evaluation findings, interpretations and recommendations are contextualised, evidence-based and practical. This article draws upon decades of the work of evaluators who have published on thinking critically and evaluatively. It identifies and explores the five primary elements of evaluative thinking – critical, contextual, creative, practical thinking and reflective practice – and offers a practical framework for examining and applying these elements of evaluative thinking. It also distinguishes evaluative thinking from evaluation planning and implementation, which it precedes and guides. Finally, it proposes that evaluative thinking should be examined and learned intentionally as a core evaluation skill just as, for example, various methodologies or methods of analysis are taught.

Keywords

Theory evaluative thinking evaluation skills critical thinking evaluation planning

What we already know:

• The importance of critical thinking as an intellectual tool of evaluation has been defined, described and discussed in the literature

• Much has been written on the importance of evaluative thinking, its features and its applications

The original contribution the article makes to theory and/or practice:

• This article brings together the major discussions and descriptions of evaluative thinking

• It provides a framework to better examine and consider the characteristics and usage of evaluative thinking

• Provides a springboard for further debate and development of these concepts

Introduction

Evaluation is characterised as a transdiscipline that harnesses a diversity of methods and frameworks (Lemire et al., 2020). It intersects with and draws upon multiple disciplines to craft effective evaluative approaches for examining complex social problems and assessing their proposed solutions. It requires sophisticated skills in analysis and problem-solving. It also involves transformational thinking operationalised through collaborative, reflective and dialogic approaches (Preskill & Torres, 2000) or the social construction of meaning. Evaluative thinking is an essential set of skills for evaluators.

This exploration of evaluative thinking draws upon decades of evaluators' published work on thinking critically and evaluatively and weaves it into a practical framework for examining and applying evaluative thinking. In doing so, it invites the reader to consider that evaluative thinking is distinctly part of evaluation and that evaluators, in general, are distinguishable by their application of evaluative thinking. This paper presents a schema for evaluative thinking based on a fusion of the ideas in the published literature and integrates decades of the work of evaluators who have published their research and reflections on a topic which is foundational to current and emerging good practice in evaluation. This author makes no claims to the generation of these concepts, only of bringing them together in a practical framework for better examination and application. This article also draws on personal reflections on 20 years of experience in program design and evaluation practice in many different countries.

Further, these observations and interpretations are examined and shared through the lens of my positionality. From this base of theory and practice, literature and experience, the paper cautiously uses inductive reasoning to elucidate five elements that characterise evaluative thinking and offers a framework to facilitate their application. In doing so, this paper could be described as an example of evaluative thinking. Before presenting the framework, however, it is useful to briefly consider the history of evaluative thinking and some associated concepts.

The development of evaluative thinking

The evolution of evaluation and evaluative thinking has benefitted from a multitude of underpinning influences from fields as diverse as philosophy, mathematics, statistics, social research and the health sciences. These disciplines offer a rich array of epistemological concepts, perspectives and understandings that were harvested by early evaluators for analysis, problem-solving and making judgements of value and utility. These were used by early evaluators to lay the foundations for critical thinking in evaluation, which in turn has provided the groundwork for the development of evaluative thinking.

Many evaluators have written about the importance of evaluative thinking and have contributed to its examination and evolution. For example, Scriven’s (Scriven & Paul, 1987, Scriven, 2007) work in logic, reasoning and critical thinking has been extensive and foundational to the development of evaluative thinking. Weiss (1988, 1998) described an ‘evaluative cast of mind’ promoted through collaborative approaches to evaluation. She characterised it as sceptical questioning, thinking critically about why a program operates as it does, and reflecting on one’s practice. Weiss also envisaged such thinking as transcendent in that it created a sustainable benefit that continued to develop beyond the life of an individual program. She argued that such a thinking practice could produce information systems that continue to provide information beyond the initial evaluation and allow the continuous review of progress. Katz (2002) described the ‘evaluation habit of mind’ as characterised by data-driven goal-related inquiry, systematic reflection and planning, and an evaluation-minded culture. Schwandt (2018, p. 127) also talked about the Evaluative Thinker and described evaluative thinking as being ‘…about how we arrive at or account for judgments about value or quality… [and]… an ongoing process of questioning, reflecting, learning, and modifying’.

Unfortunately, often published definitions of evaluative thinking are circular, that is, they describe evaluative thinking as thinking that occurs within an evaluation or as pondering about the evaluation process. Many definitions are actually descriptions that do not directly define evaluative thinking but instead talk about how evaluative thinking manifests (e.g. evaluative thinking is the way we clarify goals), is used (e.g. evaluative thinking is using evidence to test assumptions), or by its indicators (e.g. we know evaluative thinking has occurred when we see people reflect on evaluation findings). But when evaluative thinking is directly defined, it is consistently described as a form of applied critical thinking. Lu et al. (2019) definition is a typical example. They define evaluative thinking as ‘…critical thinking applied in the context of evaluation...’ The rest of their definition is more about the ‘how’ of evaluative thinking rather than what it is, that is, they then mention how it is guided ‘…motivated by an attitude of inquisitiveness and a belief in the value of evidence…’ and how it is done ‘…identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action’ (Lu et al., 2019, p. 69). Critical thinking is widely considered the basis of evaluative thinking, but evaluative thinking is not only critical thinking. Patton, for example, has made many recent contributions to the discussions of evaluative thinking. Particularly pertinent for this discussion is his distinction that evaluative thinking was inclusive of but much broader than critical thinking (Patton, 2018).

A strategic model of evaluative thinking

Before proceeding, let’s address the circularity of many published definitions of evaluative thinking: evaluative thinking is the thinking evaluators do during an evaluation. Evaluative thinking can and does occur before and separately from undertaking an evaluation.

Essentially, evaluative thinking is evaluativist-level thinking uncoupled from and often preceding formal evaluation planning and practice. A similar divergence between thinking and planning was explored in Mintzberg’s research into management and business strategy, in which he concluded that strategic thinking and strategic planning were two quite distinct processes, even going as far as to say, ‘…strategic planning is not strategic thinking. Indeed, strategic planning often spoils strategic thinking...’ (1994, p. 107). This view sees strategic planning as a rational step-by-step exercise, requiring precision, analysis and establishing of systems and procedures, much like evaluation planning. Conversely, strategic thinking is creative, dynamic and innovative, synthesising learning into a vision (Mintzberg & Waters, 1985; Mintzberg, 1994; Graetz, 2002). This concept was further elucidated in Voros’ (2001) three-stage strategic thinking and planning model, in which he analysed the distinctions and different thinking skill sets required for each of the stages of strategic thinking, strategic development and strategic planning. This pragmatic separation between the conceptual and practical stages could be equally applied to evaluative thinking and evaluation planning, that is, essentially separating evaluative thinking from evaluative doing. It also helps in understanding the different stages and thinking styles required in evaluative thinking, design, planning and implementation and may offer a valuable way of understanding and guiding the evaluation process. As with strategic thinking, evaluative thinking is purposefully contemplative and can occur before and in preparation for planning and implementation. The following table has been adapted from Voros’ strategic thinking and planning model and shows the differences between evaluative thinking and evaluative doing Table 1.

Table 1.

Differentiating between evaluative thinking and evaluative doing.

Evaluative thinking	Exploring options	The thinking required for this activity needs to be a mix of methods, but at this stage is predominantly synthetical and inductive rather than analytical and deductive. Evaluative thinking is concerned with exploration and options, not the steps needed to implement actions (the scope of evaluative planning)
Evaluation development	Making decisions and setting directions	The focus is on assessing options, examining choices, decision-making and/or setting a destination
Evaluative planning	Implementing actions	Requires thinking, which is strongly analytical, logical and deductive – for example, breaking down a goal or objective into steps, designing how the steps may be implemented, estimating the anticipated consequences of each step, and measuring how progress is being made to ensure that things stay on track

Voros explains this three-part process produces tangibles, such as planning documents, and intangibles, like the insights generated in the processes of synthesis and interpretation. Furthermore, he claims these intangible insights are possibly the most important gains because they enhance the development of a mindset of expanded perception of the possible evaluative options.

The five elements of evaluative thinking

Patton explains that evaluative thinking is more than just critical thinking: ‘Learning how to think evaluatively is learning how to learn and think critically… rigorous evaluative thinking combines critical thinking, creative thinking, inferential thinking, and practical thinking’. (2018, p. 21). In addition, Lemire and colleagues (2020) emphasise the importance of contextualisation, inferential thinking and reflection on the evaluator’s values, position, biases, and those of key stakeholders. Evaluative thinking involves interactions between multiple perspectives; commitment to the representation of truth in context; creatively navigating uncertainty, ambiguity and complexity; and the investigation of one’s own as well as others’ positionality, assumptions, motivations and biases (Vo et al. 2018, p. 44). It requires critically analysing evidence while assessing its credibility, contextualising varied perspectives to construct meaning, creatively evoking new perspectives and interpretations, pragmatically applying real-world criteria to interpreting findings and their implications and reflectively accounting for one’s position, preconceptions and predispositions. Five distinctive thinking styles are in play and inter-play during evaluative thinking: critical thinking, contextual thinking, practical thinking, creative thinking and reflective practice. The following figure offers a visual depiction of the five distinctive thinking styles that comprise evaluative thinking: Critical thinking, contextual thinking, practical thinking, creative thinking and reflective practice. The arrows in the figure suggest dynamic connection and inter-play but do not indicate direction Figure 1.

Figure 1.

The five distinctive thinking styles that comprise evaluative.

The schema described in this paper outlines each of the five elements that comprise evaluative thinking in a straightforward overview that will be followed by a richer description in the text. Each thinking style has distinct characteristics and utility and will be discussed in some detail. However, De Bono’s (1969, 2010) work on thinking provides a helpful lens through which to better appreciate some of these thinking styles' features and uses. De Bono made a distinction between what he called vertical and lateral thinking. He described logical and critical thinking as vertical thinking, which proceeded in direct sequential steps, each of which must be justified and valid in order to reach a correct solution.

On the other hand, in lateral thinking, one may intentionally divert to tangentially related or unrelated streams of thought, and one may have to be wrong at some stage to find a correct or useful solution. Both approaches are necessary and complementary. De Bono explains that lateral thinking enhances the effectiveness of vertical thinking while vertical thinking tests and develops the ideas generated by lateral thinking. De Bono’s work offers an approach to better understand and practise evaluative thinking. De Bono’s perspective suggests that evaluative thinking is comprised of the vertical thinking styles and sub-styles such as critical thinking, logic, logical inference and practical thinking. It also incorporates the lateral thinking styles of creative thinking, contextual thinking and reflective practice. The following table presents a schema that defines and summarises each of the five elements that comprise evaluative thinking Table 2.

Table 2.

A schema defining and summarising each of the five elements of evaluative thinking.

The five elements of evaluative thinking
Elements	Key features	Examples
Critical thinking	• Testing theories and identifying assumptions with stakeholders • Asking sceptical and clarifying questions • Analysing evidence while assessing its credibility • Using inference to cautiously draw conclusions • Purposefully explaining evidential, conceptual, methodological, and criteriological considerations	• Developing methodology and methods (Blundell, & Costa Dias, 2000) • Analysing data (Al-Salim et al., 2022) • Testing a hypothesis or a program theory (Waltz, 1997; McLaughlin & Jordan, 1999)
Practical thinking	• Involves both the practical – of doing or implementation and use, and the pragmatic – a realistic approach to dealing with the inevitable imperfections of all aspects of the evaluation • Diligently integrating theory and practice • Inviting interpretations and applications from non-researchers • Applying real-world criteria to interpreting findings and their implications • Consider practical criteria like understandability, meaningfulness, and cost implications	• Evaluation planning (Owen, 2020) • Formulating concrete evaluation questions, for example, how will this improve our program implementation? (Juma & Onkware, 2015; Ratan et al., 2019) • Highly pragmatic evaluation approaches include rapid evaluations used in international humanitarian action (Puri et al., 2015; Vindrola-Padros et al., 2021) • Utilisation focussed evaluations also direct practical and pragmatic thinking to enhance the utilisation of the findings (Patton, 2008)
Contextual thinking	• Harmonising multiple perspectives to map the territory • Locating the evaluand within its broader PESTEL setting • Linking the evaluation to the research and theory • Enhancing the meaning of findings and their interpretation • Evaluation context is socially constructed and negotiated with multiple perspectives and concerns	• Situational analysis (Clarke, 2007; Clarke et al., 2016) • Political economy analysis (Fritz et al., 2009; Unsworth, 2009; Haider & Rao, 2010) • Inclusion of systems and structures to ensure cultural awareness and safety (Gollan & Stacey, 2021)
Creative thinking	• Promoting in-depth exploration of questions, flexibility in thinking and opportunities for discovery • Inviting new perspectives and interpretations • Devising or using new and inventive techniques to analyse, visualise and present data • Redefining a problem or a goal in alternative ways and finding new solutions • Developing novel methods or employing creative practices to increase engagement in the evaluation process	• Developing a program theory (theory of change or program logic model) (Rogers, 2008; Funnell & Rogers, 2011) • Recreating baselines, for example, from secondary sources (Bamberger & Mabry, 2019) • Creative evaluation promotes thinking creatively about conducting evaluations by using innovative data collection techniques and applying evaluation in creative settings like art, education, and teaching (Christou et al., 2021)
Reflective practice	• Understanding how our (and others') influence is filtered by our unique experiences, interests, theoretical understandings, values, and beliefs • Acknowledging our position (socio-economic status, education, age, ethnicity, gender, etc.) precedes our preconceptions, preferences, and biases • Accounting for the evaluator’s ontological and epistemological perspectives • Being aware and examining the evaluator’s own (and stakeholders') thinking or metacognition to recognise and account for their influence • Self-reflection must also lead to self-correction to produce sustainable learning and practice improvements	• Capacity building and capability building (Cole, 2022) • Identifying and mitigating belief preservation or confirmation bias and other threats to validity (Chen & Rossi, 1987; Klayman, 1995) • Using Lessons Learned to design more efficient and effective programs (Patton, 2001)

PESTLE: political, economic, socio-cultural, technological, legislative and environmental.

Critical thinking

Logic and logical inference are foundational to critical thinking. Logical reasoning involves working step by step from a premise(s) to arrive at a valid solution or conclusion, and each step in the logical argument must be correct to reach a valid solution or conclusion. With logical reasoning, two different people would arrive at the same conclusion if given exactly the same information.

On the other hand, critical thinking goes beyond the presented facts. It involves questioning. It includes scrutinising the accuracy, reliability and completeness of the information presented, examining the underlying assumptions and considering how the interests or motivations of those providing that information may have shaped their claims or arguments.

In 1990, the American Philosophical Association convened a panel of 46 experts to determine how critical thinking should be defined and conceptualised. This Delphi panel defined critical thinking as:

‘...purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based’ (Facione, 1990, p. 3).

Critical thinking is the process of testing the accuracy and reliability of the information to confirm its true value and judging information based on the balance of probabilities. This approach includes an open process of questioning and testing the veracity of the information, for example, by playing Devil’s Advocate. Generally, but not always, critical thinking uses deductive reasoning. On the other hand, evaluative thinking generally uses both deductive and inductive with an emphasis on the latter. However, the latter is often emphasised throughout the literature on thinking evaluatively and is worth exploring a little further.

Scriven (2007, p. 1) states, ‘Evaluation is the process of determining merit, worth, or significance; an evaluation is a product of that process’. And yet, paradoxically, perhaps, when we use evaluative thinking, we must begin by suspending judgements. We must resist the rush to solutions. According to Senge (1990, as cited in Rallis & Rossman, 2000), learning organisations, in which people continually expand their capacity to create results, must be willing to suspend judgements and surface concealed assumptions, describe them and assess their plausibility. Even earlier, John Dewey (1910, as cited in Buckley et al., 2015, p. 4) stated, ‘The essence of critical thinking is suspended judgment , and the essence of this suspense is inquiry to determine the nature of the problem before proceeding to attempts at its solution’.

Two decades of evaluation work have underscored for me that the road to hell is paved with unexamined assumptions. So, an essential aspect of critical thinking in evaluation involves identifying, making explicit, testing, accounting for, and, where possible, mitigating those assumptions around which programs and program evaluations are designed and implemented (Patton, 2018; Buckley et al., 2015; Rallis & Rossman, 2000; Ennis et al., 1985). Brookfield's (1987, 2012 as cited in Archibald et al., 2016) offers a three-step process for working with assumptions that recommends: identifying explicit and implicit assumptions, considering plausible alternative explanations and applying evidence to test those assumptions and alternative explanations. Archibald et al., 2016).

Practical thinking

Practical thinking involves both the practical and the pragmatic. It entails the practical concerns of doing (or evaluation implementation and use) ensuring, for example, that a design is fit to achieve its defined purpose. However, it also incorporates pragmatism, a realistic approach to dealing patiently and meticulously with the inevitable imperfections of all aspects of the evaluation, from imperfect baselines and data systems to limited resources, time or managerial support.

Practical thinking calls for assiduously … examining the real-world implications of findings. It also involves inviting observations and interpretations from non-researchers (e.g. community members, program staff and participants) who can and will apply to the data what ordinary people refer to as ‘common sense’; and applying real-world criteria to interpreting the findings criteria like understandability, meaningfulness, cost implications and implications to address [the defined problem and assess the designed solution] (Patton, 2018, p. 23).

In other words, being able to subject any or all steps in the process of evaluation design, implementation and reporting to the iconic Australian pub-test,¹ the ultimate arbiter of common sense, for example, as a test to ensure that both the evaluation and its findings are capable of being applied.

Many evaluation approaches are highly pragmatic. For example, rapid evaluation approaches are used in evaluating international humanitarian action. On the other hand, utilisation focussed evaluations also highlight practical and pragmatic thinking in that they are planned and conducted to enhance the utilisation of the findings and leverage the evaluation process to improve performance.

Contextual thinking

The evaluator, evaluation stakeholders, evaluand and evaluation all interact with, affect, and are affected by context. Contextualisation guides the evaluation approach, links it to the relevant body of research and locates it within the political, economic, sociocultural, technological, legislative and environmental (PESTLE) setting of the evaluand (Rastogi & Trivedi, 2016; Kolios & Read, 2013). Contextualisation gives credibility to and enhances the meaning of an evaluation from its design to interpreting and reporting on the evaluation findings. Contextualisation maps boundaries and enhances the meaning of findings and their interpretation (Van Oers, 1998). Locating the evaluation data, findings and interpretations within context is essential to evaluation quality and utility, that is, determining the evaluation approach, framing the key evaluation questions, selecting the designs and methods and communicating our findings (Van Oers, 1998).

Despite repeated government calls for well-evaluated evidence-based policy, there has long been recognition that political influence in decision-making can be a feature of every step of an evaluation, from the initial decision to evaluate to the dissemination of its recommendations (Chelimsky, 1987; Weiss, 1988; Weiss, 1993; Rallis & Rossman, 2000; Guenther et al., 2010). So, Markiewicz (2008) advises evaluators to develop an astute appreciation for the political context within which they operate, the skills to negotiate the boundaries of that relationship with a range of stakeholders, and the ability to maintain their credibility, which in turn is substantially impacted by perceptions of the evaluator’s independence and objectivity. Politics, though important, is only one of many contextual domains that influence evaluation and evaluators.

An evaluation must be contextualised and anchored in theory as well as practice. We can contextualise an evaluation in at least three concentric levels of systems, the Micro, Meso and Macro-levels, in which, for example, the Micro-level could be the program implementation sites, and the Meso-level could be the whole of the program (or policy portfolio within which the program is located), and the Macro-level could include the broader regional or national socio-economic and environmental context of the policy (Bolíbar, 2016; Shehadeh, 2020; Serpa & Ferreira, 2019).

Ensuring context is one of the specific issues that an evaluator regularly checks to improve the chances that significant contextual factors will be identified early and their implications incorporated into the evaluation process (Conner et al., p. 103). There are many approaches; for example, Rog’s comprehensive Context Assessment Framework (Rog, 2012; Rog, Fitzpatrick, & Conner, 2012) uses 35 fields to assess context across five domains: (a) the nature of the problem or phenomenon, (b) nature of the intervention, (c) the broader environment of the intervention, (d) parameters of the evaluation and (e) the broader decision-making arena, examining each through the lenses of their physical, organisational, social, cultural, traditional, political and historical dimensions.

Context affects the implementation and outcomes of the evaluation. An efficacious evaluation undertakes a situational analysis to sufficiently understand the situation of the program and participants (Patton, 2008); be sensitive to the socio-political context and how it affects the use of the evaluation information (Weiss, 1993); and guide how to best disseminate and communicate the evaluation and promote its use (Alkin, and King 2017). Understanding the nature of the problem, its scope and boundaries, the context that generates and maintains it and who is affected how is essential to designing an optimal or simply suitable evaluation methodology (Rog, 2012). Of course, a sufficiently systematic and comprehensive assessment of all possible contextual variables may approach an infinite number, depending on how finely they are characterised. So, choices must be made, and the boundaries so drawn are socially constructed.

The boundaries of the problem of interest, the proposed policy solution and the programs that implement those policies comprise the evaluand. They also demarcate the socio-economic, environmental and other key features of the landscape within which the policy, program and intended beneficiaries are situated. The lines that chart that map are negotiated by the key stakeholders as they consider, question and critique options and alternatives and make decisions. So, there is a consensus rather than a single correct answer as to how the boundaries of an investigation are defined. The evaluation context is socially constructed.

Creative thinking

In creative thinking, imagination is applied in the deliberation of possibilities and alternatives of thinking or action. It involves two distinct parts ‘…a generative component facilitating the production of novel ideas and an evaluative component enabling the assessment of their usefulness’ (Ellamil et al., 2012, p. 1). Creative thinking encourages us to examine various possible answers by deviating from the traditional belief that one question can only have one answer. Instead, creative thinking fosters the exploration of the question in greater depth, promoting flexibility in our thinking and providing an opportunity for discoveries.

Creative thinking indicates a broader application of creativity that shapes the evaluation process and often results in the development of novel and more targeted use of methods and employing creative practices to achieve a specific evaluation purpose, like increasing engagement in the evaluation process. Creative thinking can involve practices like redefining a problem or a goal in different ways or finding new types of solutions and applying them across various domains of knowledge, and in doing so, harmonising multiple perspectives in the problem-solving process. In addition, creative thinking can bring fresh and inventive techniques to analyse, visualise and present data, inviting different perspectives and interpretations (Patton, 2018).

This also highlights evaluation as a transdiscipline, poaching and synthesising elements from different disciplines and flexibly applying an eclectic variety of methods and theories suitable to address complex situations. Research supports what one would intuit; that creative thinking enhances the ability to develop schemas and to form mental frameworks (Andersson & Gipe, 1983). Creative thinking requires being open to new ideas and possibilities, and it is often applied when seeking to produce original and innovative outcomes. This approach is used in Creative Evaluation. Described as an unformed constellation of evaluation approaches based on varied understandings of creativity, Creative Evaluation has been used to describe ways to think creatively about evaluation, like conducting creative data collection, and ways to apply evaluation into creative settings like art, education and teaching (Christou et al., 2021).

Creative thinking also includes inferential thinking; however, the inference which emerges from creative thinking is distinct from logical inference. Rather than the careful, meticulous plodding from one valid premise to the next, creative inference can involve intuitive leaps from one premise to an entirely new position which does not necessarily require each preceding step to be defensible. Doing so enables us to develop well-formed projections of future states or consider alternative but viable realities. Inferential thinking harnesses abductive reasoning to enable vision and innovation, that is, our ability to project what is not yet but could be, to conceive and use innovative ways of combining methodology, methods and tools into an approach uniquely suitable to the evaluation of a specific program. It is that which enables us to imagine a different future and a different reality in our work as evaluators. For example, ‘Rigorous impact evaluation requires an estimate of the counterfactual: what would have occurred in the absence of the policy or program?’ (Productivity Commission, 2020, p. 9).

In addition, it includes examining the extent to which the evidence supports the conclusions reached while acknowledging the negative results and seeking out the unanticipated or unintended consequences. For that reason, Patton observes that inferential thinking ‘…can be deductive, inductive, or abductive and often draws on and creatively integrates all three analytical processes’ (Patton, 2018, p. 23, p. 23).

Reflective practice

Wilson and Jan (1999) define reflective thinkers as those who can relate their ideas to previous, present and future experiences, ask questions, criticise, evaluate themselves and situations and think critically and creatively. In program evaluation, consciously or unconsciously, by intent or default, the evaluator guides, shapes and interprets the evidence and its analysis. They are, therefore, instrumental in constructing the meaning of an evaluation in direct relationship to the influence they had in its design, implementation and reporting. Claims to knowledge are grounded in the inquirer’s perceptual frame, comprised of their unique experiences, interests, theoretical understandings, value and beliefs (Kuhn, 1999; Rossman & Rallis, 2000).

Culture and positionality

No evaluator is immune to blind spots, biases or blunders. It is only human to share a subconscious inclination, perhaps instinct, to preserve the lives of our existing beliefs, even if it means being selective with the evidence and our interpretation of it or an inclination to belief preservation and bias that guide what we select as confirming or disconfirming evidence. (Lord et al., 1979). Our position (socio-economic status, education, age, ethnicity, gender, etc.) precedes our preconceptions, predilections and biases, and our positionality places us in context to the evaluand and its context. The evaluator is the instrument, and their purpose in participating in self-observation and self-appraisal is not only to identify and address a specific question or issue but to refine their practice in general and continuously.

This Author’s Positionality

I am a 5th generation Pākehā (a New Zealander of Anglo-European descent) who grew up in Australia and spent 12 years living in Thailand, working throughout Asia. My partner is Thai, and we have a bi-lingual/bi-cultural blended family of 8 children. My professional background was in public health before immersing myself in evaluation about 20 years ago. I have been lucky enough to have benefited from a high standard of education. Although I have studied eastern philosophies (e.g. several years of study and clinical practice in Traditional Chinese Medicine), most of my formal education has been from a western academic framework underpinned by logical and critical thinking traditions that can be traced back to Greek philosophers such as Aristotle. I acknowledge that there are many and varied other ways of knowing which are imbedded and arise from their respective cultural homes. However, I do not lay claim to expertise in them; in those ways of knowing, I am a listener not a speaker.

This paper is influenced by my positionality, and it focusses on and extends the ontologies and epistemologies in which I am immersed and to which I wish to contribute and expand.

National and international evaluation codes of ethics, standards and values for evaluation practices show that evaluators and their clients understand that an evaluator influences the evaluation, for example, affecting the definition of focus and context or its interpretations and conclusions. This influence is filtered through our position, perspectives and biases, which emerge from and are driven by our values, professional and personal. In this way, the position of the evaluator and their ontological and epistemological perspectives comprise part of the context of the evaluand. To counterbalance this influence, Buckley and colleagues (2015) advise that evaluative thinkers ‘…must use another high-level thinking skill, namely, analysis and awareness of one’s own thinking or “metacognition” to recognise and reflect on their own belief preservation tendencies’ (p. 380).

Reflective thinking includes metacognitive skills that enable knowing about one’s own and others' knowing, or metacognition. Self-reflection must also lead to self-correction for it to become a skill that produces continuous sustainable learning and perpetual incremental improvements in practice (Kuhn, 1999), so reflective thinking motivates and stimulates learning (Kurt, 2018). Hmelo and Ferari (1997, p. 197) explain that ‘Reflective thinking helps individuals develop higher-order thinking skills by prompting learners to (a) relate new knowledge to prior understanding, (b) think in both abstract and conceptual terms, (c) apply specific strategies in novel tasks and (d) understand their own thinking and learning strategies’.

Combining and applying the five elements of evaluative thinking

Firstly, this is a descriptive rather than a prescriptive model. Although the elements of evaluative thinking are presented in a list, it is not a series of steps that one must follow in sequence. Nor are they each exclusive to a specific stage in evaluative thinking or evaluative process. They overlap and come together in different combinations with one or more elements in the ascendance depending on the focus or process at hand and the skills and preferences of the evaluator. As a painter mixes their primary hues on their palette to achieve nuance and depth in multiple tones, similarly, these thinking elements provide the palette with which the evaluator works. Each application of evaluative thinking requires slightly different combinations and amplitude of the elements of evaluative thinking. While there is a dominance of one element, there is also an interdependence with the use of the other elements, for example:

• Analysis requires critical thinking

• Effective practical thinking requires contextualisation

• Synthesis requires both inferential and creative thinking processes

• The application of them all requires reflective practice and results in evaluative thinking.

The uses and benefits of evaluative thinking

An example of a familiar situation in which evaluative thinking is used could be during a program logic workshop. When developing a program logic model, one must purposefully establish and test by logical inference the sequence of cause-and-effect relationships that explain how the design and action of a program (the program theory) by which it will achieve its objectives. Each program logic component must be justifiable and valid and provide the necessary pre-conditions for achieving the next step or result. One must also identify, make explicit, test and, where possible account for the underlying assumptions upon which the program is built. This is clearly the domain of vertical thinking using logic, logical inference and critical thinking to produce reliable inferences about a projected desirable future state. But to get there, one must use lateral thinking elements – examine the context to map the boundaries of the policy problem and program solution; spontaneously generate untested ideas; make creative inferential leaps and non-judgementally synthesise different points of views, some of which may initially seem incompatible; and above all else be prepared to be intellectually playful. Both vertical and lateral thinking styles are essential in program logic mapping. De Bono observed that lateral thinking is closely related to insight, creativity and humour. These three characteristics are disruptors to linear or vertical thinking. Being intellectually playful and using paradox and humour is just as necessary for lateral thinking as being meticulous and discriminatory are for vertical thinking. De Bono (2010, p. 8) was more succinct when he stated, ‘Lateral thinking is generative. Vertical thinking is selective’.

Evaluations can be designed, implemented and reported without evaluative thinking. However, without evaluative thinking, evaluators may not make important connections, miss their biases, lose the context and meaning of their findings, or, in many other ways, inadvertently compromise the quality and utility of their evaluation. As Buckley and colleagues (2015) explain, ‘…data can be collected and analysed, but without evaluative thinking, data collected may not be useful and the individuals doing the collecting would not be well poised to use and incorporate unexpected developments or adapt and revise evaluation plans in the face of setbacks or surprises in the real world. In short, evaluative thinking is a protective factor to prevent the risk of senseless, mindless evaluation’ (p. 4).

Evaluative thinking requires we direct both our cognitive skills and our metacognition to appraise our impact on the evaluation from its inception to completion.

Conclusion

Evaluative thinking is a perpetual cycle of ‘…questioning, reflecting, learning, and modifying ... conducted all the time. It is a constant state of mind within an organisation’s culture and all its systems’ (Bennett & Jessani, 2011, p. 24). Evaluative thinking is an essential set of skills for evaluators to use for: understanding the evaluand and its context; communicating and collaborating fruitfully with key stakeholders; designing high-quality evaluations that deliver valuable insights and useful information, and for learning and evaluation capability building. Evaluative thinking provides us with a range of critical and creative, vertical and lateral thinking tools.

Policy and program designers, managers and evaluators need evaluative thinking because it takes our practice to a higher level of sophistication and utility and helps to ensure that evaluation findings and conclusions are based on evidence, not judgements that are the logical extension of ideology, stereotypes, beliefs or unchallenged assumptions.

When evaluators, program planners, researchers and educators all think evaluatively and are all engaged in the evaluation process on some level, evaluations are well planned, implementation is effective and results are used in support of program evolution. Of course, not everyone in an organisation or program team needs to be an evaluator or to do evaluation work. However, if everyone involved in planning, implementing and evaluating a program is an evaluative thinker, the program and its evaluation have the best chance for success. In an ideal world, external evaluators would not be needed because the culture of the program or organisation would demand that all personnel engage in evaluating their activities and outcomes as a matter of course.

Understanding and intentionally applying the cognitive tools necessary, that is, the five elements of evaluative thinking, is an essential step in creating such an evaluative cast of mind. Evaluative thinking should be examined and learned intentionally as a core evaluation skill, just as, for example, various methodologies or methods of analysis are taught. Building evaluation capability and capacity would be significantly enhanced and substantially more sustainable if underpinned by evaluative thinking.

The process of evaluative thinking and its synthesis of thinking styles are distinctive of the evaluation profession. While it would be scientifically foolhardy to stake the claim that evaluative thinking is unique to evaluation and evaluators, this article invites the reader to consider both that evaluative thinking is distinctly part of evaluation and that evaluators, in general, are distinguishable by their application of evaluative thinking. This paper integrates observations and lessons from the literature and practice drawing together five primary elements of evaluative thinking – critical, contextual, creative, practical thinking and reflective practice – into a practical framework for more intentional reflective and effective evaluation practice. However, it may be that a more significant and useful role for this article is to provoke further thinking and discussion among critical friends about how we, as evaluators, do what we do and how we can continue to improve it.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

Alkin

M. C.,

King

J. A.

(2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434–450. https://doi.org/10.1177/1098214017717015

Al-Salim

Darwish

A. S. K.

Farrell

(2022). Analysing data quality frameworks and evaluating the statistical output of United Nations Sustainable Development Goals reports. Renewable Energy and Environmental Sustainability, 7, 17. https://doi.org/10.1051/rees/2022003

Andersson

B. V.,

Gipe

J. P.

(1983). Creativity as a mediating variable in inferential reading comprehension. Reading Psychology, 4(3), 313–325. https://doi.org/10.1080/0270271830040314

Archibald

Sharrock

Buckley

Cook

(2016). Assumptions, conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development. Evaluation and Program Planning, 59(3), 119–127. https://doi.org/10.1016/j.evalprogplan.2016.05.015

Bamberger

M.,

Mabry

(2019). Real world evaluation: Working under budget, time, data, and political constraints. Sage publications.

Bennett

G.,

Jessani

(Eds), (2011). The knowledge translation toolkit: Bridging the know-do gap: A resource for researchers. SAGE Publications India. https://sk.sagepub.com/books/the-knowledge-translation-toolkit

Blundell

R.,

Costa Dias

(2005). Evaluation methods for non-experimental data. Fiscal Studies, 21(4), 427–468. https://doi.org/10.1111/j.1475-5890.2000.tb00031.x

Bolíbar

(2016). Macro, meso, micro: Broadening the ‘social’ of social network analysis with a mixed methods approach. Quality and Quantity, 50(5), 2217–2236. https://doi.org/10.1007/s11135-015-0259-0

Brookfield

(1987). Developing critical thinkers. San Francisco, CA: Jossey-Bass.

10.

Brookfield

(2012). Teaching for critical thinking: tools and techniques to help students question their assumptions. San Francisco, CA: Jossey-Bass.

11.

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388. https://doi.org/10.1177/1098214015581706

12.

Chelimsky

(1987). What have we learned about the politics of program evaluation? Educational Evaluation and Policy Analysis, 9(3), 199–213. https://doi.org/10.3102/01623737009003199

13.

Chen

H. T.,

Rossi

P. H.

(1987). The theory-driven approach to validity. Evaluation and Program Planning, 10(1), 95–103. https://doi.org/10.1016/0149-7189(87)90025-5

14.

Christou

Owen

Ceyhan

(2021). Concepts and contexts of creative evaluation approaches. Journal of MultiDisciplinary Evaluation, 17(40), 1–15. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/675/579

15.

Clarke

A. E.

(2007). Situational analysis. The Blackwell Encyclopedia of Sociology (pp. 1–2). https://doi.org/10.1002/9781405165518.wbeos0777

16.

Clarke

A. E.

Friese

Washburn

(2016). Situational analysis in practice: Mapping research with grounded theory. Routledge. https://doi.org/10.4324/9781315420134

17.

Cole

M. J.

(2022). Capacity-building in community-based drug treatment services health and human rights Journal, 24(1), 2022, 189–202. https://www.hsph.harvard.edu/wp-content/uploads/sites/2469/2022/06/cole.pdf

18.

Conner

R. F.

Fitzpatrick

J. L.

Rog

D. J.

(2012). A first step forward: Context assessment. New Directions for Evaluation, 2012(135), 89–105. https://doi.org/10.1002/ev.20029

19.

De Bono

(2010). Lateral thinking: A textbook of creativity.

20.

De Bono

(1969). Information processing and new ideas—Lateral and vertical thinking. The Journal of Creative Behavior, 3(3), 159–171. https://doi.org/10.1002/j.2162-6057.1969.tb00124.x

21.

Dewey

(1910). How we think. Boston, MA: D.C. Heath.

22.

Ellamil

Dobson

Beeman

Christoff

(2012). Evaluative and generative modes of thought during the creative process. Neuroimage, 59(2), 1783–1794. https://doi.org/10.1016/j.neuroimage.2011.08.008

23.

Ennis

R. H.

Millman

Tomko

T. N.

(1985). Cornell critical thinking tests level X and level Z: Manual. Midwest Publications.

24.

Facione

(1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. American Philosophical Association.

25.

Fritz

Kaiser

Levy

(2009). Problem-driven governance and political economy analysis: Good practice framework (English). World Bank Group. http://documents.worldbank.org/curated/en/469651468150576876/Problem-driven-governance-and-political-economy-analysis-good-practice-framework

26.

Funnell

S. C.,

Rogers

P. J.

(2011). Purposeful program theory: Effective use of theories of change and logic models. John Wiley and Sons.

27.

Gollan

S.,

Stacey

(2021). Australian evaluation society first nations cultural safety framework. Australian Evaluation Society. https://www.aes.asn.au/images/AES_FirstNations_Cultural_Framework_finalWEB_final.pdf

28.

Graetz

(2002). Strategic thinking versus strategic planning: Towards understanding the complementarities. Management Decision, 40(5), 456–462. https://doi.org/10.1108/00251740210430434

29.

Guenther

Williams

Arnott

(2010). The politics of evaluation: Evidence-based policy or policy-based evidence In: NARU public seminar series (Vol. 20). Darwin. https://doi.org/10.13140/2.1.2022.6564

30.

Haider

H.,

Rao

(2010). Political and social analysis for development policy and practice: An overview of five approaches. http://www.gsdrc.org/docs/open/EIRS10.pdf

31.

Hmelo

C. E.,

Ferrari

(1997). The problem-based learning tutorial: Cultivating higher order thinking skills. Journal for the Education of the Gifted, 20(4), 401–422. https://doi.org/10.1177/016235329702000405

32.

Juma

T. O.,

Onkware

(2015). The challenges of public policy formulation and evaluation through the questions what, who, how, and when. International Journal of Economics, Commerce and Management, 3(11), 830–841. https://ssrn.com/abstract=2691519

33.

Klayman

(1995). Varieties of confirmation bias. Psychology of Learning and Motivation, 32, 385–418. https://doi.org/10.1016/S0079-7421(08)60315-1

34.

Kolios

A.,

Read

(2013). A political, economic, social, technology, legal and environmental (PESTLE) approach for risk identification of the tidal industry in the United Kingdom. Energies, 6(10), 5023–5045. https://doi.org/10.3390/en6105023

35.

Kuhn

(1999). A developmental model of critical thinking. Educational Researcher, 28(2), 16–46. https://doi.org/10.3102/0013189X028002016

36.

Kurt

(2018). Quality in reflective thinking elicitation and classification of reflective acts. Quality and Quantity, 52(S1), 247–259. https://doi.org/10.1007/s11135-017-0609-1

37.

Lemire

Peck

L. R.

Porowski

(2020). The growth of the evaluation tree in the policy analysis forest: Recent developments in evaluation. Policy Studies Journal, 48(S1), S47–S70. https://doi.org/10.1111/psj.12387

38.

Lord

C. G.

Ross

Lepper

M. R.

(1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109. https://doi.org/10.1037/0022-3514.37.11.2098

39.

S. K.

Elliot

S. J.

Perlman

C. M.

(2019). Perceived facilitators and barriers to evaluative thinking in a small, development NGO. Canadian Journal of Program Evaluation, 34(1), 68–83. https://doi.org/10.3138/cjpe.43118

40.

Markiewicz

(2008). The political context of evaluation: What does this mean for independence and objectivity? Evaluation Journal of Australasia, 8(2), 35–41. https://doi.org/10.1177/1035719X0800800205

41.

McLaughlin

J. A.,

Jordan

G. B.

(1999). Logic models: A tool for telling your program’s performance story. Evaluation and Program Planning, 22(1), 65–72. https://doi.org/10.1016/S0149-7189(98)00042-1

42.

Mintzberg

(1994, January-February). The rise and fall of strategic planning. Harvard Business Review. https://hbr.org/1994/01/the-fall-and-rise-of-strategic-planning

43.

Mintzberg

H.,

Waters

J. A.

(1985). Of strategies, deliberate and emergent. Strategic Management Journal, 6(3), 257–272. https://doi.org/10.1002/smj.4250060306

44.

Owen

J. M.

(2020). Program evaluation: Forms and approaches. Routledge. https://doi.org/10.4324/9781003116875

45.

Patton

M. Q.

(2001). Evaluation, knowledge management, best practices, and high quality lessons learned. American Journal of Evaluation 22(3), 329–336. https://doi.org/10.1177/109821400102200307

46.

Patton

M. Q.

(2008). Utilisation-focused evaluation. Sage.

47.

Patton

M. Q.

(2018). A historical perspective on the evolution of evaluative thinking. In Vo

A. T.

Archibald

(eds.), Evaluative thinking. New Directions for Evaluation, 2018(158), p. 11–28. https://doi.org/10.1002/ev.20325

48.

Preskill

H.,

Torres

R. T.

(2000). The learning dimension of evaluation use. New Directions for Evaluation, 2000(88), 25–37. https://doi.org/10.1002/ev.1189

49.

Productivity Commission. (2020). A guide to evaluation under the indigenous evaluation strategy. Commonwealth of Australia.

50.

Puri

Aladysheva

Iversen

Ghorpade

Brück

(2015). What methods may be used in impact evaluations of humanitarian assistance? https://doi.org/10.23846/WP0022

51.

Rallis

S. F.,

Rossman

G. B.

(2000). Dialogue for learning: Evaluator as critical friend. New Directions for Evaluation, 2000(86), 81–92. https://doi.org/10.1002/ev.1174

52.

Rastogi

N.,

Trivedi

M. K.

(2016). PESTLE technique – a tool to identify external risks in construction projects. International Research Journal of Engineering and Technology, 3(1), 384–388. https://www.irjet.net/archives/V3/i1/IRJET-V3I165.pdf

53.

Ratan

S. K.

Anand

Ratan

(2019). Formulation of research question–Stepwise approach. Journal of Indian Association of Pediatric Surgeons, 24(1), 15–20. https://doi.org/10.4103/jiaps.JIAPS_76_18

54.

Rog

D. J.

(2012). When background becomes foreground: Toward context-sensitive evaluation practice. New Directions for Evaluation, 2012(135), 25–40. https://doi.org/10.1002/ev.20025

55.

Rog

D. J.

Fitzpatrick

J. L.

Conner

R. F.

(Eds), (2012). Context: A framework for its influence on evaluation practice: New directions for evaluation, number 135. John Wiley and Sons.

56.

Rogers

P. J.

(2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), 29–48. https://doi.org/10.1177/1356389007084674

57.

Rossman

G.,

Rallis

S. F.

(2000). Critical inquiry and use as action. The expanding scope of evaluation use. New Direction in Program Evaluation, 88, 55–69. https://doi.org/10.1002/ev.1191

58.

Schwandt

T. A.

(2018). Evaluative thinking as a collaborative social practice: The case of boundary judgment making. In Vo

A. T.

Archibald

(eds), Evaluative thinking. New Directions for Evaluation, 2018(158), 125–137. https://doi.org/10.1002/ev.20318

59.

Scriven

(2007). The logic of evaluation, Dissensus and the search for common ground. Ontario Society for the Study of Argumentation (pp. 1–16). https://scholar.uwindsor.ca/cgi/viewcontent.cgi?article=1390&context=ossaarchive

60.

Scriven

M.,

Paul

(1987). Defining critical thinking. 8th annual international conference on critical thinking and education reform. https://www.quia.com/files/quia/users/medicinehawk/1607-Thinking/defining.pdf

61.

Senge

P. M.

(1990). The art and practice of the learning organization. In Rossman

Rallis

S. F.

. Critical inquiry and use as action. The expanding scope of evaluation use. New Direction in Program Evaluation, 88, 55–69. https://doi.org/10.1002/ev.1191

62.

Senge

(1990). The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Doubleday.

63.

Serpa

S.,

Ferreira

C. M.

(2019). Micro, meso and macro levels of social analysis. International Journal of Social Science Studies, 7(3), 120–124. https://doi.org/10.11114/ijsss.v7i3.4223

64.

Shehadeh

(2020). Contextualising your research project. In Coombe

Anderson

N. J.

Stephenson

(eds). (2020), Professionalising your English language teaching (pp. 327–335). Springer.

65.

Unsworth

(2009). What's politics got to do with it?: Why donors find it so hard to come to terms with politics, and why this matters. Journal of International Development, 21(6), 883–894. https://doi.org/10.1002/jid.1625

66.

Van Oers

(1998). From context to contextualizing. Learning and Instruction, 8(6), 473–488. https://doi.org/10.1016/S0959-4752(98)00031-0

67.

Vindrola-Padros

Brage

Johnson

G. A.

(2021). Rapid, responsive, and relevant?: A systematic review of rapid evaluations in health care. American Journal of Evaluation, 42(1), 13–27. https://doi.org/10.1177/1098214019886914

68.

A. T.,

Archibald

(2018). New directions for evaluative thinking. New Directions for Evaluation, 2018(158), 139–147. https://doi.org/10.1002/ev.20317

69.

A. T.

Schreiber

J. S.

Martin

(2018). Toward a conceptual understanding of evaluative thinking. In Vo

A. T.

Archibald

(Eds.), Evaluative thinking. New directions for evaluation (Vol. 158, p. 29–47). https://doi.org/10.1002/ev.20324

70.

Voros

(2001). A primer on futures studies, foresight and the use of scenarios. Prospect: The Foresight Bulletin, 6(1).

71.

Waltz

K. N.

(1997). Evaluating theories. American Political Science Review, 91(4), 913–917. https://doi.org/10.2307/2952173

72.

Weiss

C. H.

(1988). If program decisions hinged only on information: A response to Patton. Evaluation Practice, 9(3), 15–28. https://doi.org/10.1177/109821408800900302

73.

Weiss

C. H.

(1993). Where politics and evaluation research meet. Evaluation Practice, 14(1), 93–106. https://doi.org/10.1016/0886-1633(93)90046-R

74.

Weiss

C. H.

(1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21–33. https://doi.org/10.1177/109821409801900103

75.

Wilson

J.,

Jan

L. W.

(1999). Thinking for themselves: Developing strategies for learning. Eleanor Curtain.