Where Impact Measurement Meets Evaluation

Where Evaluation and Impact Measurement Meet

Evaluation is an old practice. Some date it back to the Chinese civil service exams—a form of personnel evaluation (Fitzpatrick, Sanders, & Worthen, 2011). Evaluation as it exists today, at least in the West, can be traced to actions taken in response to demands for improvement in social conditions across diverse sectors. Take, for example, Britain’s need to answer calls for educational reform in the 1800s, the U.S.'s efforts to stimulate social and economic recovery after the Great Depression in the 1940s (i.e., the New Deal legislation as proposed during Roosevelt’s presidency), and the U.S.'s response to address racial injustice and civil unrest through Johnson’s Great Society programs in the 1960s (Hogan, 2007). To be certain, countless examples of attempts to support social welfare and development exist around the globe. The diversity of values and value systems specifically in the West, including conceptions of democracy and the infrastructure that upholds it (e.g., policies, institutions), have greatly influenced the manner in which governments and its citizens choose to take on these challenges. Power structures and the manner in which power is distributed, on the other hand, tend to be more homogenous. The potential for misalignment between power and values creates space for and underscores the significance of transparency and accountability—social drivers that underlie evaluation as we know it.

Impact investing (and social investment)—a practice that can be traced to centuries-old Jewish and Islamic laws that guide ethical financial decision-making—gave way to the notion that strategic resource allocation can catalyze changes in one’s environment (SRI Conference & Community, 2018; Trelstad, 2016). Contemporary literature attributes the desire to know about the nature of those changes, and the genesis of impact measurement as a legitimate way of knowing, to the enactment of the National Environmental Policy Act (NEPA) in 1970. NEPA was designed to ensure careful a priori consideration of intended and unintended effects of country-building initiatives; thus, early efforts to understand impact placed great emphasis on changes to biological and physical shifts in the environment (Burdge, 2003; Dendenda & Corsi, 2015; Dufour, 2015). Interest in social and economic impact subsequently gained traction and evolved alongside such phenomena as international development, development evaluation, globalization, and corporate social responsibility (International Association for Impact Assessment, 2018; Picciotto, 2015a). While evaluation and impact measurement meet at the intersection of learning and decision-making, they diverge as private entities take their places alongside governments in the effort to drive social change. Achievement of social and environmental impact are secondary aims while financial return is prioritized for many independently run enterprises that commission or conduct impact measurement studies.

Evaluation and impact measurement share common roots; however, there is also variation on a number of other dimensions. Table 1, which was developed based on a cursory review of recent evaluation and impact measurement literature, offers a summary of these similarities and differences (Picciotto, 2015a, 2015b; Reisman & Olazabal, 2016; Szijarto et al., 2018; Vo et al., 2016). Because the overarching purposes that drive each enterprise are different, chief funding sources, primary end users of study results, evaluative approaches, analytic units and methods, and the nature of findings also differ to varying degrees. Human drivers and end users of evaluation, for example, have tended to include public- and private-sector funders, program staff, and tax-paying program beneficiaries, whereas private-sector funders and social entrepreneurs have greatly influenced the impact investment and, subsequently, impact measurement spaces. Likewise, evaluation and impact measurement have each developed in response to different stakeholders and societal demands. As such, the number of evaluative approaches, units of analyses, analytic methods, and study designs that inform the practice of each vary in turn. Specifically, the literature suggests that there seems to be greater emphasis on the social and political aspects of how impact is determined when conducting evaluation compared to impact measurement. Further, because both evaluation and impact measurement are mechanisms that inform decision-making, ongoing and end-of-cycle reporting are common practices.

OPEN IN VIEWER

Table 1.

Areas of Convergence and Divergence Between Evaluation and Impact Measurement.

Dimension	Evaluation	Impact Measurement
Social driver(s)	Accountability Decision making: • Program improvement • End of cycle judgement	Decision making: • Financial development • Sustainability • Strategic planning
Human driver(s)	Public sector funders (e.g., government) Private sector funders (e.g., foundations, corporations) Program staff Program beneficiaries	Private sector funders (e.g., foundations, corporations, angel/impact investors) Social entrepreneurs Public sector funders (e.g., government)
End user(s)	Public sector funders (e.g., government) Private sector funders (e.g., foundations, corporations) Program staff Program beneficiaries	Private sector funders (e.g., foundations, corporations, angel/impact investors) Social entrepreneurs Public sector funders (e.g., government)
Evaluative approaches	Context, Input, Process, Product (CIPP) model Complexity theory Critical systems heuristics Developmental evaluation Empowerment evaluation Evaluability assessment Participatory evaluation Theory-driven evaluation	Social accounting Valuation
Analytic unit(s) of interest	Inputs Activities Outputs Outcomes	Activities Outputs
Analytic method(s)/ Design(s)	Case study/Descriptive designs Contribution analysis Experimental design Mixed methods Most significant change Outcome harvesting Pre-post outcomes measurement Quasi-experimental design	Balanced Scorecard Case comparisons Logic model analysis Monitoring Performance measurement Pre-post outcomes measurement Social Return on Investment
Nature of findings	System specific Organization specific Program specific Population specific Site specific	Program specific Population specific Site specific
Dissemination practice(s)	Ongoing End of cycle	Ongoing End of cycle

This Forum

This forum—Where Impact Measurement Meets Evaluation—was organized in the spirit of continuing the conversation about potential intersections between two fields that have, up until recently, existed largely independently of one another. The issues and topics addressed in this forum were inspired by prominent themes that permeated the 2016 American Evaluation Association Annual Conference. Held in Atlanta, Georgia, on the heels of the Social Value International Meeting, attendees at these meetings encountered, exchanged, and wrestled with ideas and ways of solving evaluative problems, both old and new. Common postsession questions included “What is meant by ‘impact’ (in the evaluation and impact measurement communities and vice versa)?” “How is impact measurement different from evaluation?” and “If bridges between evaluation and impact measurement are needed, then how are they best built?”

Our hope is that a forum such as this will foster further thinking about the unique and distinct challenges as well as successes that evaluators and impact analysts encounter in practice. We anticipate that this continued dialogue will dovetail into ongoing evaluation field-building efforts, whose work thus far has heavily emphasized examinations of what evaluation is, who evaluators are, and what evaluators do. Equally important in this mission, however, is knowing what evaluation is not, what evaluators do not do, where evaluative work intersects with related fields, and how to grow from these reflections.

Papers in This Forum

Papers in this forum address three issues that strike at the heart of evaluation and impact measurement: demand for meaningful data and evidence, the need for analytic capacity, and measurement challenges. Each essay offers an introduction to the issue of interest, grounds the topic in evaluation and impact measurement work, and proposes solutions that aim to move our thinking forward.

Jane Reisman, Veronica Olazabal, and Shawna Hoffman coauthored the first paper in which they describe the landscape of impact investing, ways in which social investors have previously sought information about return on investment, challenges encountered, gaps in approaches, and ways in which the evaluation field can help address those gaps. Among solutions that Reisman, Olazabal, and Hoffman propose are setting out to understand needs and expectations in both communities, as challenges encountered thus far are likely rooted in miscommunication rather than “mismatch of interests and values.” Reisman is the founder and former president of ORS Impact, an independent evaluation firm. Olazabal and Hoffman conduct social impact measurement work at the Rockefeller Foundation.

The second paper is coauthored by Karim Harji and Edward Jackson who address the challenge of building capacity within the field to respond to demands for better information about social impact. They argue that unclear financial responsibility for evaluation and resistance to discuss venture failure are the roots of this challenge. Potential solutions, however, could include streamlining evaluation tools and approaches, ensuring diverse stakeholder perspectives are represented throughout the process, understanding private-sector nomenclature, targeted professional development, and growing and participating in communities of practice. Harji and Jackson are principals at Purpose Capital (an impact investment consulting firm) and ET Jackson and Associates (a management consulting firm), respectively.

Kate Ruff (faculty at the Sprott School of Business) and Sara Olsen (founder of the Social Venture Technologies Group) coauthored the third essay. They addressed challenges surrounding efficient and meaningful measurement of impact across organizations, across contexts, and over time. Ruff and Olsen suggest that leveraging operations data (or, administrative data); using standards that are “bounded,” but also “flexible”; and building a pipeline of diverse impact analysts to facilitate this work as potential solutions.

Finally, Anna-Marie Harling (Union Bank of Switzerland) and Lisa Hehenberger (faculty at ESADE Ramon Llull University) offer closing comments and reflections on the issues that contributing authors raised. They highlight prominent cross-cutting themes, gaps not addressed, and offer thoughts on new directions in which both the evaluation and impact measurement communities can embark to build these respective fields.