Understanding evaluation practice within the context of social investment

Abstract

Social impact investing is the use of private investment to fund social programs in various public welfare sectors. It is currently unclear which evaluative practices are used to determine the impact of social investment. This study describes how impact investments are evaluated and the factors that help explain variations in practice through survey (N = 161) and interview (N = 13) data from investors, intermediaries, entrepreneurs, and analysts. Study findings indicate that analysts typically rely on descriptive quantitative and qualitative data and analyses. Analysts’ background, training, and role; perceptions of the goals of the work; and the complexity of a program’s theory influence their evaluation approach. More nuanced or indirect impacts of investments may not be captured because evaluation activities tend to be limited in scope and scale. Future research should explore the work of those directly responsible for impact activities to better understand the conditions under which more sophisticated evaluation approaches might be employed.

Keywords

evaluation research on evaluation social impact investing social impact measurement

The past several decades have witnessed a rise in social betterment activities aimed at achieving a blend of social, environmental, cultural, and financial outcomes for communities and beneficiaries. Social investment, strategic philanthropy, international development, and social entrepreneurism represent just a few of the approaches that fall within this sector (Bannick et al, 2014; World Bank, 2016). The use of private capital to effect positive social change – what we will refer to as social investment – has grown in popularity for a variety of reasons. Education, housing, labor, and other public welfare initiatives, traditionally funded by the state in countries such as Germany and the United Kingdom, have experienced reductions in resources since at least 2008, at the advent of the global economic crisis (Benford et al., 2014; Deloitte LLP, 2014). The decrease in breadth and scope of public services, as well as increased barriers to accessing services, have created a space for social entrepreneurs to fill important voids as service providers in this market (Sharir and Lerner, 2006; Weisbrod, 2004). These conditions have also given way to demands for innovations and services that tend to be privately held and privately funded. Therefore, as public sector funds continue to diminish, the need for and prominence of social entrepreneurship will certainly continue to grow.

The success of a social enterprise hinges on a number of related factors including the entrepreneur’s ability to secure investment, the viability of the business plan, and the enterprise’s ability to use impact data to inform decision making and to adapt and respond to fluctuating market conditions (Initiative for Global Development, 2013). Further, because a successful enterprise must, by definition, be income-generating, gauging the extent to which the organization is able to realize its social goals as a result of having sought out and received private investments has become an integral aspect of this work. Not only does ‘success’ mean many things within the social investment space, but demonstrating this success requires attention to many moving parts in order to gauge its impact.

There has been increased focus on how to best evaluate whether social investments have achieved an impact financially, in terms of return on investment, and socially, in terms of promoting social change (Brest and Born, 2013; Initiative for Global Development, 2013). Understanding the extent to which a social enterprise is successful – that is, impactful – can be accomplished through various means. If interested primarily in an enterprise’s financial health and status, one may elect to focus on ‘bottom line’ reports. If organizational processes and procedures are of importance, a whole evaluation system involving the consideration of the relationship among inputs, activities, outputs, and outcomes is likely required. On the other hand, if social impact is of interest, then a more summative approach requiring the analysis of well-defined and far-reaching outcomes is necessary. In general, evaluating such activities requires knowledge of approaches to measuring both financial and social outcomes and is typically referred to as impact measurement (or impact assessment, per Reeder and Colantonio, 2013). The practitioners who conduct these studies are often known as impact analysts (Seymour et al., 2013). In this article, we use this nomenclature to describe the evaluation activities and the practitioners who conduct this work in social investment environments.

Initially, impact measurement of a social enterprise appears to resemble other evaluative processes. More careful consideration suggests, however, that the process can be understood as one important aspect of evaluation practice broadly conceived. The evaluation and impact measurement literatures reflect quite a diverse set of overarching approaches and technical procedures that can be used to determine degree of goal attainment (Olsen and Galimidi, 2008; Rossi and Freeman, 1993; Seymour et al., 2013; Shadish et al., 1991). Not well understood, though, is how impact analysts – evaluators who specialize in measuring social impact – go about their practice. Little is also known about the factors that contribute to variation in the process of determining social impact. A clearer understanding of the impact measurement landscape, established through systematic inquiry, will become increasingly important as social, political, and economic conditions continue to change in response to market forces. In an effort to respond to this need for empirical knowledge, we set out to address the following questions:

What are impact analysts doing to help organizations determine the extent to which social impact has been achieved?

What factors influence and inform how impact analysts evaluate social impact?

Our hope is that the insights gained from this investigation will contribute to an improved understanding of evaluation practice in the social investment context so that current practice can be informed and further improved. Study results will also be useful in developing research agendas that have the potential to subsequently shape the methods, approaches, and techniques used to study social measurement across a multitude of sectors.

Literature review

Definitions of impact measurement

Contemporary social impact measurement grew out of the early recognition of the limitations of financial accounting. Known as social accounting (or social and environmental accounting) in the 1970s, it includes items that do not have an established dollar value and maintains a focus on stakeholders other than investors (e.g. beneficiaries, volunteers, society, government, etc.; Bebbington et al., 1999; Richmond et al., 2003). Understood in this fashion, Quarter et al. (2003) defined social accounting as ‘a systematic analysis of the effects of an organization on its communities of interest or stakeholders, with stakeholder input as part of the data that are analyzed for the accounting statement’ (p. 3). Vanclay (2003), in tandem, suggested that social impact measurement is the activity of analysing, monitoring, and managing the social consequences of development. These effects can be detected in changes to people’s ways of life, their cultures, their communities, their political systems, their environments, their health and wellbeing, their personal and property rights, and/or their fears and aspirations. He argued that the assessment of impact is more than just a technique; it is also a philosophy about development and democracy. Consideration of development goals and processes, who is in the position to set goals and determine courses of action, and the manner in which they are modified is at the heart of Vanclay’s (2003) position.

Expanding upon this notion, the World Bank (2003) described social assessment as a process through which its borrowers better understand how the social-cultural, institutional, historical, and political contexts influence the social development outcomes of specific investment projects and sector policies. It is a means to enhance equity and social cohesion, a mechanism to identify opportunities and constraints, a framework for dialogue on development priorities, and an approach to identify and mitigate social risks. In whole, social impact measurement, as it has existed to date, reflects converging efforts to emphasize non-monetized outcomes and represent broader stakeholder perspectives.

Despite the manner in which the definition of impact measurement has evolved, an emphasis on accountability continues to drive efforts to improve social and environmental outcomes. Funders, taxpayers, clients, and concerned citizens have increasingly demanded greater transparency with respect to fundraising and spending along with the ways in which programs are managed. In the same vein, impact measurement exists to shed light on the extent to which investments have contributed to actual social or environmental impact and to drive decision making by various stakeholder groups (Brest and Born, 2013).

Relatedly, a more prevalent focus has grown around the determination of impact, or the demonstration of results in addressing complex social problems such as poverty and inequality (Ebrahim and Rangan, 2010). This orientation is driven both by funders interested in discerning the beneficial effects of their investments and by non-profit leaders and social entrepreneurs interested in resolving social problems. Both face the challenge of determining the legitimate allocation of vital resources, where the assessment and evaluation of impact becomes instrumental in programmatic decision making (House and Howe, 2000). Impact is also seen as driven by auditors and evaluators, who represent a professionalization of the field and the institutionalization of common norms of practice (Hwang and Powell, 2009; Power, 1999).

Existing impact measurement frameworks

The literature suggests neither one best method for determining social impact nor a singular problem that impact measurement is expected to solve. The international development field, for example, views social impact as ‘significant or lasting changes in people’s lives, brought about by a given action or series of actions’ (Roche, 1999: 21). Thus, impact measurement could be approached using descriptive, qualitative research methods to understand the extent to which the ‘root cause’ of a social problem has been effectively targeted and the desired outcomes have been achieved (Crutchfield and Grant, 2012).

Given the array of approaches, it comes as no surprise that several frameworks have been created to summarize and describe impact measurement practices, including activities, tools, and schemas for thinking about impact; methods for data gathering and analysis; artifacts such as reports, data sets, models and forms of evidence; and standards that aim to deliver common approaches to impact measurement across similar sectors (Seymour et al., 2013). Ebrahim and Rangan (2010), for example, suggested that what is measured (i.e. inputs, activities, outputs, and/or outcomes) is primarily contingent upon two things: the complexity of the cause and effect relationship underlying the intervention and the complexity and scope of the operational strategy the organization employs to implement its mission. Thus, impact analysts are advised to tailor their measurement and data collection inquiries to the various stages of investment (e.g. making investment decisions; identifying and mitigating risk; capturing long term value; tracking progress; and being accountable to stakeholders, etc.) and to select tools that best answers the impact questions of interest (Best and Harji, 2012; Reeder and Colantonio, 2013).

Emerson (2003) introduced the idea of blended value accounting – an evaluative approach applied to non-profit organizations and social enterprises based on their ability to produce financial, social, and environmental outcomes – and Nicholls (2009) considered the manner in which it can be presented as a spectrum of results reporting logics to decision makers. These span from positivist-driven assessments – focusing on financial value and quantitative data – to interpretive-driven assessments, focusing on social value and qualitative data. Nicholls argued that rather than subscribing to one static schema, this presents a range of social impact reporting options that can be combined to capture the holistic complexity of organizational outputs and impacts.

Situating social impact measurement in frameworks such as those described above has become an increasingly challenging obstacle to overcome, as it continues to grow and adapt to innovative strategies of development. Questions persist about what is to be measured, how it should be measured, and how to conduct measurements that are both meaningful and reliable. These questions sit alongside concerns about aligning measurement and reporting with mission objectives, setting parameters around results aggregation and benchmarking, determining what resources should be dedicated to evaluation, fostering use, and considering the influences of those in power.

A conceptual framework on impact measurement for this investigation

This study sought to expand the present body of knowledge surrounding impact measurement in the context of social investment by examining current practices in the field. To do so, we started with a review of the literature and have extracted and organized themes into a conceptual framework that guided the present investigation (see Figure 1). Specifically, we nested the concepts in the social impact literature within those found in the evaluation literature.

Figure 1.

Conceptual framework on impact measurement.

Through this framework, we suggest that the conduct of impact measurement can be characterized by several dimensions commonly found in the evaluation literature: the problem context, audience/users of study findings, study purpose, methodology, perceived credibility of the evidence used, and ways in which use of results can be fostered. This schema mirrors and is complementary to the framework for program evaluation that the Centers for Disease Control and Prevention (CDC) developed in the United States (Milstein and Wetterhall, 1999). The CDC model outlines six necessary steps in evaluation: engage stakeholders, describe the program, focus the evaluation design, gather credible evidence, justify conclusions, and ensure use and share lessons learned.

Impact measurement and evaluation are both cognate practices. As such, there are many parallel concepts between the framework that we developed and the CDC model. Consider, for example, the Problem Context, Primary Users, and Purposes dimensions in Figure 1. Problem Context aims to capture variation in the nature and scope of projects where an impact measurement takes place, while Primary Users refers to stakeholder groups typically involved in the impact measurement process. The Purposes dimension reflects motivations that would lead primary users to commission an impact study. We view these ideas as analogous to understanding who the primary stakeholders of an evaluation are; what social, organizational, and political contexts they and their programs operate within; what motivates them to engage in evaluation; and how evaluation addresses their information needs – ideas that are explicitly emphasized in the CDC model.

The Methodology, Credibility of Evidence, and Fostering Use dimensions in Figure 1 are more closely aligned with the final steps in the CDC model, each dealing with the strategies, tools, and approaches used to examine data and guide subsequent decision making. Impact measurement is one important kind of evaluation activity. Engaging in evaluation and measuring social impact require the evaluator and the impact analyst, respectively, to adapt their practices according to the constraints and affordances of each unique situation. Below we describe what we have learned about impact measurement practices from impact analysts. These findings deepen not only our understanding of the impact measurement field but also the evaluation field. First, however, we provide an overview of our research methods.

Methods

Study participants

This study involved participants who play various roles in the social investment space, such as impact investors, intermediaries, social entrepreneurs, and impact analysts. Participants completed a survey and an interview. A brief description of those who participated in this investigation and the nature of their involvement appear below. We provide a more detailed description of study participants in our results section.

Survey participants

Survey participants were identified using a hybrid purposive-snowball sampling approach. Potential respondents were sampled from the membership of the Social Impact Analysts Association (SIAA) and the Social Return on Investment (SROI) Network.¹ Members of these organizations also forwarded the survey to other professionals in the field. Stakeholders at the Bertelsmann Foundation supported recruitment of survey participants in a similar fashion.

The number of respondents varied for each survey item. A total of 160 respondents answered at least some of the survey items. In total, 75 respondents completed the entire survey. Skip patterns and nonresponse reduced the number of total respondents for some questions. As a result, the number of survey respondents who completed each question varies.

Interview participants

A hybrid purposive-snowball sampling approach was also used to identify social enterprises and affiliated stakeholder groups for this investigation. To be a part of the sampling frame, social enterprises had to meet the following inclusion criteria:

Located in Germany or the United Kingdom;

Affiliated with either the Social Impact Analysts Association or the Bertelsmann Foundation;

Have personnel who were able to communicate in English;

Have English web-based materials to facilitate potential follow-up conversations;

Have a return on investment model.

The research team identified 49 potential social enterprises to include in the study. Thirty-six of these enterprises were based in the United Kingdom, while 14 were based in Germany.

Data concerning the size of investment that these enterprises received in the last two to three fiscal years and the investment model that each was implementing were also gathered. This information was used to ensure variability and representativeness of funding levels among social enterprises that were included in the study. Thirteen enterprises in the United Kingdom were excluded because they were funded with grants or impact bonds² or because investment information was not available. Eleven enterprises in Germany were excluded for similar reasons. Of the remaining 23 UK enterprises, four had received ‘small’ investments while five and 14 had received ‘medium’ and ‘large’ investments, respectively. The three German enterprises that remained had each received a ‘small,’ ‘medium,’ and ‘large’ investment. In this study, small investments were defined as €250,000 or less; medium investments were between €250,000 and €1 million; and large investments were €1 million or more.

We recruited interview participants from within these United Kingdom and German social enterprises via email. Typically, interviews with impact investors were conducted first and then referrals to other stakeholder groups were requested. As summarized in Table SD1, in total, six social enterprises were included in the study (three in the United Kingdom and three in Germany). We conducted 13 interviews with key informants: five impact investors representing six enterprises; one intermediary; five social entrepreneurs; and two impact analysts representing three enterprises.

Tools and procedures

We used descriptive methods in this investigation. In particular, we conducted approximately 13 key informant interviews/case studies to understand how impact analysts, investors, and investees approached impact measurement and impact investment. Interviews/cases also served as opportunities for us to pilot the survey with SIAA members. These tools and protocols are described below.

Social impact analysis survey

We administered the survey electronically in English in September 2014 using SurveyMonkey.com’s online interface. Members of SIAA and the SROI Network were emailed a link to the survey and were offered an incentive to complete it. The survey contained closed- and open-ended questions about social impact analysts’ professional backgrounds and experiences, their recent experiences with impact measurement projects, and the ways in which they had conducted impact measurement.

Key informant interviews

Key informant interviews were conducted between January 2014 and December 2014. Each conversation lasted approximately 45–60 minutes. When necessary, interviews were conducted in multiple shorter intervals until all necessary information was successfully collected. Interviews were recorded and transcribed, and summaries of the major themes that arose during the interviews are presented below.

Study limitations

We sought to bolster the rigor and quality of this investigation by using a mixed method design, which aims to leverage the complementary strengths and minimize the non-overlapping weaknesses of various data collection tools (Brewer and Hunter, 1989; Tashakkori and Teddlie, 1998). Still, there are several limitations regarding the approach taken that the reader should keep in mind when interpreting study findings.

Limitations with survey data

Use of a hybrid sampling approach (i.e. purposeful and snowball) allowed us to reach the target audience when administering the online survey. While the overall response rate could be considered adequate, the relatively low number of impact analysts who completed the entire survey poses a number of limitations. For example, readers should be conservative in attempts to generalize survey findings beyond the scope of the study. Analytically, this also means that using more sophisticated statistical techniques to examine the data was not feasible. Instead, we relied upon descriptive analyses.

Limitations with interview data

With respect to data collected via interviews, the sample is not representative of all enterprises or stakeholders in the social investment and impact measurement sectors. There were more investors/intermediaries represented than any other group, and the findings may reflect their perspectives over other groups. Impact analysts were the least represented group and conclusions about their perspectives had to be drawn from a small number of interviews. While a considerable amount of effort was dedicated to ensuring that stakeholders’ views were represented fairly, the nature of the interviews as qualitative data and limiting data collection to English-speaking participants prevent us from making general statements about all investors or impact analysts, or about the field as a whole.

In spite of these limitations, information captured via the surveys and interviews collectively allows us to identify themes that may justify further research or related projects in the impact measurement and social investment areas.

Results

We proposed two central research questions in order to better understand the nature of impact measurement in the context of social investment. The first line of inquiry investigated impact analysts’ measurement practices when they are tasked with determining social impact. The second question explored the factors that influence and inform how impact is measured in the context of social investment.

What are impact analysts doing to help organizations determine the extent to which social impact has been achieved?

The first research question of this study investigated the steps or activities impact analysts engaged in to measure an organization’s impact in the context of social investment. The survey and interviews both confirmed that respondents relied on a mixture of quantitative and qualitative approaches when assessing the impact of a social investment. These approaches permeated various aspects of the study, including methods for data collection, kinds of data analysis strategies, and approaches to reporting on impact findings. Surveys were the most common data collection technique (64% said they used them for many or all outcomes) and consisted of pre–post questionnaires or post-only, while interviews were the second most common activity, used by 41 percent for many or all outcomes (see Figures 2 and 3). For each of these methods, data were mostly collected from beneficiaries and staff members at the program level.

Figure 2.

Tools or methods used by respondents to collect impact measurement data (n = 28).

Figure 3.

Respondents’ approaches to impact analysis measures (n = 37).

As indicated in Figure 3, survey respondents used comparison groups less often than other approaches, although roughly half (51%) indicated they used this type of approach. With respect to data analysis, impact analysts most often used spreadsheet analyses or ratios of benefits to costs (see Figure 4). They were much less likely to use more complicated techniques such as advanced statistical procedures or existing assessment instruments, perhaps reflecting their backgrounds, training, or years of experience in the field.

Figure 4.

Quantitative methods used by respondents to estimate impacts (n = 32).

The last major finding for impact measurement activities pertains to the presentation of impact results, including both the primary audience and how the findings are presented. Impact analysts said they took into consideration their primary audience for findings and tailored their reporting to meet the needs of this group. In general, they reported having three major aims when reporting results – informing program development, securing future financial investments, and strengthening strategic investments (see Figure SD1).

In the survey, impact analysts reported that program staff were the primary audience for their results, which is consistent with the intent of improving a program (see Figure 5). This theme of program improvement was also found in the interviews, where informants indicated that impact results were often used to develop a program’s effectiveness. However, the interviews included other stakeholders as well (e.g. investors, intermediaries, and social entrepreneurs). As such, there was an additional emphasis on funders as the primary audience of impact results. In several of the interviews, participants said the purpose of reporting impact data was to monitor financial sustainability or to progress towards meeting program objectives. In addition, stakeholders commented that impact analysts would often utilize a mixture of quantitative and qualitative data in their reporting as a way of tailoring their message to the needs of their specific audience.

Figure 5.

Primary audience for impact measurement project results (n = 31–37).

What factors influence and inform how impact analysts evaluate social impact?

Both the social impact analysis survey and the key informant interviews revealed several factors that may influence the manner in which impact is measured in a social enterprise. First, several salient themes arose from the surveys regarding the background, training, and role of analysts in social enterprises that may influence their approaches to conducting impact measurements.

The majority of respondents reported being currently employed at non-profit organizations/charities (37%; n = 38) or consultancy firms (22%; n = 22) and were between 30 and 59 years old (73%; n = 83). Nearly one third of all respondents had studied business or economics – one of the largest subgroups compared to the other disciplines. The social sciences also represented another large subgroup (see Table 1).

Table 1.

Fields in which respondents were trained.

Survey sample characteristics	n	%
Primary area of study (n = 109)
Business- or management-related	17	16%
Economics	17	16%
Political science	12	11%
Public administration/Public or social policy	9	8%
Psychology	7	6%
International-related area	6	6%
English or literature	5	5%
Geography	5	5%
History	5	5%
Other social science	5	5%
Health-related	4	4%
Science-related	4	4%
Engineering	3	3%
Math or statistics	2	2%
Other	8	7%
Total	109	100%
Highest level of education (n = 117)
Bachelor’s (including BA, BS)	28	24%
Master’s (including MA, MS, MBA)	69	59%
Doctorate (including PhD, JD, DBA)	15	13%
Other (please specify)	5	4%
Total	117	100%

The majority of survey respondents were trained at the master’s level, followed by the bachelor’s level (see Table 1). With respect to training and amount of professional experience, most respondents had completed introductory courses in research methods and evaluation, and the majority had four years of experience or fewer conducting impact analyses, while 40 percent reported experience of 5 years or more (see Figures 6 and SD2).

Figure 6.

Respondents’ completion of courses related to impact analysis (n = 66–85).

The majority of respondents indicated that they preferred relatively simple evaluation designs for their studies, such as descriptive surveys or interviews. In addition, the professional training that impact analysts received may contribute to how they measure impact in the field. The survey revealed that analysts’ top three goals in conducting impact measurement (reported by 86–93% of respondents) were influencing decision makers, measuring social impact, and improving program performance (see Figure 7). The least important goal (endorsed by 60% of the analysts) was meeting the needs of program beneficiaries, which suggests that the investors, leadership, and program administration may have the greatest influence on how they conduct their work.

Figure 7.

Respondents’ purposes of conducting impact measurement (n = 88–90).

This contrasts slightly with findings from the interviews, which were conducted mainly with investors, intermediaries, and social entrepreneurs. This group of stakeholders viewed the primary goal of impact measurement as providing evidence for the financial viability of an organization. Thus, one’s role in the social enterprise appears to be related to one’s perspective on the goals or purposes of impact analyses, which certainly influences how the measurement of social impact is carried out.

Again, drawing from the interviews, three primary factors appeared to play a role in analysts’ approaches to measuring social impact, including the complexity of the program’s theory of change (or, logic model), the age of the program, and the availability of financial resources. Impact analysts said they tended to tailor their measurement to the program’s theory of change and the outcomes upon which it focused. For example, they might capture client satisfaction using a single post-survey while attitudinal changes or learning outcomes might be measured with psychological surveys or other assessments administered over time and with more sophisticated designs (e.g. quasi-experimental, interrupted time series). Similarly, new social enterprises would be subject to less rigorous measurement approaches, while more established programs would need to demonstrate more evidence of impact.

Study data suggest that measurement and financial issues are related. That is, the availability of financial resources to conduct impact measurement plays a significant role in influencing the types and quantity of data that can be collected, analysed, and presented to various stakeholder groups. This seems reasonable, as less established enterprises would likely prefer to allocate the majority of their resources to developing and offering services. The pressure to demonstrate impact at these early stages might also not be felt as deeply, depending on the amount of investment that the enterprise has enjoyed. In contrast, we expect that organizations benefiting from larger sums of investment and that have become more established would shift their focus towards determining effectiveness. Supporting this information need would require them to distribute resources in manners that are not only different from nascent enterprises, but more importantly in ways that would allow them to be more accountable, and perhaps transparent, to their funders.

Discussion

This study documented the major measurement activities and approaches impact analysts employ when conducting impact measurement for a social enterprise, and we found several important factors that influence how impact measurement is conducted. These include the characteristics of the impact analysts themselves, the context in which their practice is situated, and the tools and approaches they elect to use. The findings contribute to a greater understanding of the many processes involved with impact measurement in the social investment sector; this study represents an attempt to capture some of the factors that contribute to this diversity and to explain how analysts view the purpose and nature of their own work in this growing field. In this section, we discuss the study findings and draw on the extant literature to further enhance our understanding of the results.

It is important to note that while our discussion draws primarily from the US evaluation literature, when possible we refer to relevant scholarship from outside of the United States. We of course acknowledge that evaluation is practiced across the globe, and the same is true for studies that examine evaluation practice. Nevertheless, because impact measurement in the context of social investment is a more recent development, the empirical and theoretical literature on it is sparse. Our intention is that this manuscript will help to address that gap.

Background characteristics of impact analysts

The background, training, and role of impact analysts appear to influence the types of approaches they employ in their measurements as well as how they view the purposes or goals of their work. This finding resonates with results from Christie and Masyn’s (2010) study, which sought to understand evaluator practice using latent profile analysis. They noted that evaluator practice can be distributed along a continuum – from indistinct, to method-focused, to user-focused, to robust. In their study, evaluators whose practices fell into the method-focused and robust categories tended to be more structured in nature. That is, these evaluators reported relying on social science and evaluation theory, along with expertise in research methods, when going about evaluations. This is perhaps a reflection of their tendency to serve as external evaluators and an artifact of their advanced degree attainment, as well as more years of evaluation experience. In contrast, those whose practices were classified as indistinct or user-focused were less grounded in theory and tended to prioritize in-the-moment information needs. These individuals identified more frequently as internal evaluators, and they possessed less formal training and evaluation experience overall.

If the assumption that impact measurement is part and parcel of evaluation holds across geographic and disciplinary contexts, then Christie and Masyn’s (2010) findings about US-based evaluators serve as an interesting point of comparison. Impact analysts in our study have primarily served in external capacities and they possess a moderate level of advanced training. These results suggest that practitioner background profiles of impact analysts in the United Kingdom and Germany differ from those of US evaluators. This not only has implications for the training of future impact analysts and evaluators, but the selection of current practitioners to join study teams or social enterprises as well.

With respect to the education of impact analysts specifically, our findings lead us to question whether formal evaluation and impact measurement training opportunities are readily available in the United Kingdom and Germany. We also wonder about the value that is placed on advanced education and experience in the impact investment context. As for the role of impact analysts, the literature indicates that the value of ‘conducting an evaluation within an organization becomes an opportunity to produce knowledge thus reflecting on and transforming the existing system of activities and practices’ (Ivaldi et al., 2015: 498). Our results lead us to question the extent to which impact analysts view their roles, contributions, and obligations as such, and more than part of a business transaction. Unfortunately, the survey sample size was too small to conduct a more detailed analysis of how background characteristics were related to impact measurement goals. A more careful examination of analysts’ background characteristics would shed additional light on this finding.

Tools for and challenges of practice

Impact analysts identified the activities and methodological approaches they used when working with organizations to assess social impact, which could potentially influence how they measure impacts. These impact analysts relied on logic models (or, some articulation of a program’s theory of action or change) and straightforward, descriptive quantitative and qualitative data techniques. So and Staskevicius (2015), through an interview-based study, had similar findings. We surmise that this practice stems from the use of logic models to evaluate development aid by organizations such as USAID – a practice also adopted by other state entities including the UK Treasury and the European Commission (Cracknell, 1996; Leeuw, 2002). The cross-disciplinary nature of impact measurement allows practitioners the freedom to employ a diverse set of approaches in their work.

While these results substantiate findings from other studies of evaluator practice and training (Christie and Masyn, 2010; Christie et al., 2014; Jarosewich et al., 2006), we remain curious about the challenges that impact analysts encounter in their attempts to measure impact, including how impact is defined across sectors and social enterprises and among investors and entrepreneurs. These issues are consequential for the impact investment field because they guide its overall development.

Relatedly, outcome definition and measurement issues are currently at the forefront of the impact measurement field. Consider the operationalization issue, for instance. If those in the impact measurement field view input (in logic model terms) as investment of private equity and output as a directly observable and enumerable unit (e.g. the number of beneficiaries served, the number of service hours delivered), then what qualifies as an outcome or impact? How is each measured? Or, is tracking and monitoring such metrics sufficient to qualify as measuring impact? And, in the event that outcomes and impact are characterized as latent, indirectly observable units, what are the implications for efforts to measure them then?

Several organizations are trying to address these challenges, including the Global Impact Investing Network (GIIN), which has developed a catalog of performance measures that investors are using to determine a social enterprise’s impact – the Impact Reporting and Investment Standards, or IRIS (GIIN, 2014). Tools such as the IRIS will certainly contribute to the impact measurement field’s evolution and influence impact analysts’ practice. Although our study did not address issues related to outcome definition and measurement specifically, they strike us as worthy of further investigation.

Context of social impact studies

With respect to contextual issues, impact analysts reported being particularly mindful of factors that bear on how a social enterprise functions. Issues that appear to influence the manner in which analysts measure social impact include the complexity, maturity, and financial standing of the social enterprise. This suggests that they are attuned to contextual issues that have been described in the evaluation theory literature. Consider, for example, Rossi and Freeman’s (1993) writings about evaluation models and methods that explicitly encourage evaluators to be mindful of a program’s stage of development, its human and fiscal resources, the political environment in which the program exists, and ways in which this information can be used to guide decisions about the evaluation’s scope and design.

Additionally, analysts reported that they consider a diverse set of audiences and their information needs when conducting impact analyses. Specifically, they believed their work supported decision-making needs of investors, social entrepreneurs, and staff of social enterprises in the areas of program improvement and demonstration of the organization’s financial sustainability. In the case of investors in particular, impact analysts reported that availability of resources affected the types of studies and the complexity of analyses that they can conduct. This seems a fairly reasoned observation since ‘most organisations that measure impact do so because they received funds from a social investor or venture philanthropists’ (Petrick and Weber, 2013: 5–6). The data collected for the present study, unfortunately, do not allow us to probe cost-related issues further as they were not of direct interest to the investigation’s research questions, but it is a relevant topic and worthy of further study in future work.

Still, these results appear to complement and expand upon findings reported in previous research about evaluator practice. Jarosewich et al. (2006), for instance, found that members of the Independent Consulting Division within the American Evaluation Association provided services to a broad client base, including non-profit organizations, community-based organizations, government agencies, and foundations (see also Bonnet, 1992). The seven areas that their services most commonly supported were program evaluation, technical assistance, organizational development, program planning, strategic planning, basic research, and performance measurement. Depending on the nature of each impact analysis and evaluation study, we anticipate that audiences and information needs will often align across these knowledge-seeking endeavors and are analogous to each other.

The juxtaposition of these findings leads us to view the audiences and foci described by the impact analysts in our investigation along with the evaluators in Jarosewich et al.’s (2006) and Bonnet’s (1992) studies as similar. Despite the different terminologies, we believe that the challenges and successes that impact analysts encounter in their efforts to support investors and entrepreneurs – including understanding clients’ underlying motivations for commissioning impact studies and defining a study’s focus – likely parallel those that evaluators have had to address.

Still, we wonder about the extent to which the differences between our study results and those reported in the evaluation literature could be an artifact of shifting economic and political landscapes. A decade has lapsed since the publication of Jarosewich et al.’s (2006) study, and more than 20 years have passed since Bonnet’s (1992) work was published. Their investigations were conducted when the world economy – and the US economy in particular – was growing. The client base and scope of practice in the evaluation field were relatively diverse at the time. Our study, in contrast, was conducted in the context of worldwide economic recovery. Changes in the political and economic landscapes since 1992 have likely contributed to the institutionalization of a sector such as impact investing and practices such as impact measurement (Bonini and Emerson, 2005; Bugg-Levine and Emerson, 2011).

Final comments

Measuring social impact is complex. Many stakeholders are involved at all levels of a social investment study, including investors, intermediaries, program staff, and beneficiaries. We found that evaluation activities intended to measure the impact of social investments may be very limited in their scope and scale, and therefore may not capture some of the more nuanced or indirect impacts of investments. Future studies should expand upon the findings presented here to gain a more in-depth perspective on the conditions under which more sophisticated inquiry might be conducted, as well as to develop a fuller understanding of the work of those who are directly responsible for impact activities. A knowledge base about impact measurement is needed if the field is expected to flourish in the same manner as its cognate disciplines.

Footnotes

Funding

This study was conducted under the joint sponsorship of the Bertelsmann Foundation and the Social Impact Analysts Association (now Social Value International).

Supplemental material

The supplementary files are available at

Notes

Anne T. Vo is Assistant Professor and Associate Director of Evaluation at the Keck School of Medicine of USC. She leads the Keck Evaluation, Institutional Research/Reporting, and Assessment Office.

Christina A. Christie is Professor and Chair of the Education Department in the Graduate School of Education and Information Studies, University of California, Los Angeles.

Kristen Rohanna is a PhD student in the Social Research Methodology Division of the Graduate School of Education & Information Studies, University of California, Los Angeles.

References

Bannick

Goldman

Lee

. (2014) International development: Subject paper of the International Development Working Group. Available at: http://www.socialimpactinvestment.org/reports/International%20Development%20WG%20paper%20FINAL.pdf.

Bebbington

Gray

Owen

(1999) Seeing the wood for the trees: Taking the pulse of social and environmental accounting. Accounting, Auditing, and Accountability Journal 12(1): 47–51.

Benford

Birnbaum

Dombrowski

(2014) Social Impact Investing: Financing Social Change. Gütersloh, Germany: Bertelsmann Stiftung.

Best

Harji

(2012) Guidebook for Impact Investors: Impact Measurement. Toronto, Canada: Venture Deli/Purpose Capital.

Bonini

Emerson

(2005) Maximizing blended value: Building beyond the blended value map to sustainable investing, philanthropy and organizations. Available at: http://www.blendedvalue.org/wp-content/uploads/2004/02/pdf-max-blendedvalue.pdf.

Bonnet

(1992) Independent evaluation consulting TIG: 1992 industry survey. Evaluation Practice 13(3): 225–31.

Brest

Born

(2013) Up for debate: When can impact investing create real impact? Stanford Social Innovation Review 22–31.

Brewer

Hunter

(1989) Multimethod Research: A Synthesis of Styles. Newbury Park, CA: SAGE.

Bugg-Levine

Emerson

(2011) Impact Investing: Transforming How We Make Money While Making a Difference. San Francisco, CA: Jossey-Bass.

10.

Christie

Masyn

(2010) Latent profiles of evaluators’ self-reported practice. Canadian Journal of Program Evaluation 2: 225–54.

11.

Christie

Quinones

Fierro

(2014) Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation 35(2): 274–90.

12.

Cracknell

(1996) Evaluating aid development. Evaluation 2(1): 23–33.

13.

Crutchfield

McLeod Grant

(2012) Forces for Good: The Six Practices of High-Impact Nonprofits, Vol. 403. New York: Wiley.

14.

Deloitte

LLP

(2014) The State of the State 2014–2015: Government’s Inflection Point. New York: Deloitte LLP.

15.

Ebrahim

Rangan

(2010) The limits of nonprofit impact: A contingency framework for measuring social performance (Working Paper No. 10–099). Cambridge, MA: Harvard Business School General Management Unit.

16.

Emerson

(2003) The blended value proposition: Integrating social and financial returns. California Management Review 45(4): 35–51.

17.

Global Impact Investing Network (2014) Impact reporting and investment standards (IRIS) fact sheet. Available at: https://iris.thegiin.org/assets/files/IRIS%20Factsheet%20August%202014.pdf.

18.

House

Howe

(2000) Deliberative democratic evaluation. New Directions for Evaluation 85: 3–12.

19.

Hwang

Powell

(2009) The rationalization of charity: The influences of professionalism in the nonprofit Sector. Administrative Science Quarterly 54(2): 268–98.

20.

Initiative for Global Development (2013) The business value of impact measurement. Available at: http://www.igdleaders.org/wp-content/uploads/Business_Value_of_Impact_Measurement_FINAL.pdf.

21.

Ivaldi

Scaratti

Nuti

(2015) The practice of evaluation as an evaluation of practices. Evaluation 21(4): 497–512.

22.

Jarosewich

Essenmacher

Lynch

. (2006) Independent consulting topical interest group: 2004 industry survey. New Directions for Evaluation 111: 9–21.

23.

Leeuw

(2002) Evaluation in Europe 2000: Challenges to a growth industry. Evaluation 8(1): 5–12.

24.

Milstein

Wetterhall

(1999) Framework for program evaluation in public health. Morbidity and Mortality Weekly Report 48(RR-11): 1–39.

25.

Nicholls

(2009) We do good things, don’t we? Blended value accounting in social entrepreneurship. Accounting, Organizations and Society 34(6): 755–69.

26.

Olsen

Galimidi

(2008) Catalog of Approaches to Impact Measurement: Assessing Social Impact in Private Ventures. New York: Rockfeller Foundation.

27.

Petrick

Weber

(2013) The social impact investment ecosystem in Germany: Input for the meeting of the social impact investing taskforce established by the G8. Gütersloh, Germany: Bertelsmann Stiftung. Available at: http://www.bertelsmann-stiftung.de/fileadmin/files/user_upload/Impact_Investing_Ecosystem_Germany.pdf.

28.

Power

(1999) The Audit Society: Rituals of Verification. Oxford: Oxford University Press.

29.

Quarter

Mook

Richmond

(2003) What Counts: Social Accounting for Nonprofits and Cooperatives. Englewood Cliffs, NJ: Prentice Hall.

30.

Reeder

Colantonio

(2013) Measuring impact and non-financial returns in impact investing: A critical overview of concepts and practice (EIBURS Working Paper No. 2013/01). London: London School of Economics and Political Science. Available at: http://files.lsecities.net/files/2013/10/Measuring_Impact-full-length-Oct-2013.pdf.

31.

Richmond

Mook

Quarter

(2003) Social accounting for nonprofits: Two models. Nonprofit Management and Leadership 13(4): 308–24.

32.

Roche

(1999) Impact Assessment for Development Agencies: Learning to Value Change. Oxford: Oxfam.

33.

Rossi

Freeman

(1993) Evaluation: A Systematic Approach, 5th edn. Newbury Park, CA: SAGE.

34.

Seymour

Ormiston

Dickson

(2013) Exploring the relationships between social impact assessment, strategy, practitioners, and praxis. Paper presented at the annual meeting of the Society for the Advancement of Socio-Economics, Milan, Italy, June.

35.

Shadish

Cook

Leviton

(1991) Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: SAGE.

36.

Sharir

Lerner

(2006) Gauging the success of social ventures initiated by individual social entrepreneurs. Journal of World Business 41: 6–20.

37.

Staskevicius

(2015) Measuring the ‘Impact’ in Impact Investing. Cambridge, MA: Harvard Business School Social Enterprise Initiative.

38.

Tashakkori

Teddlie

(1998) Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, CA: SAGE.

39.

Vanclay

(2003) International principles for social impact assessment. Impact Assessment and Project Appraisal 21(1): 5–12.

40.

Weisbrod

(2004) The pitfalls of profits. Stanford Social Innovation Review 2: 40–7.

41.

World Bank (2003) Social Analysis Sourcebook: Incorporating Social Dimensions into Bank-Supported Projects. Washington, DC: World Bank.

42.

World Bank (2016) World Bank Group support tops $61 billion in fiscal year 2016. [Press Release]. Available at: http://www.worldbank.org/en/news/press-release/2016/07/12/world-bank-group-support-tops-61-billion-in-fiscal-year-2016.