The Counterfactual Definition of a Program Effect

Abstract

Evaluators are often called upon to assess the effects of programs. To assess a program effect, evaluators need a clear understanding of how a program effect is defined. Arguably, the most widely used definition of a program effect is the counterfactual one. According to the counterfactual definition, a program effect is the difference between what happened after the program was implemented and what would have happened if the program had not been implemented, but everything else had been the same. Such a definition is often said to be linked to the use of quantitative methods. But the definition can be used just as effectively with qualitative methods. To demonstrate its broad applicability in both qualitative and quantitative research, I show how the counterfactual definition undergirds seven common approaches to assessing effects. It is not clear how any alternative to the counterfactual definition is as generally applicable as the counterfactual definition.

Keywords

counterfactual definition of an effect program effects cause and effect qualitative and quantitative methods

Evaluators often assess the effects of programs. To assess program effects, it helps to be clear about how an effect of a program is to be defined. Toward this end, I explicate a counterfactual definition of an effect. Such a definition is often said to be restricted to the use of quantitative methods such as randomized experiments and quasi-experiments. Other methods for assessing program effects are said to require different ways of thinking about cause and effect. In particular, counterfactual thinking is often said to be anathema to qualitative methods.

For example, Mohr (1999) goes so far as to define qualitative methods by the absence of counterfactual reasoning: “A design whose purpose is to determine impact will be considered qualitative if it relies on something other than evidence for the counterfactual to make a causal inference” (p. 71). Correspondingly, Mohr defines quantitative methods by the presence of a counterfactual definition of causality: “it will be helpful to define the quantitative approach to impact analysis as one that relies on the counterfactual definition of causality” (p. 71). In keeping with the distinctions he draws, Mohr provides separate definitions of causality for qualitative and quantitative methods.

Johnson et al. (2019) present multiple theories of causation and, like Mohr (1999), argue that some apply better to qualitative than to quantitative methods. In particular, Johnson et al. (2019) note that the counterfactual approach is popular among quantitative researchers but that:

In qualitative observations, we are able to see specific local causation in action, and in qualitative interviews, we are able to learn about specific local causation in action, including agency causation. There is no need for variables or counterfactuals (Mohr, 1995) in this direct approach. (p. 154)

Gates and Dyson (2017) argue that researchers need to be “familiar with multiple ways of thinking about causality” (p. 35) because different ways of thinking about causality underlie different methods. For example, they note that “counterfactual logic” (p. 35) is the basis for randomized experiments and quasi-experiments while alternative notions of causality underlie methods such as realist evaluation, process tracing, contribution analysis, and “participatory, story-centered approaches” (p. 35).

Delahais and Toulemonde (2012) also link counterfactual analysis to some methods rather than to others when they emphasize that contribution analysis enabled them “to assess impacts in a conclusive and useful manner while a counterfactual-based analysis would not have been feasible” (p. 282). And Befani and Mayne (2014) argue that “while it is common to adopt a counterfactual view on causality…there are other approaches to considering causality, in particular regularity approaches, configurational approaches and generative approaches” (p. 17) and that contribution analysis and process tracing “both seek to make causal inferences about cause and effect using non-counterfactual approaches” (p. 18).

I take a different stance. I argue that counterfactual thinking need not be limited to a narrow range of methods and, in particular, it need not be linked to quantitative methods more than to qualitative methods. It is true that counterfactual thinking is the basis for randomized and quasi-experimental designs. But counterfactual thinking can be used just as well as the basis for a wide range of other methods for assessing program effects, including qualitative methods. To present my case, I show how the counterfactual definition of an effect can be used as the foundation for seven common approaches to estimating effects. Because these seven approaches can be used by both qualitative and quantitative researchers, I suggest that the counterfactual definition is not tied to one research tradition more than to the other.

My presentation begins with a detailed explication of the counterfactual definition of an effect. I show how this definition is sufficiently flexible for the difficult task of assessing effects in program evaluation. I then describe seven approaches for assessing program effects and explain how they are compatible with a counterfactual definition of an effect. I also explain the difference between defining a cause and defining an effect, along with implications for evaluation practice.

Evaluators are free to use a definition of a program effect besides the counterfactual one. But if evaluators are to use an alternative definition of a program effect, they should be as explicit as I am here in showing how their methods for assessing program effects are consistent with their definition of program effects.

The Ideal Comparison

According to the counterfactual definition, an effect of a program is the difference between what happened after the program was implemented and what would have happened if the program had not been implemented, but everything else had been the same (Reichardt, 2006, 2011, 2019). For greater detail, consider Figure 1. Suppose a program is introduced at Time 1 (as in Figure 1) and an outcome is subsequently assessed at Time 2. Further, suppose instead that the program had not been introduced at Time 1 and an outcome had subsequently been assessed at Time 2. Finally, suppose everything else, besides the difference between implementing and not implementing the program, had been the same at Time 1. Then, the difference in outcomes at Time 2 is an effect of the program, as compared to no program.

Figure 1.

Time line and alternative outcomes for the counterfactual definition of a program effect.

Consider a commonplace example of taking a couple of aspirin. Suppose a person took two aspirin an hour ago because that person had a headache. Further, suppose that person’s headache is now gone. Also, suppose that if the person had not taken the aspirin an hour ago, but everything else had been the same at that time, the person would still have the headache now. Then, the difference between having a headache now and not having a headache now is an effect of having taken the two aspirin an hour ago, as compared to not taking the two aspirin.

The effect of a program as compared to an alternative program (rather than as compared to no program) can also be defined using the counterfactual definition. Simply substitute “Alternative Program” for “No Program” in Figure 1 and in the definition. For example, a researcher could substitute taking a placebo in place of taking no aspirin in the preceding illustration. If the person’s headache persisted after taking the placebo, the difference between having a headache now and not having a headache now is an effect of having taken the two aspirin as compared to taking a placebo. For simplicity, I will consider only comparisons of a program with no program. But the same conclusions apply for comparisons of a program with an alternative program.

The definition of an effect that I have just given is called a counterfactual definition because the what-would-have-happened-if-the-program-had-not-been-implemented-but-everything-else-had-been-the-same condition cannot arise in practice. For example, it is impossible, as required in the definition, for a person to receive a program and not to receive the program at the same time. Nor is it possible to implement a program to see what happens and then roll time backward so that the program could not be implemented, with everything else being the same. Because it cannot arise in practice, the what-would-have-happened-if-the-program-had-not-been-implemented-but-everything-else-had-been-the-same condition is contrary to fact, hence the name of a counterfactual definition of an effect.

The Multitude of Potential Program Effects

There are always a multitude of effects of a program, which can be described using five size-of-effect factors (Reichardt, 2006, 2011, 2019). The five size-of-effect factors are the program, participants, time, setting, and outcome variable. Change any one of these five factors and the size of an effect can change. Being able to specify different effects by varying the five size-of-effect factors makes the counterfactual definition sufficiently flexible to apply to program evaluation settings. Consider the multiple specifications provided by each of the five size-of-effect factors.

An effect of a program depends, obviously, on the nature of the program. A program could be anything from an all-encompassing intervention to a small intervention component. A program could be multifaceted or not. A program could take either a short or long time to be implemented. A counterfactual definition of an effect can accommodate all of these different specifications of a program.

An effect of a program depends on the participant or participants who receive the program. The participant or participants could be people, classrooms, schools, clinics, cities, and so on. An effect can vary across participants. When defining an effect as a counterfactual, a researcher need not seek to obtain a general law that holds across participants. Even if the program to be evaluated were directed to one participant or one group of participants, an effect can be defined for (and can vary across) individual participants. A counterfactual definition of an effect can apply whether the focus is on idiographic or nomothetic outcomes.

An effect depends on the chronological time at which the program is introduced and the time lag between when the program is introduced and an effect is assessed. Vary either the chronological time or time lag and an effect can vary. For example, a program to increase job training could have a different effect during the time of an economic expansion than during the time of an economic downturn. And an effect varies depending on whether the time lag is specified to be one minute or an one hour after taking aspirin. So, specifying the time at which the program is introduced and an effect is assessed must be part of the specification of an effect. By specifying chronological times and time lags, a counterfactual definition can be applied (multiple times if needed) even if program effects are nonlinearly dynamic or have feedback loops.

A program effect depends on the setting or context in which the program is implemented. For example, the effect of an educational program might be different whether implemented in a classroom or in an after-school setting. If the effect of a program depends on the presence of other programs or other causes besides the program being evaluated, the other programs and causes become part of the setting or context. So in one setting (with other causes present), a program might have one effect while in another setting (with the other causes not present), the program might have a different effect or no effect at all. Among other things, this specification of the setting means the counterfactual definition does not presume that an effect has only a single cause. If a program has an effect only in the presence of other causes, those other causes are part of the setting. By specifying the details of the setting or context, a researcher can use a counterfactual definition even when multiple causes or when program interactions are operating to produce an effect.

An effect depends on the outcome variable being assessed. For example, an educational program can have a different effect on reading achievement than on math achievement. As a result, specifying an effect requires specifying the outcome variable.

By specifying different programs, participants, times, settings, and outcome variables, a counterfactual definition of an effect can apply to any patterns of causes and effects. For example, suppose there is a complex pattern of effects whereby Program X causes a change in Outcome Y, which at a later time recursively causes a change in Program X. The counterfactual definition can capture both effects by specifying two separate causal patterns: first, that Program X causes a change in Outcome Y at a first point in time and, second, that a change in Outcome Y causes a change in Program X at a later point in time.

Seven Approaches to Assessing Program Effects

As noted, the counterfactual comparison that defines an effect is a Platonic ideal in the sense that the counterfactual comparison is impossible to obtain in practice. Instead, an evaluator must assess a program effect in some alternative way. For evaluators who subscribe to a counterfactual definition of an effect, the alternative way requires that the contrary-to-fact outcome (i.e., what would have happened if the program had not been implemented but everything else had been the same) is replaced by an estimated counterfactual condition.

The next seven sections describe seven approaches to assessing program effects, which use estimated counterfactual conditions. The seven approaches to assessing program effects are comparisons across participants, before–after comparisons, what-if assessments, just-tell-me assessments, direct observations, theories-of-change assessments, and the modus operandi (MO) method. All seven approaches can be used by either qualitative or quantitative researchers though some tend to be used more by one type of researcher than the other.

Approach 1: Comparisons Across Participants

In a comparison across participants, the performances of two separate groups of research participants are assessed, where one group receives the program and the other group does not. An effect of the program is assessed by comparing the performances of the participants in the two groups, after the program was given to the program participants. The performances of the participants who do not receive the program are used to assess how the participants who did receive the program would have performed had they not received the program. How the participants who did not receive the program perform is the replacement for the impossible-to-obtain counterfactual outcome. That is, a comparison across participants uses an estimated counterfactual condition. So, estimating a program effect using a comparison across participants is undergirded by a counterfactual definition of an effect.

If the participants in a comparison across participants were assigned to the two different program conditions at random, the comparison would be a randomized experiment. Otherwise, the comparison across participants is a quasi-experiment. More elaborate quasi-experiments (such as the regression discontinuity design) are also possible. Randomized and quasi-experiments are usually labeled as such by quantitative researchers. If qualitative researchers assess an effect of a program by comparing the performances of participants who receive the program to the performances of participants who do not receive the program, they are conducting either a randomized or a quasi-experiment, whether they use those labels or not.

Approach 2: Before-After Comparisons

In a before-after comparison, the outcomes of one or more participants are assessed (a) at a single point in time based on the results before they receive the program (the pretest) and (b) at a single point in time after they receive the program (the posttest). The pretest outcome can be collected either prospectively or retrospectively (Hill & Betz, 2005; Pratt et al., 2000; Shadish et al., 2002). Prospective pretests are collected before the program is introduced. Retrospective pretests (also called thentests) are collected after the program is implemented. With retrospective pretests, program participants are asked to think back in time to before the program was implemented and report what their outcomes had been at that point in time.

With either prospective or retrospective pretests, the effects of the program are assessed by comparing the pretest (or thentest) performances to the posttest performances of the program participants. The pretest (or thentest) serves as the replacement of the counterfactual condition for what would have happened if the program had not been implemented. In this way, a before-after comparison uses an estimated counterfactual condition. So, estimating a program effect using a before-after comparison can be justified by a counterfactual definition of an effect.

Elaborations of a before-after comparison are possible wherein performances are assessed at multiple time points before and/or after the program is introduced. One such comparison might be an interrupted time-series design where the estimated counterfactual condition is derived from an estimated trend line. In either elaborated or simple form, before-after comparisons are quasi-experiments and are usually labeled as such by quantitative researchers. If a qualitative researcher assesses an effect of a program by drawing a before-after comparison, the qualitative researcher is performing a quasi-experiment whether the comparison is labeled that way or not.

Approach 3: What-If Assessments

Evaluators (most commonly qualitative evaluators) sometimes ask program participants to speculate about how they would have performed if they had not participated in the program. For example, Leeuw (2012) notes that evaluators sometimes ask participants “if and to what extent their behaviour would have been changed if policy intervention Y had (not) been implemented” (p. 359). Similarly, Rathbun (2008) says “Another useful method is to ask counterfactuals in the form of ‘why didn’t you do X’ or ‘what if you had done Y in situation Z?’” (p. 693). With answers to these questions, an evaluator assesses the effect of the program by comparing the actual performance of participants after they received the program to how the participants say they would have performed if they had not received the program. Abell (2004) calls such a comparison a “subjective counterfactual” (pp. 295–296), and Mueller et al. (2014) call such a comparison a “counterfactual self-estimation of program participants” (p. 8). I call it a what-if assessment.

Alternatively, it has been suggested that evaluators could ask experts to predict what would have happened if participants had not received the program (BetterEvaluation, 2013). Then, the effects of the program would be assessed by comparing the actual performance of participants who received the program to the experts’ predictions of what would have happened if the participants had not received the program.

It is not usually specified how the program participants or experts are to derive their what-if answers. Lam and Valencia (2019) argue that participants make a what-if assessment by thinking retrospectively of how they had performed before the program began. But regardless of how the what-if answer is derived, it serves as an estimated counterfactual condition. That is, the answer to the what-if question serves as the replacement for the impossible-to-obtain counterfactual outcome. In this way, estimating a program effect using a what-if assessment is consistent with a counterfactual definition of an effect.

Approach 4: Just-Tell-Me Assessments

Another assessment option (most often used by qualitative evaluators) is to simply ask participants to attribute effects to a program. As Copestake (2014) explains, “If we are interested in finding out whether particular men, women or children are less hungry as a result of some action it seems common-sense just to ask them” (p. 413). I call this approach the just-tell-me assessment. In answering such a just-tell-me question, an evaluator could have program participants either think forward or backward.

In thinking forward (from cause to effect), the evaluator identifies the program that is being evaluated and asks participants to identify effects of the program. What participants identify as the effects are taken to be the effects (perhaps after vetting the responses based on expert opinion). It is not usually specified how program participants are to answer such forward-thinking just-tell-me questions. One obvious approach would be for participants to imagine not having received the program and speculate about what would have happened subsequently. The effect of the program would then be assessed by comparing what actually happened after the program was received to the speculation about what would have happened if the program had not been received. In that case, the speculation about what would have happened serves as a replacement for the impossible-to-obtain counterfactual outcome.

In thinking backward (from effect to cause), the evaluator first asks participants about the changes that have occurred on the outcome variables of interest and then asks them what caused the changes. For example, Copestake (2019) explains that the Qualitative Impact Protocol (QuIP) asks “respondents what major changes they have experienced in each domain during a specified time period and then encourages them to elaborate on what they think is driving these changes” (p. 35). If the participants mention the program as being a driver of the changes, the program is taken to be a cause of at least some of the observed changes. Like the QuIP, both Outcome Harvesting (Wilson-Grau & Britt, 2013) and the Success Case Method (Brinkerhoff, 2005) also rely on just-tell-me assessments. (Also see Gates and Dyson (2017), Roughley and Dart (2009), and White and Phillips (2012)).

As with forward-thinking just-tell-me assessments, it is not usually specified how program participants are to answer a backward-thinking just-tell-me assessment. But an obvious option is for the participants to again use an estimated counterfactual condition. The participants would think back over time to speculate about what might have been a cause of the observed changes. Having found a potential cause, the participants would then speculate about what would have happened if that cause had not been present. If the speculated cause were the program, the effect of the program would be assessed by comparing the outcomes that actually took place to the outcomes that were speculated to have taken place if the program had not been implemented. In this way, the speculations about what would have happened if the program had not been implemented serve as a replacement for the impossible-to-obtain counterfactual outcome.

It is possible an alternative, noncounterfactual method of answering the just-tell-me questions could be used, but, as far as I know, such an alternative has not been specified by evaluators. In any case, using an estimated counterfactual condition would be a natural way to implement just-tell-me assessments.

Approach 5: Direct Observation

Patton (2015) and Scriven (2005b, 2008, 2009) argue that, under certain conditions, causality can be assessed through direct, critical observation. According to Scriven (2008), direct observation of cause and effect entails an observer simply “seeing causation, not inferring it” (p. 19). In contrast, I would argue that though direct observation of cause and effect might appear to involve no cognition besides basic perception, it in fact requires the inferential use of estimated counterfactual conditions.

Consider the example Scriven (2005a, 2008, 2009) gives wherein he says a driver can directly observe that pushing on the brakes of a car causes the car to slow down. On close inspection, the purported direct observation of this cause and effect relationship involves an estimated counterfactual condition in the form of a before-after comparison. That is, to determine that pushing on the brake causes the car to slow down, the observer compares what happens before the brake is pushed to what happens after the brake is pushed. For example, suppose the car was already slowing down before the brake was pushed. Then, any further slowing after the (potentially malfunctioning) brake was pushed might simply be a continuation of the prior slowing and would not automatically lead to the conclusion that pushing on the brake was the cause of further slowing. In other words, an observer only knows a brake is working by comparing what happened before the brake was pushed with what happened after the brake was pushed.

Conversely, researchers are not fooled by purported direct observation when they determine that a rooster’s crowing in the morning does not cause the sun to rise. Researchers are not misled because they have extensive knowledge of what happens when roosters do not crow in the morning, which are estimated counterfactual conditions. That is, because the sun still rises on those occasions when the rooster does not crow, researchers know the rooster’s crowing has no effect. So, what might appear to be critical observations that are made automatically without inference do, in fact, require inferences involving an estimated counterfactual condition.

Approach 6: Theories-of-Change Assessments

A sixth approach to assessing effects is the theories-of-change approach (Astbury & Leeuw, 2010; Coryn et al., 2011; Weiss, 1997). A number of evaluation strategies assess program effects by relying on the theories of change of a program. I am referring to strategies that go by such names as theory-driven evaluation (Chen, 1990), process tracing (Beach & Pedersen, 2019; George & Bennett, 2004), contribution analysis (Mayne, 2008, 2012), and realist evaluation (Mark et al., 1998; Pawson & Tilley, 1997).

Note that a theories-of-change approach can be used in either of two ways. It can be used to determine which causes are responsible for observed outcomes. Or it can also be used to assess what effects are the result of a given cause. (See the section Alternative Definitions of Causes and Effects.) In keeping with the present focus, I am concerned only with the use of the method for assessing program effects.

In assessing program effects, a theories-of-change approach elaborates the theory of change by which the program to be evaluated operates. That is, a theory of change specifies the purported mechanisms or causal linkages that connect program activities to outcomes. After elaborating a program’s theory of change, the evaluator collects data to see whether the program was implemented as specified and whether the participants were partaking of program activities as planned. That is, the researcher determines whether the purported mechanisms and causal linkages in the program theory are present. The evaluator also collects data on the presumed mechanisms identified in the theories of change of alternative explanations for observed outcomes. If the chain of mechanisms of the program’s theory of change is found to be present and evidence for alternative explanations is found not to be present, the evaluator concludes that observed outcomes are due to the program.

It is sometimes claimed that such theories-of-change approaches assess the effects of programs without using estimated counterfactual conditions. For example, Waldner (2016) concludes “Insofar as one has the full set of invariant causal mechanisms, one can use the fully identified causal model as a substitute for the missing observations about the counterfactual states of the world” (p. 31). Cook (2000), Gerring (2010), and Leeuw (2012) attribute similar sentiments to others as well. But I believe those sentiments are incorrect. That the theories-of-change approach relies on counterfactual comparisons can be illustrated in three ways.

First, to implement the theories-of-change method to assess program effects, the evaluator establishes that a difference in outcomes has occurred following the implementation of the program. For example, in their theory-of-change assessment, Befani and Mayne (2014) specify that the evaluator first shows that performance improved and then specify that one of the main questions to ask is “Can the improved outcomes be ‘claimed by’ the intervention?” (p. 27). In their QuIP method, which employs a theories-of-change analysis, Copestake et al. (2019) are equally clear about establishing that a difference is present when they note the QuIP approach requires “…change monitoring: the empirical task of measuring the direction and magnitude of change in these selected goals over time (or proxy indicators of them)” (p. 2). And in discussing theories-of-change approaches, White and Phillips (2012) list “What is the observed change in outcomes of interest?” (p. 25) as an important early question to be asked. Only by demonstrating that a change over time is present is there prima facie evidence that the program being evaluated makes a difference. If a theories-of-change approach determines that the program being evaluated had an effect, the approach uses such observed differences in outcomes over time to determine the size of the effect. In this way, a theories-of-change approach uses an estimated counterfactual condition in the form of a before-after comparison.

Second, in theories-of-change approaches, evaluators collect data on whether the program was implemented as specified and whether the participants were partaking of the program activities as planned. That is, the researcher determines whether the purported mechanisms and causal linkages in the program theory are present and operating. For example, if a job training program is meant to provide job skills, the evaluator determines whether training was taking place and whether the presumed job skills were acquired. If a job training program is meant to help with obtaining and successfully completing job interviews, the evaluator determines whether these activities were taking place. If the job training program is meant to cause participants to obtain jobs, the evaluator determines the extent to which meaningful employment was attained. But just knowing that a sequence of events took place would not reveal the size of the effects of each of these program linkages (and in particular would not reveal whether each linkage had any effect at all). Just because a sequence of events is present does not mean the program events caused the outcomes that were observed. To assess effects, the evaluator would need to know more. One way to obtain the needed information would be to use estimated counterfactual conditions. That is, evidence that a causal linkage or mechanism of a program component is present and having an effect could be obtained by determining that an outcome would not have occurred without the program. Such determinations could be obtained using what-if assessments, just-tell-me assessments, or direct observation. In any case, estimated counterfactual conditions are likely involved if the causal linkages of the program are to be claimed to not just be present but to have effects.

Third, consider the analogies that Bennett (2010) uses to describe theories-of-change approaches:

This mode of analysis is closely analogous to a detective attempting to solve a crime by looking at clues and suspects and piecing together a convincing explanation, based on fine-grained evidence that bears on potential suspects’ means, motives, and opportunity to have committed the crime in question. It is also analogous to a doctor trying to diagnose an illness by taking in the details of a patient’s case history and symptoms and applying diagnostic tests that can, for example, distinguish between a viral and a bacterial infection. (p. 180)

Note how a criminal detective and a doctor operate to determine that an outcome is the result of a given crime or disease. A criminal detective demonstrates that an injury of some sort has occurred and then determines based on counterfactual reasoning that the subsequent injury would not have occurred except for the crime. Such thinking applies the but for rule in legal arguments, which specifies a defendant is guilty only if an injury would not have occurred but for the defendant’s actions. Similarly, a doctor diagnoses illnesses using counterfactual thinking wherein it is determined that the patient would not have the given symptoms except for the presence of the specified illness. The point is that counterfactual thinking is consistent with the logic of theories-of-change approaches.

I am not alone in the conclusion that counterfactual thinking is involved in theories-of-change approaches for assessing program effects. Ricks and Liu (2018) state “Counterfactuals are vital to process-tracing” (p. 844). White and Phillips (2012) state “Thus, mechanism-based explanations include implicit counterfactuals; they set out not only to find rigorous empirical evidence that supports the assumptions of one explanation, but also to plausibly demonstrate that it is absent for alternative counterfactual hypotheses” (p. 18). (Also see Beach (2016), Coryn et al. (2011), Fearon (1991), Leeuw (2012), Leeuw and Vaessen (2009), Patton (2015), Rohlfing and Schneider (2018), and White (2011)).

Approach 7: The MO Method

Scriven (1976, 2009) champions the MO method. The logic of the method is that an evaluator identifies empirical signatures (also called tracers or telltales) of the program and investigates to see whether the signatures are present. The presence of signatures of the program and the absence of signatures for alternative causes suggest the program was responsible for observed results. Scriven (1976) gives an example of evaluating “the performance of a small scale bureau we set up on a campus to improve undergraduate and other teaching” (p. 109). Scriven suggests that a signature for such a program might consist of specific wordings of procedures recommended by the bureau. Then, the evaluator monitors “the blood stream of information through the university later” to “detect the passage of ‘signed’ material and assess deterioration, implementation, and so forth” (pp. 109–110). Scriven goes on to explain:

MO analysis of this sort can often show that the kind of assistance rendered simply was not available elsewhere, either when required or later. Nor would it have been available in the absence of the bureau because it would have required expertise and time only a central office could supply. (p. 113)

That the assistance rendered by the bureau would not “have been available in the absence of the bureau” is an estimated counterfactual outcome. A signature, of course, is something unique to a person or entity. In particular, a signature of the bureau Scriven is evaluating is something that would be present if the bureau were operating and not present in the absence of the bureau. In this way, a signature implies a counterfactual comparison. To show that the bureau was having an effect, the MO method is relying on counterfactual reasoning of what happens both when a program is and is not present.

Alternative Definitions of Causes and Effects

To avoid confusion when addressing issues about cause and effect, it is important to distinguish between two questions: the causes-of-effects question and the effects-of-causes question (Goertz & Mahoney, 2012; Holland, 1986; Mahoney & Goertz, 2006; Reichardt, 2006, 2011, 2019). The causes-of-effects question specifies an effect and asks what is the cause of that effect. Conversely, the effects-of-causes question specifies a cause and asks what is the effect of that cause. There are other causal questions as well: How did the program have an effect? What component of the program had an effect? and Will the effect generalize to other settings? (Stern et al., 2012). But I will focus in this section only on the two questions of causes of effects and effects of causes.

The causes-of-effects question is one of the primary causal questions asked, for example, by forensic investigators and automobile mechanics. Such experts are presented with an effect and need to know what caused it. For example, a forensic investigator is presented with a structure destroyed by fire and is asked to determine whether the fire was caused by arson or otherwise. In a similar fashion, an automobile mechanic is presented with a malfunctioning vehicle and hunts for the cause of the problem. In contrast, the effects-of-causes questions reverse the role of cause and effect. In the effects-of-causes question, researchers are presented with a cause and seek to know its effects. When researchers are asked to assess the effects of a given program, they are being asked to answer the effects-of-causes question.

Of course, both questions can play a significant role in program evaluation. Consider an example involving a job training program. Before a job training program is designed and implemented, a pool of previously successful job applicants might be interviewed with the intent of soliciting their perspective on how they were able to obtain jobs without the program. This is the causes-of-effects question. An effect is present (the acquisition of jobs), and the cause is sought (what is responsible for the acquisition of jobs). Based on the responses of the successful job applicants, a program for job training might be designed and implemented. At that point, a major question to be asked is whether the program has its intended (perhaps as well as unintended) effects. This is the effects-of-causes question. A cause (the job training program) is present, and the effects (trainees obtaining jobs) are sought.

In the preceding example, researchers began with the causes-of-effects question and proceeded to the effects-of-causes question. The reverse sequence can also be the case. That is, researchers can start with the effects-of-causes question, which leads them subsequently to the causes-of-effects question. For example, when evaluators assess the effects of a program, they are asking the effects-of-causes question. But in answering that question, evaluators also ask about the presence of confounds, which is to ask about what other causes of observed outcomes might be present—which is the causes-of-effects question. In other words, to ask which threats to internal validity might be present in an observed outcome is to ask about the causes of effects.

Although both causal questions are involved in program evaluation, the questions are nonetheless distinguishable. So as not to be confused, it is important to be clear about which question is being asked at any given stage of an evaluation.

It is important to be clear about which causal question is the focus at a given stage of inquiry because answers to the two causal questions require different definitions. The causes-of-effects question requires a definition of what qualifies as a cause while the effects-of-causes question requires a definition of what qualifies as an effect. Let me explain. In the effects-of-causes question, a researcher is given an effect and needs to identify a cause. To do so, a researcher needs to know what qualifies as a cause. So, a researcher needs to know how a cause is defined—to know whether a researcher has found one. In contrast, in the effects-of-causes question, a researcher is given a cause and needs to identify an effect. To do so, a researcher needs to know what qualifies as an effect. So, a researcher needs to know how an effect is defined—to know whether an effect has been found. And the two definitions can be dramatically different.

Consider the causes-of-effects question, which requires a definition of a cause. Numerous definitions of a cause are available (Mahoney, 2008; Mahoney et al., 2009; Mayne, 2012; Tacq, 2011). Some of the different definitions of a cause are that a cause is (1) a necessary condition for an effect, (2) a sufficient condition for an effect, (3) both a necessary and a sufficient condition for an effect, (4) an insufficient but necessary part of a condition that is itself unnecessary but sufficient for an effect (an INUS condition; Mackie, 1965), (5) a sufficient but unnecessary part of a condition that is insufficient but necessary for an effect (an SUIN condition; Mahoney, 2008), (6) a condition that raises the probability that an effect occurs (Mahoney, 2008), (7) a condition that is always or regularly followed by an effect (i.e., a “regularity” definition often attributed to Hume, 1748), (8) a condition that results in an effect via manipulation (Brady, 2008), (9) a condition that occupies “a necessary slot in the physical causal scenario pertinent to” the effect (physical causation; Mohr, 1999, p. 73), (10) a condition where there is a mechanism that links the cause to its effect (Johnson et al., 2019), and (11) a counterfactual condition such that if the cause had not been, the effect would not have been (Hume, 1748).

In contrast, I have focused on a single definition of an effect. That definition is the counterfactual definition. The counterfactual definition is commonly used throughout science including in economics (Angrist & Pischke, 2009), education (Rubin, 1974), political science (Goertz & Mahoney, 2012), psychology (West & Thoemmes, 2010), sociology (Morgan & Winship, 2015), and statistics (Holland, 1986). (However, it is worth noting that the counterfactual definition has sometimes been given different names such as the “potential outcomes” definition or the Rubin, 2005, causal model.) It is commonplace for a methodology of assessing effects to be built explicitly on the counterfactual definition (Reichardt, 2019; Shadish et al., 2002).

Methods for assessing causes differ from the methods for assessing effects. To confuse the definition of a cause for the definition of an effect can lead to confusion about appropriate methods. For example, methods (such as qualitative comparative analysis) used to determine a cause as defined by either an INUS or an SUIN condition are not the same as the methods (as described herein) used to assess a counterfactual effect (Mahoney, 2008; Pattyn et al., 2019). Providing methods for one purpose does not provide methods for the other. So, when assessing effects, do not be led to believe that a definition of a cause provides a definition of an effect.

Not only do methods for assessing causes differ from methods for assessing effects, methods for assessing effects depend on the definition of an effect that is used. Consider an alternative to the counterfactual definition of a program effect that might be used. To take an extreme example, instead of the counterfactual definition, suppose an evaluator defines a program effect as an event that necessarily follows after a program is implemented. To assess a program effect assuming such a necessity definition, a researcher would collect data on the outcomes of a substantial number of participants who received the program to see whether the outcome always followed. Such an accounting is not the focus of any of the seven approaches to assessing effects I have described. In contrast, as I have shown, the counterfactual definition of an effect provides an adequate foundation for all the different approaches to assessing program effects considered here.

Threats to Validity

Not only do the methods for assessing effects depend on how an effect is defined, the specification of potential biases that can arise in assessing effects depends on how an effect is defined. For example, the well-known threats to internal validity from the Campbellian tradition (Campbell & Stanley, 1966; Cook & Campbell, 1979; Shadish et al., 2002) were derived based on a counterfactual definition of an effect. Let me explain. As noted, with the counterfactual definition, what happens after a program was implemented is compared with what would have happened had the program not been implemented. In the ideal comparison, such a difference in outcomes is specified to arise with everything else having been the same. In practice, the counterfactual condition where everything else is the same must be replaced with an estimated counterfactual condition. A threat to internal validity is the difference between the true counterfactual condition and the estimated counterfactual condition. In other words, a threat to internal validity is specified in the context of the counterfactual definition of an effect. Researchers who use the Campbellian notion of threats to internal validity are using conceptualizations that presuppose the counterfactual definition of an effect. The Campbellian threats to internal validity would not necessarily apply if an evaluator used an alternative definition of an effect such as with the necessity definition.

Threats to internal validity can arise in all seven of the approaches to assessing effects. In essence, all of the approaches for assessing program effects can be undergirded by either a comparison across participants or a before-after comparison. For example, the what-if assessment draws a comparison between what happened after a program was implemented and what participants (or experts) speculate would have happened if the program had not been implemented. Such speculation can be derived by considering how the participants behaved before the program was implemented (a before-after comparison) or by considering how participants behaved who did not participate in the program (a comparison across participants). Similarly, theories-of-change approaches rely on before-after comparisons in demonstrating that change has occurred that might be due to the program. And in the direct observation approach, I argued that the effect of pressing on the brake is assessed by comparing the speed of the car before the brake was pressed to the speed of the car after the brake was pressed (a before-after comparison).

The threats to internal validity that can arise in comparison across participants can be different than the threats to internal validity that can arise in before-after comparisons. Comparisons across participants are particularly susceptible to the threats to internal validity due to selection differences (including interactions with selection), differential attrition, and noncompliance with the program protocols. Selection differences, for example, are initial differences between the participants who receive the program and the participants who do not receive the program. The presence of such differences can produce differences in outcomes between the groups of participants separate from any program effects and thereby bias the assessment of program effects. In contrast, before-after comparisons are particularly susceptible to threats to internal validity due to history, maturation, attrition, testing, instrumentation, cyclical effects, and regression toward the mean (Reichardt, 2019; Shadish et al., 2002). For example, history effects are events, other than the program being evaluated, that take place at the same time the program is implemented in a before-after comparison. In the example given in the section on assessment by direct observation, a history effect would arise if the engine died at the same time the car brake was pushed, so the effect of the car slowing down might be mistakenly attributed to the brake rather than to the history effect of the engine failure. Threats to internal validity are well explicated in the quantitative literature, and evaluators of all stripes are referred to that literature (e.g., Reichardt, 2019; Shadish et al., 2002).

Evaluators should take threats to internal validity into account when designing studies, collecting data, and interpreting outcomes. For example, in what-if and just-tell-me assessments, evaluators should bring the potential threats to internal validity to the attention of participants (or experts) when they make their estimates of counterfactual outcomes. To illustrate, evaluators might ask whether participants are using a before-after comparison or a comparison across participants in making their judgments in what-if or just-tell-me assessments. Then, evaluators would explicate the relevant threats to internal validity and walk the participants through them. To consider just one threat to internal validity, if the participants were using a before-after comparison, the evaluator could ask the participants whether some other historical event might have taken place at the same time the program was implemented, where that event might have been responsible for some or all of the changes in outcomes. The point is that what-if and just-tell-me assessments should be derived only after participants take account of potential threats to internal validity.

Similarly, when evaluators (rather than the participants or experts) are responsible for drawing the comparisons upon which assessments are based, the evaluators should also consider each threat to validity in turn. For example, Scriven (2008) cites an evaluation of an aid program in East Africa and specifies that,

after determining that a substantial improvement in welfare has followed the arrival of aid, and has been sustained for a few years, we check for the presence of more than a dozen other possible causes of this observed subsequent increase in welfare, including efforts by the country’s government that have actually trickled down to the village level, analogous effects by other philanthropies, self-help gains resulting from inspired leadership in the local communities, increased income from family members traveling to well-paid job openings elsewhere and remitting money back home, increased prices for milk or calves in the local markets, the beneficial results of a few years of good weather or the improved water supply, or of technology-driven improvements in the quality of available commercial feed, veterinary meds or services, or grass seed for improving pastures. (p. 22)

An emphasis on threats to internal validity is in keeping with Scriven’s (2008) general elimination methodology (GEM), which is used to derive inferences about program effects by ruling out alternative explanations. According to Scriven (2008), the GEM is the basis for all causal inference, whether qualitative or quantitative.

Conclusions

A question to ask at this point is whether there are viable alternatives to the counterfactual definition of an effect upon which to base methods for assessing effects. There are numerous discussions of alternative ways to conceptualize causality but, as far as I know, few, if any, alternative definitions of an effect in the literature.

For example, consider Gates and Dyson (2017). The first sentence in their abstract makes it clear Gates and Dyson (2017) are concerned with estimating program effects: “Making causal claims is central to evaluation practice because we want to know the effects of a program, project, or policy” (p. 29). One of the major purposes of the Gates and Dyson (2017) article is to help people become “literate in multiple ways of thinking about causality” (p. 29). And Gates and Dyson (2017) explicate five ways of thinking about causality, which are the successionist, narrative, generative, causal package, and complex systems ways. In describing these different ways of thinking about causality, Gates and Dyson (2017) provide a few different definitions of causes but do not give a definition of an effect that differs from the counterfactual definition.

The article by Johnson et al. (2019) is similar. One of the primary purposes of their article is to explicate the “major theories and concepts of causation” (p. 144) to provide “some powerful options for thinking about causation” (p. 147). To do so, Johnson et al. (2019) detail seven “major theories of causation” (p. 147) that they label probabilistic causation, counterfactual causation, regularism, necessary and sufficient causation, manipulation and invariance causation, mechanistic causation, and agency causation. They argue that “the more theories of causation that are logically satisfied, the stronger one’s evidence of causation” (p. 158). But like Gates and Dyson (2017), Johnson et al. (2019) provide explicit definitions of causes but do not devote attention to providing a definition of an effect that differs from the counterfactual definition.

Finally, consider Mohr (1999) who emphasizes two definitions of causality: the physical and the factual. Factual causality is based on the counterfactual definition of cause and effect. The alternative is physical causality in which there is “a physical connection in the natural world between the proposed cause and effect” (p. 71). Mohr gives examples of physical causes and effects but provides no definition of an effect that differs from the counterfactual definition.

The point is that none of the three articles mentioned here provides an explicit definition of an effect that is a viable alternative to the counterfactual definition of an effect. Certainly, none provides an explicit alternative definition of an effect that is shown to be capable of undergirding common approaches to assessing effects.

I suspect there is no viable alternative to the counterfactual definition of an effect and that when the counterfactual definition is not given explicitly, it is being used implicitly. But whether my suspicions are correct, there is much to be gained by acknowledging explicitly whatever definition of an effect is being used. If an evaluator is to estimate program effects, the evaluator needs to know how an effect is defined. Without knowing how an effect is defined, it is not possible to know whether an estimate of a program effect has been adequately derived nor if biases have been adequately addressed.

I have shown that seven approaches for assessing program effects are compatible with a counterfactual definition of an effect. Extensive literatures exist on using these approaches for assessing program effects. Extensive knowledge also exists about potential biases that can arise when using these approaches and how to cope with these potential biases. Evaluators might consider using the counterfactual definition of a program effect so they can make use of such literatures and knowledge.

Of course, evaluators are free to use an alternative to the counterfactual definition of a program effect, if an adequate alternative can be found. But if an alternative definition is used, evaluators should explicitly describe that alternative definition and forthrightly demonstrate how their definition undergirds their methodology—as I have done here for the counterfactual definition of an effect.

Footnotes

Acknowledgment

I thank Mel Mark, the associate editor, and four anonymous reviewers for their very helpful comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Charles S. Reichardt

References

Abell

(2004). Narrative explanation: An alternative to variable-centered explanation? Annual Review of Sociology, 30, 287–310.

Angrist

J. D.

Pischke

J.-S.

(2009). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Astbury

Leeuw

F. L.

(2010). Unpacking the black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381.

Beach

(2016). What are we actually tracing? Process tracing and the benefits of conceptualizing causal mechanisms as systems. Qualitative and Multi-Method Research, 14(1/2), 15–22.

Beach

Pedersen

R. B.

(2019). Process-tracing methods: Foundations and guidelines (2nd ed.). The University of Michigan Press.

Befani

Mayne

(2014). Process tracing and contribution analysis: A combined approach to generative causal inference for impact evaluation. IDS Bulletin, 45(6), 17–36.

Bennett

(2010). Process tracing and causal inference. In Brady

H. E.

Collier

(Eds.), Rethinking social inquiry: Diverse tools, shared standards (pp. 179–189). Rowman & Littlefield.

BetterEvaluation. (2013). Understand causes of outcomes and impacts. http://betterevaluation.org

Brady

H. E.

(2008). Causation and explanation in social science. In Box-Steffensmeier

J. M.

Brady

H. E.

Collier

(Eds.), The Oxford handbook of political methodology (pp. 217–270). Oxford University Press.

10.

Brinkerhoff

R. O.

(2005). Success case method. In Mathison

(Ed.), Encyclopedia of evaluation (pp. 402–403). Sage.

11.

Campbell

D. T.

Stanley

J. C.

(1966). Experimental and quasi-experimental designs for research. Skokie, IL: Rand McNally.

12.

Chen

H. T.

(1990). Theory-driven evaluations. Sage.

13.

Cook

T. D.

(2000). The false choice between theory-based evaluation and experimentation. New Directions for Evaluation, 87, 27–34.

14.

Cook

T. D.

Campbell

D. T.

(1979). Quasi-experimentation: Design and analysis issues for field settings. Rand McNally.

15.

Copestake

(2014). Credible impact evaluation in complex contexts: Confirmatory and exploratory approaches. Evaluation, 20(4), 412–427.

16.

Copestake

(2019). Comparing the QuIP with other approaches to developmental impact evaluation. In Copestake

Morsink

Remnant

(Eds.), Attributing development impact: The qualitative impact protocol (QuIP) case book (pp. 29–57). Practical Action Publishing.

17.

Copestake

Remnant

Morsink

(2019). Introducing the causal attribution challenge and the QuIP. In Copestake

Morsink

Remnant

(Eds.), Attributing development impact: The qualitative impact protocol (QuIP) case book (pp. 1–27). Practical Action Publishing.

18.

Coryn

C. L. S.

Noakes

L. A.

Westine

C. D.

Schröter

D. C.

(2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.

19.

Delahais

Toulemonde

(2012). Applying contribution analysis: Lessons from five years of practice. Evaluation, 18(3), 281–293.

20.

Fearon

J. D.

(1991). Counterfactuals and hypothesis testing in political science. World Politics, 43(2), 169–195.

21.

Gates

Dyson

(2017). Implications of the changing conversation about causality for evaluators. American Journal of Evaluation, 38(1), 29–46.

22.

George

A. L.

Bennett

(2004). Case studies and theory development in the social sciences. The MIT Press.

23.

Gerring

(2010). Causal mechanisms: Yes, but…. Comparative Political Analysis, 43(11), 1499–1526.

24.

Goertz

Mahoney

(2012). A tale of two cultures: Qualitative and quantitative research in the social sciences. Princeton University Press.

25.

Hill

L. G.

Betz

D. L.

(2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501–517.

26.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–970.

27.

Hume

(1748). An enquiry concerning human understanding. Bebbington.

28.

Johnson

R. B.

Russo

Schoonenboom

(2019). Causation in mixed methods research: The meeting of philosophy, science, and practice. Journal of Mixed Methods Research, 13(2), 143–162.

29.

Lam

T. C. M.

Valencia

(2019). Retrospective pretest and counterfactual self-report: Different or same? Journal of MultiDisciplinary Evaluation, 15(33), 37–53.

30.

Leeuw

F. L.

(2012). Linking theory-based evaluation and contribution analysis: Three problems and a few solutions. Evaluation, 18(3), 348–363.

31.

Leeuw

F. L.

Vaessen

(2009). Impact evaluation and development. NONIE and World Bank.

32.

Mackie

J. L.

(1965). Causes and conditions. American Philosophical Quarterly, 2(4), 245–264.

33.

Mahoney

(2008). Toward a unified theory of causality. Comparative Political Studies, 41(4/5), 412–436.

34.

Mahoney

Goertz

(2006). A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis, 14(3), 227–249.

35.

Mahoney

Kimball

Koivu

K. L.

(2009). The logic of historical explanation in the social sciences. Comparative Political Studies, 42(1), 114–146.

36.

Mark

M. M.

Henry

G. T.

Julnes

(1998). A realist theory of evaluation practice. New Directions for Evaluation, 78, 3–32.

37.

Mayne

(2008). Contribution analysis: An approach to exploring cause and effect (ILAC Brief No. 16). The Institutional Learning and Change Initiative.

38.

Mayne

(2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280.

39.

Mohr

L. B.

(1995). Impact analysis for program evaluation. Sage.

40.

Mohr

L. B.

(1999). The qualitative method of impact analysis. American Journal of Evaluation, 20(1), 69–84.

41.

Morgan

S. L.

Winship

(2015). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). Cambridge University Press.

42.

Mueller

C. E.

Gaus

Rech

(2014). The counterfactual self-estimation of program participants: Impact assessment without control groups or pretests. American Journal of Evaluation, 35(1), 8–25.

43.

Patton

M. Q.

(2015). Qualitative research and evaluation methods (4th ed.). Sage.

44.

Pattyn

Molenveld

Befani

(2019). Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation. American Journal of Evaluation, 40(1), 55–74.

45.

Pawson

Tilley

(1997). Realistic evaluation. Sage.

46.

Pratt

C. C.

McGuigan

W. M.

Katzev

A. R.

(2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349.

47.

Rathbun

B. C.

(2008). Interviewing and qualitative field methods: Pragmatism and practicalities. In Box-Steffensmeier

J. M.

Brady

H. E.

Collier

(Eds.), The Oxford handbook of political methodology (pp. 685–701). Oxford University Press.

48.

Reichardt

C. S.

(2006). The principle of parallelism in the design of studies to estimate treatment effects. Psychological Methods, 11(1), 1–18.

49.

Reichardt

C. S.

(2011). Evaluating methods for estimating program effects. American Journal of Evaluation, 32(2), 246–272.

50.

Reichardt

C. S.

(2019). Quasi-experimentation: A guide to design and analysis. Guilford Press.

51.

Ricks

Liu

A. H.

(2018). Process-tracing research designs: A practical guide. PS: Political Science and Politics, 51(4), 842–846.

52.

Rohlfing

Schneider

C. Q.

(2018). A unifying framework for causal analysis in set-theoretic multimethod research. Sociological Methods and Research, 47(1), 37–63.

53.

Roughley

Dart

(2009). Developing a performance story report: Users guide. Commonwealth of Australia.

54.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.

55.

Rubin

D. B.

(2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.

56.

Scriven

(1976). Maximizing the power of causal investigations: The modus operandi method. In Glass

G. V.

(Ed.), Evaluation studies review annual (Vol. 1, pp. 101–118). Sage.

57.

Scriven

(2005a). Can we infer causation from cross-sectional data? [Conference session]. A Symposium on the Use of School-Level Data for Evaluating Federal Education Programs, Board on Testing and Assessment, The National Academies Press, Washington, DC, United States.

58.

Scriven

(2005b). Causation. In Mathison

(Ed.), Encyclopedia of evaluation (pp. 44–48). Sage.

59.

Scriven

(2008). A summative evaluation of RCT methodology: An alternative approach to causal research. Journal of MultiDisciplinary Evaluation, 5(9), 11–24.

60.

Scriven

(2009). Demythologizing causation and evidence. In Donaldson

S. I.

Christie

C. A.

Mark

M. M.

(Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 134–152). Sage.

61.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Harcourt.

62.

Stern

Stame

Mayne

Forss

Davies

Befani

(2012). Broadening the range of designs and methods for impact evaluation: Report of a study commissioned by the Department of International Development (Working paper 38). Department of International Development.

63.

Tacq

(2011). Causality in qualitative and quantitative research. Quality and Quantity, 45(2), 263–291.

64.

Waldner

(2016). Invariant causal mechanisms. Qualitative and Multi-Method Research, 14 (1–2), 28–34.

65.

Weiss

C. H.

(1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 76, 41–55.

66.

West

S. G.

Thoemmes

(2010). Campbell’s and Rubin’s perspectives on causal inference. Psychological Methods, 15(1), 18–37.

67.

White

(2011). Theory-based impact evaluation: Principles and practice (3ie Working paper 3). International Initiative for Impact Evaluation.

68.

White

Phillips

(2012). Addressing attribution of cause and effect in small n impact evaluations: Towards an integrative framework (3ie Working paper 15). International Initiative for Impact Evaluation.

69.

Wilson-Grau

Britt

(2013). Outcome harvesting. Ford Foundation Middle East and North Africa Office.