Evidence-Based Interventions Are Necessary but Not Sufficient for Achieving Outcomes in Each Setting in a Complex World

EBIs

Traditionally, an intervention is considered evidence based if it has been proven effective at reaching positive outcomes through rigorous evaluations often including randomized controlled clinical trials. Several federal funding agencies require the use of EBIs for implementation and replication in large-scale initiatives (e.g., Health Resources and Services Administration Home Visiting Program, http://mchb.hrsa.gov/programs/homevisiting/grants.html; Office of Adolescent Health (OAH) Teen Pregnancy Prevention Program, https://wayback.archive-it.org/3909/20140324181922/http://www.hhs.gov/ash/oah/grants/assets/foa_tpp_tier_2.pdf). Currently, the OAH 2015–2020 grantee program funded 75 Teen Pregnancy Prevention Replication grantees (US$400,000–4 million per year for 5 years for each grant http://www.hhs.gov/ash/oah/grants/closed-grants.html/). The grants required use of EBIs from a list of evidence-based teen pregnancy prevention programs. The rationale for using EBIs is that since they have been shown to lead to positive outcomes in research studies (our scientific journals are filled with such studies), the logical expectation is that their widespread implementation with fidelity will lead to large-scale improvement of outcomes. However, evaluations of multisite interventions implementing EBIs often indicate that programs do not reach outcomes when compared with comparison groups (e.g., multimillion dollar impact evaluation of a randomized control trial of Headstart vs. non-Headstart children (http://www.acf.hhs.gov/sites/default/files/opre/executive_summary_final.pdf). The Nurse–Family Partnership program, which was the subject of three randomized control trials in the United States that showed outcomes, did not achieve a significant difference in outcomes versus a comparison group in the United Kingdom (Robling et al., 2016). Some of the outcomes were worse in the intervention condition, and the recommendation was that the trial be halted. A possible reason for the finding was that the usual care in the United Kingdom is different than the usual care in the United States. This variability in outcomes demonstrates that we live in a complex world where context matters, and even the most carefully studied interventions may not exhibit external validity at scale (Green, 2001; Leviton & Trujillo, 2016). Even in programs that do achieve significant differences in outcomes versus comparison sites on a large scale, such as community partnership models like Communities that Care (Hawkins et al., 2012) and PROSPER (Spoth, Guyll, Lillehoj, Redmond, & Greenberg, 2007), they often do so only on the average. For example, Monahan, Hawkins, and Abbott (2013) performed a meta-analysis that demonstrated considerable variability on delinquency outcomes across Communities that Care sites. In sum, many large-scale evaluations (e.g., Headstart) do not show outcomes, many programs that have had successful outcomes in a randomized control trial do not necessarily work in new contexts (e.g., Nurse–Family Partnership), and in multisite studies that have shown outcomes versus a comparison group, there is considerable variation among the intervention sites (e.g., Communities that Care). While statistically significant differences may be adequate for determining the overall level of success or failure of a program for policy makers, at the “on the ground” level, each program implementer should have an interest in achieving outcomes because he or she is accountable to the population served by their own program.

Accountability

Accountability is a widely used term, often attributing responsibility and sometimes blame. In results-based accountability (RBA), Friedman (1997) describes performance accountability and focuses on whether customers are better off as a result of services delivered. The performance measures also look at the quality and efficiency of the services. The RBA asks three simple questions: “How much did we do?” “How well did we do it?” “Is anyone better off?” (http://resultsaccountability.com/about/what-is-results-based-accountability/).

Building on Friedman’s work, we think that a proactive approach to accountability involves being strategic and results-oriented with limited time, energy, and money. Later, we describe the approach to accountability to help achieve outcomes in a specific setting (e.g., a specific school, hospital, or community). First, we illustrate the challenge of implementing EBIs at scale. Second, we present empowerment evaluation (EE) and specifically Getting To Outcomes^® (GTO) ¹ as an approach that can support effective implementation of EBIs in practice to achieve outcomes in specific settings. Third, we describe how GTO can help implement complex interventions in dynamic, real-world contexts and present examples of how it may be used to support quality implementation of a multisystem-level intervention to improve health outcomes. We conclude with implications for stakeholders interested in using EE and GTO to support context-appropriate quality implementation of EBIs in a specific setting.

Challenges of Implementing EBIs at Scale

Although we agree that EBIs are an important ingredient for achieving outcomes (e.g., Wandersman, Imm, Chinman, & Kaftarian, 2000), many evaluation studies raise questions about the generalizability of achieving outcomes using EBIs, noting that replicating an EBI alone is not sufficient for large-scale program success. In this section, we use the example of the global experience with the surgical safety checklist to illustrate: Why it is important to be strategic and accountable in including context while planning and implementing an intervention, why it is important to look at the quality of implementation, and why it is important to evaluate an intervention in each setting in which it is implemented.

Death and complications from surgery are a major problem internationally. Surgical checklists were introduced and evaluated as a means of reducing complications and mortality as part of the “Safe Surgery Saves Lives” initiative of the World Health Organization. They were piloted globally in eight hospitals from a variety of settings (from U.S. university hospitals to hospitals in developing countries). Haynes, Weiser, Berry et al. (2011) analyzed surgical deaths and complications before and after the introduction of the checklist in the pilot sites and demonstrated significant reductions in mortality and complications. Based on this evidence, the World Health Organization officially launched the surgical checklist in 2008 (World Health Alliance for Patient Safety, 2008) and recommended global adoption and use.

As illustrated in a study by Urbach, Govindarajan, Saskin, Wilton, and Baxter (2014), problems arise when going to scale. A policy in Ontario, Canada, encouraged the use of the surgical checklist in all of its hospitals. The authors analyzed outcomes data in 101 hospitals that endorsed the policy—before and after the introduction of the checklist. They concluded that the use of surgical checklists was not associated with significant reductions in mortality or complications. Despite a high degree of reported compliance in using the checklist, no hospitals in the study had a significant reduction in operative mortality, and the evidence on surgical complications was mixed, with six hospitals reporting significantly positive outcomes and three reporting significantly negative ones. This is not due to the lack of use of the checklist (98% of the hospitals self-reported compliance).

Because the study did not collect data on how the checklists were implemented and on the reasons for success and failure, it is impossible to attribute a cause to the overall findings. Variability in the types of hospitals and in the volume of procedures (at the low end, one of the hospitals performed only two procedures in the 3 months following the implementation of the checklist) could have contributed variations in implementation context that affected the results. Other checklist studies (Averling, McCuloch, & Dixon-Woods, 2013; Haynes et al., 2009) mention the Hawthorne effect, differences in resources, hierarchical relationships, physician indifference, and skepticism as possible facilitators and barriers to the achievement of outcomes. In a personal communication (April 26, 2015), Dixon-Woods hypothesized that there may have been a ceiling effect in the Ontario study; many of the hospitals were already performing well, and there might have been little need for increased changes in outcomes resulting from the checklist implementation (which might account for the significant number of hospitals where the mean effect was 0).

The experience with the surgical safety checklist is not unique. For example, when grantees in the OAH program (2010–1015) attempted to replicate evidence-based teen pregnancy prevention programs in numerous settings, some of the sites reported improved outcomes, while several others did not (Farb, Margolis, & Gosling, 2015). In summary, the Ontario and OAH studies show that it is challenging to evaluate the effectiveness of multisite EBIs without understanding the particular factors affecting local implementation. The natural corollary of this observation is that the design and implementation of multisite EBIs needs to be designed to maximize the possibility of achieving outcomes at each site; the design must attend to particular factors that affect the fidelity of the implementation and the fit of intervention to the local context.

From a practice perspective, the question of whether EBIs are effective at scale becomes irrelevant for practitioners accountable for outcomes because the important question is whether they are doing what they need to do in their own setting in order to be successful. Returning to the checklist example, in a hospital where poor implementation fidelity is the cause of a lack of outcomes, more team training could be offered; in a hospital which is already at a generally high level of quality on an outcome, a decision may be made to require the checklist only in departments where improved performance is needed. Therefore, the relevant evaluation question to be answered at scale is not “does the EBI result in outcomes?” but rather “how do we achieve outcomes in each setting?”

This approach is echoed in an Institute of Medicine report, described in a Journal of the American Medical Association article (Pincus & England, 2015), which provided guidelines for better quality of implementation of psychosocial interventions. The report recommends the development of quality metrics that include structure and process in addition to outcomes and further research into the development of methods for implementing interventions in a way that leads to successful outcomes.

So where does this leave us? Typically, most funders and all organization administrators want to achieve outcomes in a specific setting (e.g., in their own school, in their own hospital, in their own community). The reality is that each organization is complex, faces complicated issues in its own unique context, and the quality of implementation is influenced by this context (Patton, 2011). If you think (as we do) that the examples above are just the tip of the iceberg regarding the overall effectiveness of EBIs at scale in a complex world, then a more strategic approach is needed to achieve outcomes, and evaluators should be an integral part of this approach. We suggest that what is needed are approaches in which the evaluator assists program implementers to adapt and tailor programs to meet local needs and provides ongoing feedback to support program implementation. There has been increasing interest in such approaches with a focus on the use of formative or learning methods (evaluation intended to furnish information to guide program improvement) versus more traditional or summative evaluation approaches (evaluation intended to render a summary judgment on certain critical aspects of the program’s performance) (Rossi, Lipsey, & Freeman, 2004).

A variety of formative evaluation approaches have been proposed; most emphasize the importance of collaboration with stakeholders and the use of an evaluation approach that is flexible and can adapt to the changing needs of local contexts (e.g., developmental evaluation, Patton, 2011; FORECAST, Katz et al., 2013; learning evaluation, Balasubramanian et al., 2015). We focus on the method of EE, which puts the logic and tools of evaluation into the hands of practitioners and consumers to equip them to address challenges related to program planning, implementation, and evaluation in their setting. We will show how the GTO approach to EE provides a comprehensive planning, implementation, and evaluation strategy that, if properly implemented, helps practitioners achieve a results-based approach to accountability in their own-specific settings.

Applying GTO to Planning, Implementing, and Evaluating a Complex Multilevel Intervention: The “Improving Sickle Cell Care in Adolescents and Adults in Chicago” (ISSAC) Case Example

To illustrate how GTO can be used to address a complex problem that requires multilevel solutions, we present a case example of an actual response to a call for proposals from the National Institute of Health’s National Heart Lung and Blood Institute for a multisystem approach to helping people diagnosed with sickle cell disease (SCD) in Chicago. The ISAAC program is designed to address the many barriers to care and to improve outcomes for adolescents and adults with SCD. The overall goal of ISAAC is to systematically develop, test, and implement multilevel, multimodal interventions that address barriers to evidence-based care (Patient-Centered Outcomes Research Institute [PCORI], 2016) and take into account the specific needs and resources in the Chicago area.

SCD is the most common genetic blood disorder in the United States, affecting over 90,000 people, the majority of whom are African American. Each year, SCD patients nationally have over 190,000 emergency department (ED) visits and 110,000 hospital admissions (Centers for Disease Control, 2015). Severe pain and other complications among SCD patients lead to frequent medical absences from school or work and high rates of unemployment and disability. Despite seven academic medical centers with predominantly pediatric sickle cell centers, there is an access disparity for youth and adults with SCD in Chicago, resulting in high ED and hospitalization rates.

Despite the growing availability of evidence-based approaches for treating SCD (National Heart, Lung, & Blood Institute, 2014), a number of critical barriers inhibit access to adequate health care and exacerbate negative outcomes for adolescents and adults suffering from SCD. In particular, there is strong evidence for the benefit of hydroxyurea, the only disease-modifying medication approved by the Food and Drug Administration for SCD in adults and in children. Adherence to hydroxyurea therapy has been shown to reduce pain crises and acute chest syndrome, decrease the need for blood transfusions, and improve survival. Although its beneficial effects are well documented, this medication is underprescribed by health-care workers and poorly adhered to by patients. In addition to the inadequate use of hydroxyurea, research has shown that SCD patients receive inadequate longitudinal regular ambulatory care and inadequate treatment of acute pain episodes (Ware, 2010).

Therefore, implementation of a successful program for improving SCD outcomes has to be a multilevel process involving communities, patients and their support systems, health system administrators, and health-care providers. Each of these stakeholder groups needs to understand the barriers to achieving outcomes from its own perspective and context, and this knowledge will help to design the implementation strategy for ISAAC that anticipates implementation issues rather than reacting to them. Table 1 shows how GTO thinking at various levels was used to plan the ISAAC intervention for the proposal.

OPEN IN VIEWER

Table 1.

Application of GTO to ISAAC Planning.

GTO Step	Accountability Question	Community = Cook County Schools, Jobs, and Policies	Hospital Health-Care System	Providers: Sickle Specialist, Primary Care Providers, and ED Providers	Patient and Household
1	Needs/resources	Limited community understanding of SCD and its impact appropriate opioid use versus drug abuse appropriate use of medical transportation appropriate absences from school or job Ambulance transportation or nonemergency transportation CHW/PN pool exists in Chicago, trained for other health conditions Disability policies (FMLA, 504)	Balance service and costs Insurance pays for care	Provide “best practices” care for a rare disease Reduce fears of narcotic diversion/abuse Avoid redundant/conflicting care Avoid burnout	Mr. Smith has episodes of severe pain and wants the best treatment available Early effective pain treatment has a better chance of shortening the pain episode Emotional stress increases the pain He cannot drive while in severe pain, so he wants direct access to transportation to medical care He cannot fulfill childcare duties while in severe pain and in hospital He would like his workplace to accept appropriate absences from work for medical reasons Sometimes he needs IV pain treatment on 2 or more consecutive days but wants to sleep at home
2	Goals	Increase community understanding of SCD pain Utilize resources to support patient during treatment for SCD pain Increase retention in employment and school	Optimal use of health service resources	Maximize adherence to NHLBI guidelines Maximize use of Acute Care Center/Day Hospital for SCD pain treatment Avoid fragmented care among providers and hospitals	Maximize function in normal life activities Reduce days of pain Reduce nights in hospital Maximize convenience of care Personalized consistent treatment Not be suspected of drug abuse or malingering or frivolous use of med transportation
3	Best practice	Peers/CHW/PN coach about arranging for medical appointments and acute care Ambulance or nonemergency transport to med care Disability accommodations (FMLA, 504) personalized to maximize job and school success	Acute Care Center/Day Hospital—reduces hospitalizations, decompresses ED, saves money Complex case management team optimal multidisciplinary team including psychosocial care	Acute Care Center/Day Hospital (expert, personalized, compassionate, rapid, allows repeat visits) Acute care per NHLBI 2014 guidelines Provide timely medication refills Ready access to specialist and familiar provider Coordinated care via CHW ISAAC plan in electronic medical record facilitates communication of plans	Go to specialized Acute Care Center/Day Hospital Avoid triggers for SCD pain
4	Fit	Flexible access to transportation for medical appointments and acute care CHW/PN has proven useful for underserved minorities. Disability accommodations: Support to use FMLA and work from home Intermittent SCD awareness for school (EBP) and job site (by extension)	Services covered by appropriate insurance Minimize 30-day readmissions because of penalties ISAAC pain passport works around electronic records that lack interconnection.	ISAAC treatment plans can convey expertise and delegate care to other sites and providers: (1) ISAAC pain passport for acute care (2) chronic care chart for longitudinal care plan (3) Hydroxyurea algorithm (4) chronic pain treatment plan (5) transportation plans (6) job/school accommodations Convene with care coordination team about plans and changes Speedy access to acute and chronic records from other sites	Approach is in line with Mr. Smith’s values and lifestyle needs Reduce unpredictable medical absences for scheduled absences Maximize convenience of care Ambulance or nonemergency transportation
5	Capacities	Social worker knows disability resources but only very part-time Training curriculum for CHW in SCD has already trained 8 CHW/PN Ambulance transportation and nonemergency transportation have rules	Insurance can authorize coverage for innovations. Business analysts & quality of care analysts collect metrics CAPriCORN regional data sharing	Generate ISAAC treatment plans —general template & personalization Need to distribute treatment plans using Electronic Health Record or work-around Need to hire & train & regularly meet with care coordination team: APN, CHW, PN	Needs help arranging transportation/child care assistance Needs paperwork to have absences excused from work Acute pain is unpredictable, often weekend when Acute Care Center is closed Ambulance or nonemergency transportation has rules
6	Plan	ISAAC plan taps community resources/support appropriate to get to medical care ISAAC plan asks ambulance transport to bring patient to the preferred ED ISAAC plan asks nonemergency transport can bring patient to Acute Care Center ISAAC plan has template FMLA & 504 to personalize for job & school success. Resource lists for CHW	Get sickle cell treatment plans into the Electronic Health Record Collect metrics, pool metrics regionally	Develop ISAAC treatment plans for each patient—general template & personalization Disseminate ISAAC plans using Electronic Health Record or work-around Maintain trained and compassionate staff in the Acute Care Center Close communication with care coordination team: APN, CHW, PN Develop awareness of sickle cell resources and treatment plans	Update ISAAC treatment plan at medical appointments Provide/carry ISAAC plan at visits to ED Use UIC Acute Care Center as much as possible transportation allows Go to consistent ED Develop plans for transportation and excused absences from work before he is in pain
7	Process	CHW/PN teach disability rights CHW/PN provide ISAAC template for FMLA & 504 to personalize CHW/PN provide ISAAC template for transportation to personalize	Get ISAAC treatment plans into the Electronic Health Record Collect metrics, pool metrics regionally	Update treatment plans for at each appointment, plus regular contact with care coordination team Identify “outsiders” patients who only get acute care and send CHW/PN to follow up on them Disseminate ISAAC plans using Electronic Health Record, letters to PCP, or work around Quarterly newsletter for PCP from SCD team APN communicates with CHW team daily, weekly with SCDAI and HAH Sickle specialists meet weekly with APN and care coordination team Monthly meeting with community groups	Attend medical appointments to update ISAAC plan for pain treatment regularly (3–6 months) Work with CHW/PN to attend medical appointments to update ISAAC plan, regularly Work with CHW/PN to plan transportation/childcare when he needs Acute Care Center or ED Work with CHW/PN to personalize FMLA or 504 template for job or school Midcourse corrections to plan are made if needed Close communication between patient, CHW, sickle team, PCP, job/school
8	Outcome evaluation	All people with SCD have good transportation to Acute Care All people have appropriate job or school accommodation plan ISAAC plan updated regularly (6–12 months)	All people with SCD are in regional registry and linked with CHW/PN	Percentage of eligible patients on hydroxyurea Percentage no shows for appointments Percentage with ISAAC plan, especially pain passport Increase the number of patients with at least 1 visit/year	Mr. Smith gets prompt individualized pain control with ISAAC plan He keeps up with longitudinal health care to update ISAAC plan PROMIS Scale shows that absences from work and family activities are minimal Reduce nights in the hospital Reduce events when transportation or job blocked access to Acute Care Center pain treatment
9	Improve	PDSA led by CHW and PN Close communication between patient, CHW, sickle team, PCP, job/school More CHW get training on SCD and ISAAC	PDSA	PDSA	Mr. Smith goes to Acute Care Center more than ED He agrees to attend at least 75% of medical visits He communicates with care team about acute needs or appointment cancellations
10	Sustain	Funding for APN/CHW/PN	Funding for APN/CHW/PN	Funding for high role in complex disease management Keep ISAAC plans updated Keep steady CHW/PN team	Plan for 4 months and CHW/PN supports development/update of ISAAC plan

Note. SCD = sickle cell disease; ED = emergency department; CHW/PN = community health worker/patient navigator; FMLA = Family and Medical Leave Act; IV = intravenous; NHLBI = National Heart, Lung, and Blood Institute; ISAAC = Improving Sickle Cell Care in Adolescents and Adults in Chicago; EBP = evidence-based practice; PCP = primary care provider. CAPriCORN = Chicago Area Patient Centered Outcomes Research Network; APN = advanced practice nurse, nurse practitioner; UIC = University of Illinois at Chicago; SCDAI = Sickle Cell Disease Association of Illinois—community-based group; HAH = Have a Heart for Sickle Cell Anemia Foundation—community-based group; PROMIS = patient-reported outcomes measurement information system; PDSA= Plan Do Study Act.

Table 1 shows how needs and resources needed for the implementation of ISAAC vary across each level of the system through which care is provided. Needs at the community level include the requirement for better understanding of SCD and its impact, so that support for appropriate medication use or transportation or accommodations at work can be provided by the appropriate community agency (Smith, Jordan, & Hassell, 2011). Needs at the provider level require improved care coordination between providers involved in SCD care to prevent redundant and/or conflicting care (Kanter & Kruse-Jarres, 2013). At the health system level, protocols must be developed for managing a variety of customized treatment plans at various service and cost levels. These plans depend upon the level of care coordination provided by the physicians, the extent to which community support is available, and the needs of the patient. Our GTO-based approach identified these needs and their interdependencies at each level, which in turn led to the setting of interrelated goals at each level in Step 2. The entries in Table 1 were developed through an iterative process of discussion with the University of Illinois at Chicago team and were refined over time. For example, the identification of best practices was guided by understanding fit (Step 4) and capacities (Step 5). Changing the insurance policy to make sure patients are able to receive care from primary care and specialist physicians (e.g., insurance “carve out” or individual exemption) was initially identified as a best practice for improving the hospital health-care system. However, the stakeholders had doubts about their capacity to implement such a change, and the appropriate use of an acute care center and complex case management team was selected as evidence-based changes that fit the context better.

Moving forward into action (Steps 6–10), the table provides a guide for how each level of the system can contribute to their accountability for the achievement of outcomes. The SCD team perceived this approach as more structured, and with a greater emphasis on stakeholder alignment, compared to previous improvement science efforts in sickle cell (Oyeku et al., 2012).

The example above helps operationalize a conceptual model for research to reduce health disparities. Under a contract with the Agency for Healthcare Research and Quality to provide assistance in improving its approach to reducing disparities through its research and translation programs, Chin and Goldmann (2011) outlined a multilevel conceptual model with six levels of influence: policy, health-care delivery entities, microsystems, clinicians, patients/families, and community. The SCD example operationalize the policy level (developing a network of centers to develop and research procedures that reduce the negative outcomes of SCD) and the health-care delivery entities, clinicians, patients/family, and community levels. The detail provided in this example may be helpful in advancing the conceptual model for research on reducing health disparities.

Finally, the multilevel approach described above can be applied to content areas other than health such as education. For example, if we were to apply Table 1 in an educational context such as implementing programs for special needs children, we can change the last four columns of Table 1 to school district, school, teacher, and student (Maras, Wandersman, Splett, Flaspohler, & Weist, 2015). It is important to note that while the logic of EE and of GTO are straightforward, like implementation of most new programs or processes, there is a considerable amount of work to be done to get organizations to use it with quality. Those involved in developing GTO have developed an extensive support system to help users implement it with quality. The support model follows an evidence-based system for innovation support (Wandersman, Chien, & Katz, 2012) that includes tools (several manuals are available for free download on the RAND website), training, technical assistance, and quality assurance/quality improvement.

Conclusion

We make a point (as do many others) that knowledge of EBIs alone is insufficient for reaching outcomes at scale in a complex world. We describe the following: Why it is important to look at the particular context of each setting for implementation and why it is important for those accountable for implementation to be strategic in planning for implementation to ensure that the EBI achieves outcomes for their setting. The term accountability has become a buzzword for those seeking to improve outcomes. There is an assumption that if service providers and others are held “accountable” for achieving results by external stakeholders (e.g., external evaluators, funders, etc.), outcomes will presumably be obtained. But given that practitioners and funders have finite availability of time, money, and resources, the mandate of accountability alone without a proactive and effective strategy to achieve results usually fails. A systematic implementation method, such as GTO, proactively fosters a sense of personal commitment and ownership by providing support and tools to enable those responsible for results at multiple implementation levels to be truly accountable.

GTO may also be used by funders to incorporate quality implementation of EBIs in grant proposals. To demonstrate, we expand on the example of the OAH described earlier. The OAH in a 2015–2020 Teen Pregnancy Prevention grant program has taken lessons learned from the results (and lack of results) of its previous 2010–2015 grantee program to heart. While still requiring the use of EBIs from a federal list of evidence-based programs for teen pregnancy prevention, it is also requiring that each grantee (which receives between US$500,000–2 million per year for 5 years) to use GTO to guide implementation. In addition, OAH funded another eight grantees to provide capacity building assistance to organizations on implementing EBIs using GTO; these grantees receive US$400,000–750,000 per year for 5 years. Use of GTO is designed to help assure that the interventions are aligned with the context, needs, and capacities that each grantee faces; that the interventions should be well planned, well implemented, and evaluated; and that continuous quality improvement should take place.

The GTO approach to accountability—where accountability for achieving outcomes is considered during the initial planning and where stakeholders use evaluation strategies to guide implementation—presents new opportunities for evaluators. In an EE approach to accountability, the role of evaluator includes the building of local capacity and provision of support to stakeholders at all levels to ask and answer the accountability questions. Resources for evaluators interested in using an EE approach and specifically interested in using GTO include texts (Fetterman, Kaftarian, & Wandersman, 2015) and GTO “how-to” free manuals (see http://www.rand.org/health/projects/getting-tooutcomes.html).

In summary, our primary purpose in this article is to help increase the probability of achieving outcomes at any level of stakeholder accountability—whether it be each individual, each program within an organization (e.g., each program in a Boys and Girls Club), each organization (e.g., the East Spartanburg Boys and Girls Club), each community (e.g., East Spartanburg), each county (e.g., Spartanburg County), each state (e.g., South Carolina), and each country (e.g., the United States). We propose that EE, using GTO as an example, provides important tools and resources for supporting effective planning, implementation, and evaluation for using EBIs in each setting.