Getting to Outcomes

Abstract

The Labin et al. logic model describes the why, how, what, and potential outcomes of evaluation capacity building (ECB). Getting To Outcomes offers a frame and empirical results for operationalizing the ECB logic model of Labin et al. and for deepening the science and practice of ECB.

Keywords

evaluation capacity building getting to outcomes empowerment evaluation accountability

In this article, I will briefly provide an example that illustrates the rationale, science, and practice of evaluation capacity building (ECB). Getting To Outcomes^® (GTO^®)¹ is a results-based approach to accountability. GTO involves asking and answering 10 accountability questions or “steps” (See Table1): assessing needs and resources (GTO Step 1); setting goals and desired outcomes (GTO Step 2); selecting evidence-based (or promising) practices, for example policies, programs (GTO Step 3); assessing fit (GTO Step 4); assessing individual/organizational/community capacity for an innovation (GTO Step 5); planning (GTO Step 6); implementation and process evaluation (GTO Step 7); outcome evaluation (GTO Step 8); continuous quality improvement (GTO Step 9); and sustainability (GTO Step 10).

Rationale

GTO was developed as a “how-to” for empowerment evaluation. Interest in empowerment evaluation arose out of the dissatisfaction that came with many evaluations showing a lack of outcomes. A major motivation for the development of empowerment evaluation was the rationale that the logic and tools of evaluation could be useful to key stakeholders of a program/initiative “up-front” (Fetterman & Wandersman, 2005). In other words, if key stakeholders including program staff had the capacity to use the knowledge and tools of evaluation to help them plan more systematically, implement with quality, self evaluate, and use the information for continuous quality improvement, then they would be more likely to achieve outcomes. This rationale is clear in the expanded definition of empowerment evaluation (Wandersman et al., 2005):

An evaluation approach that aims to increase the probability of achieving program success by: (1) providing program stakeholders with tools for assessing the planning, implementation, and self-evaluation of their program, and (2) mainstreaming evaluation as part of the planning and management of the program/organization. (2005, p. 26)

Although GTO can be used in more traditional evaluations (e.g., to retrospectively or concurrently assess accountability in an initiative), it was initially developed as a prospective “how-to” to build capacity to achieve outcomes.

Science

The science of ECB using GTO can be illustrated in several ways: logic models, strategy, measurement, and empirical results of using GTO.

Logic models and theories

There are many logic models and theories of ECB (e.g., Preskill & Boyle, 2008; Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013). Labin, Duffy, Meyers, Wandersman, and Lesesne (2012) presented an ECB logic model (on the How, Why, What, and Outcomes of ECB) to guide their research synthesis (Figure 1). I suggest that the Labin et al.’s logic model can be reorganized into a “hands-on” practical approach to building ECB using the 10 GTO steps to plan, implement, evaluate, continuously quality improve, and sustain an ECB initiative (Figure 2). GTO builds organizational capacity to be comprehensive and accountable.

Figure 1.

Integrative evaluation capacity-building model. *Collaborative and participatory aspects and processes should be included in defining and operationalizing nearly all elements of the model.

Figure 2.

Using GTO to operationalize the Why, What, How, and Outcomes of evaluation capacity-building.

A strategy for building ECB

Since 1999, my colleagues and I have been developing and implementing strategies to build capacity to use GTO to achieve outcomes (Wandersman, Imm, Chinman, & Kaftarian, 2000). We have developed tools (GTO manuals) in several content areas including substance abuse prevention (Chinman, Imm, & Wandersman, 2004), positive youth development (Fisher, Imm, Chinman, & Wandersman, 2006), preventing underage drinking (Imm et al., 2007), teen pregnancy prevention (Lesesne et al., 2007, 2012), and homeless veteran programs (Hannah, McCarthy, & Chinman, 2011). We found that providing manuals is necessary but not sufficient. We have added key support components of training and technical assistance to our approach (e.g., Chinman et al., 2008; Lesesne et al., 2008). More recently, we proposed an evidence-based approach to innovation support that includes tools, training, technical assistance, and quality assurance/quality improvement (Wandersman, Chien, & Katz, 2012).

Measurement

Chinman et al. (2008) have used measurement approaches based on the work of Hall and Hord (2010) to assess gains from using tools, training, and TA to build capacity to implement GTO at the individual and program level. For example, innovation configuration (IC) maps are a framework that can be tailored to evaluate the quality of use of any innovation (Hall & Hord, 2010). The GTO-IC Map has 14 items (called components) tied to the 10 steps of the GTO model. Each component has seven possible response choices that include the ideal performance of prevention practices targeted by GTO and six other possible variations ranging from “highly faithful to” to “highly divergent from” what is specified in the GTO model. Each component’s seven response choices are also accompanied by descriptions of observable behaviors. The steps of the GTO model (e.g., conducting outcome evaluation) are good prevention practices in general; thus, it is possible that the comparison programs may demonstrate varying levels of performance, making the GTO-IC Map applicable for assessment of both intervention and comparison programs.

Empirical results

Several studies have empirically studied the effects of using GTO. For example, Chinman et al. (2008) tested the effectiveness of using GTO to improve prevention capacities and prevention programming outcomes of community coalitions in a 2-year longitudinal trial funded by the Centers for Disease Control and Prevention (CDC). Results indicated significant improvements in staff capacity and program performance among programs using GTO versus comparison programs. A dose–response relationship trend between GTO and improved capacity to engage in prevention activities was observed. In other words, greater exposure to GTO was associated with greater gains in prevention capacity, at the individual and program levels. In addition, programs that used GTO documented greater outcomes.

In a formative evaluation, Chinman, Tremain, Imm, and Wandersman (2009) found that programs using a web-based application of GTO (iGTO) demonstrated an increase in the quality of performance of key prevention practices over non-iGTO programs.

Chinman et al. (2013) presented interim findings from a randomized trial testing Assets-Getting To Outcomes (AGTO), a 2-year intervention to build prevention practitioners’ capacity to implement positive youth development-oriented prevention practices in 12 prevention coalitions in Maine. The results show that opportunity to learn AGTO was not enough. Those who used the opportunities offered improved their prevention capacity, for example, improvement was associated with the number of technical assistance hours received.

Table 1.

The GTO 10 Steps.

Accountability questions	What is addressed in each GTO chapter	Relevant literature
Step #1: What are the underlying needs and conditions to address? (NEEDS)	Provides information about conducting a needs and resource assessment and links to additional resources	Needs assessment; resources assessment
Step #2: What are the goals, priority populations, and objectives (i.e., desired outcomes)? (GOALS)	Provides worksheets for defining priority populations and creating realistic and measurable goals and objectives	Goal setting
Step #3: Which science- (evidence-) based models and best practice programs can be useful in reaching the goals? (BEST PRACTICES)	Overviews evidence-based programming and what works in prevention across various domains (e.g., individual, family, peer, school, and community) and provides links to the evidence-based program literature	Science and best practices
Step #4: What actions need to be taken so the selected program fits the community context? (FIT)	Prompts readers to review the characteristics of existing programs and priority populations to reduce duplication and facilitate collaboration with other area programs	Collaboration; cultural competence
Step #5: What organizational capacities are needed to implement the program? (CAPACITY)	Prompts readers to assess several aspects of organizational capacity or the resources the organization possesses to direct and sustain a program	Capacity building
Step #6: What is the plan for this program? (PLANNING)	Presents information and worksheets for key planning elements such as an implementation timeline, assignments of responsibility, needed and available resources, and locations for activities	Planning
Step #7: How will the quality of program and/or initiative implementation be assessed? (PROCESS EVALUATION)	Provides information and several tools to assist practitioners in assessing which activities were implemented, the quality of the implementation, and the strengths and weaknesses of the implementation	Process evaluation
Step #8: How well did the program work? (OUTCOME EVALUATION)	Presents outcome evaluation and a basic framework for measurement; several evaluation designs; brief overviews of quantitative and qualitative methods; and topics including sample size, timing of assessments, informed consent, confidentiality and anonymity, data storage, and establishing benchmarks	Outcome and impact evaluation
Step #9: How will CQI strategies be incorporated? (CQI)	Prompts practitioners to reassess Questions 1–8 after completing the program to assess and feedback evaluation information about planning, implementation, and outcomes to improve the program	Total quality management; continuous quality improvement
Step #10: If the program is successful, how will it be sustained? (SUSTAIN)	Presents several factors that practitioners should consider when attempting to sustain an effective program: (a) “buy in,” (b) effectiveness, (c) diversity of funding, (d) staff training, (e) presence of a program champion, and (f) political capital of the program	Sustainability and institutionalization

Note. GTO = Getting To Outcomes.

Practice

GTO is being widely used. Several of the manuals (Chinman, Imm, & Wandersman, 2004; Imm et al., 2007) are available for free downloading from the RAND website http://www.rand.org/health/projects/getting-to-outcomes.html, and there have been over 80,000 downloads of the manuals. An example of the use of GTO is in a CDC-funded initiative on teen pregnancy prevention http://www.cdc.gov/TeenPregnancy/PreventTeenPreg.htm. GTO is being used to help community-based organizations plan, implement, evaluate, use continuous quality improvement, and sustain evidence-based programs.

Concluding Comment

There are many articles written about ECB, but the bottom line appears to be that the research and practice of ECB is very limited (Labin, Duffy, Meyers, Wandersman, & Lesesne , 2012). GTO offers a frame for operationalizing the ECB logic model of Labin et al. (2012) and for deepening the science and practice of ECB.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Chinman

Acosta

Ebener

Burkhart

Clifford

Corsello

M.,

… Tellet-Royce

(2013). Intervening with practitioners to improve the quality of prevention: One year findings from a randomized trial of assets-getting to outcomes. Journal of Primary Prevention, 34, 173–191.

Chinman

Hunter

S. B.

Ebener

Paddock

S. M.

Stillman

Imm

Wandersman

(2008). The getting to outcomes demonstration and evaluation: An illustration of the prevention support system. American Journal of Community Psychology, 41, 206–224.

Chinman

Imm

Wandersman

(2004). Getting to outcomes 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation. (No. TRTR101). Santa Monica, CA: RAND Corporation. Retrieved from http://www.rand.org/publications/TR/TR101/

Chinman

Tremain

Imm

Wandersman

(2009) Strengthening prevention performance using technology: A formative evaluation of interactive getting to outcomes. American Journal of Orthopsychiatry, 79, 469–481.

Fetterman

Wandersman

(2005) Empowerment evaluation: Principles in practice. New York, NY: Guilford Press.

Fisher

Imm

Chinman

Wandersman

(2006) Getting to outcomes with developmental assets. Minneapolis, MN: Search Institute.

Hall

Hord

(2010). Implementing change: Patterns, principles, and potholes. 3rd ed. Boston, MA: Pearson.

Hannah

McCarthy

Chinman

(2011). Getting to Outcomes in services for homeless Veterans: 10 steps for achieving accountability. Philadelphia, PA: National Center on Homelessness Among Veterans. Retrieved from http://www.va.gov/HOMELESS/NationalCenter_Additional_Information.asp

Imm

Chinman

Wandersman

Rosenbloom

Guckenburg

Leis

(2007). RAND, TR-403-SAMHSA. Santa Monica, CA: RAND Corporation. Retrieved from www.rand.org/pubs/technical_reports/TR403/ (available for free)

10.

Labin

Duffy

Meyers

Wandersman

Lesesne

(2012) A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.

11.

Lesesne

C. A.

Lewis

K. M.

Moore

Fisher

Green

Wandersman

(2007, 2012). Promoting science-based approaches to teen pregnancy prevention using getting to outcomes: Draft 2007. Atlanta, GA: Centers for Disease Control and Prevention.

12.

Lesesne

C. A.

Lewis

K. M.

White

C.P.

Green

Duffy

Wandersman

(2008). Promoting science-based approaches to teen pregnancy prevention: Proactively engaging the three systems of the interactive systems framework. American Journal of Community Psychology, 41, 379–392.

13.

Preskill

Boyle

(2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443–459.

14.

Taylor-Ritzler

Suarez-Balcazar

Garcia-Iriarte

Henry

D. B.

Balcazar

F. E.

(2013). Understanding and measuring evaluation capacity: A model and instrument validation. American Journal of Evaluation, 34, 190–206.

15.

Wandersman

Chien

Katz

(2012). Toward an evidence–based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50, 445–460.

16.

Wandersman

Imm

Chinman

Kaftarian

(2000). Getting to outcomes: A results-based approach to accountability. Evaluation and program planning, 23, 389–395.

17.

Wandersman

Snell-Johns

Lentz

Fetterman

Keener

Livet

… Flaspohler

(2005) The principles of empowerment evaluation. In Fetterman

Wandersman

(Eds.), Empowerment evaluation principles in practice (pp. 27–41). New York, NY: Guilford Press.