Using the Community Readiness Model as an Approach to Formative Evaluation

Abstract

This article presents an adapted version of an established model for assessing community readiness along with an illustrative case example from the evaluation of Positive Action, a school-based social and character development intervention, implemented as part of a randomized controlled trial in Chicago Public Schools from 2004 through 2010. Community readiness is an emerging assessment approach that can be used to gauge the level of understanding, desire, and ownership that community members have regarding a community problem and/or intervention. This approach is useful in engaging the community and leveraging particular aspects of readiness that the community may exhibit in order to maximize an intervention’s successful implementation. The article concludes with a discussion of ways in which a community readiness model may be useful in health promotion practice, both in schools and in other community settings.

Keywords

formative evaluation community readiness school-based programs

Introduction

In order to improve program design and delivery, formative evaluation strategies—an examination of program delivery, implementation procedures, and organizational context—are essential. One of the many implementation problems that can thwart program effectiveness is a lack of readiness for programming among a priority population’s members (Donnermeyer, Plested, Edwards, Oetting, & Littlethunder, 1997). If program implementation occurs before the community agrees there is a problem, or if a community is engaged with other priorities and is not ready to address a new problem, then implementation problems may arise (Donnermeyer et al., 1997). Efforts to conduct an assessment of a community’s readiness to engage in programming appear only sporadically in the literature, yet gauging a community’s readiness can lend insight into how programs function at the community level and what further program support may be needed to ensure successful implementation. This article describes an existing community readiness model (CRM) and its adaptation to assess readiness in school communities to implement a social-emotional behavioral health program, Positive Action (PA). The article concludes with a discussion of implications for the adapted model’s use both in schools and in other community-based health promotion settings.

Community Readiness Model

Researchers at the Tri-Ethnic Center at Colorado State University developed the CRM (Donnermeyer et al., 1997). The model’s development emerged from a need to better understand how individual community members’ awareness of a problem relates to their engagement in prevention programs. The developers defined community readiness as “the relative level of acceptance of a program, action or other form of decision-making activity that is locality-based” (Donnermeyer et al., 1997, p. 68). They sought a method for measuring levels of community readiness in order to better understand how a community’s readiness contributes to program implementation (Donnermeyer et al., 1997).

In creating the CRM, the developers looked to existing theories describing how individuals or communities ultimately adopt a program or intervention, including the transtheoretical model (Prochaska & DiClemente, 1992), the social action process (Beal, 1965), and the diffusion of innovations model (Rogers, 1994). They concluded that none of these models acknowledge that readiness may consist of multiple dimensions (Donnermeyer et al., 1997; Oetting et al., 1995). They developed a multidimensional model that allows for the possibility that a community’s readiness on one dimension, such as knowledge about the problem, may be at a different “stage” of readiness than on another dimension, such as leveraging resources to address the problem (Edwards, Jumper-Thurman, Plested, Oetting, & Swanson, 2000). The six dimensions of the final CRM include the following: (1) the community’s knowledge of the problem, (2) current efforts aimed at addressing the problem, (3) the community’s knowledge of current efforts, (4) leadership taken in these efforts, (5) community climate, and (6) resources that have been put toward the efforts (Edwards et al., 2000).

According to the CRM, a community may currently be measured at one of nine readiness stages for each of the six dimensions. These range from a lack of awareness of the problem to a high level of ownership over the efforts aimed to address the problem. The nine stages are labeled as (1) no awareness, (2) denial/resistance, (3) vague awareness, (4) preplanning, (5) preparation, (6) initiation, (7) stabilization, (8) confirmation/expansion, and (9) community ownership (Edwards et al., 2000).

After a priority problem and a priority population are identified, the next step in the CRM process is conducting key respondent interviews and scoring them to determine readiness stages. CRM developers contend that as few as three to four key informants are sufficient to adequately assess readiness. Once readiness stages across the multiple dimensions are identified, the assessors can work with the community to develop appropriate program implementation strategies consistent with readiness stages (Edwards et al., 2000).

Previous applications of the CRM have been used to assess a community’s readiness around needs that are sometimes not readily acknowledged as problems, such as substance abuse or AIDS/HIV (Oetting et al., 1995; Thurman, Vernon, & Plested, 2007). Results have informed decisions to focus initial efforts on raising community awareness rather than immediate program implementation (Oetting et al., 1995; Thurman et al., 2007). Evidence suggests that communities utilizing CRM experience higher levels of implementation success and stakeholder involvement compared with those not utilizing such an approach (Thurman et al., 2007). The work described in this article is the first documented use of this model in a school community, defined as the school’s students, teachers, other staff members, and administrators.

Using Community Readiness to Inform School-Based Program Implementation

Overview

PA is a school-based social-emotional behavioral health program that aims to address student self-concept, social skills, risk behavior, academic achievement, and school climate (Flay & Allred, 2010; see www.positiveaction.net). Developed in 1977, the program is based on self-concept theory, which states that self-concept is determined more by actions than by thoughts or feelings and that healthy choices therefore result in a positive sense of self-worth (Purkey & Novak, 1996). PA program components include a K–12 classroom curriculum and resources to support implementation and create a school culture supportive of PA. These include (1) teacher training in curriculum implementation, (2) schoolwide climate development materials designed to encourage students and staff members to take positive actions throughout the school day, (3) manuals for the principal and the designated PA coordinator to guide them in planning PA assemblies and other schoolwide activities, (4) school counselor tools that can be used with individual students or with student support groups, and (5) family materials that can be used at home or during in-school parent activities.

Beginning in September 2004, PA was implemented as a randomized trial in the Chicago Public Schools.¹ The program was implemented in seven schools that were randomly selected within each of seven matched pairs of elementary schools. Schools were selected from the population of all Chicago elementary schools, although in order to ensure the selection of high-risk schools (in accordance with program priorities), schools were excluded if more than 50% of students in the school passed the Illinois State Achievement Test or if fewer than 50% of students received free lunch. Data collection began in 2004 when the program was first implemented in the intervention schools. A cohort of students, in third grade during the first year of implementation, was surveyed over a 3-year period along with their teachers and parents.

The first phase of funding ended after the 2006-2007 academic year and the second phase of funding was not available until the 2008-2009 academic year, leaving a gap year (2007-2008) in programming. Although positive effects of PA were reported on a substantial number of measures (Li et al., 2011), evaluators noted variability among intervention schools in levels of program implementation. Before the second phase of funding began in 2008, evaluators anticipated that program schools would vary in their levels of reengagement with the program based on gap year contextual factors such as school staff attrition or changes in other school resource levels. CRM was adopted to assist in reestablishing program implementation efforts after the gap year by informing individualized strategies to support schools based on varying readiness levels.

Model Adaptation

PA evaluators adopted the process recommended by CRM developers and conducted semistructured interviews with key informants at each of the seven intervention schools in the fall of 2008, 2009, and 2010. In 2008 and 2009, three to four key informants were selected at each school, and in the final year, five to six informants were selected. Each year, the research team invited the principal (or other administrator) and any school personnel that were likely to be knowledgeable about PA implementation (e.g., the PA coordinator) to participate in interviews. In 2008 and 2009, at least one of the current cohort year teachers (seventh grade and eighth grade, respectively) was also included. In 2010, because most cohort students had moved on to high schools as ninth graders, the focus shifted to understanding schoolwide implementation, and therefore one randomly selected teacher from a sample of grades was selected. The interviews took approximately 30 minutes and were conducted in person at each school. The interviewers took handwritten notes and typed them afterward.

Questions included in the interview guide were designed to assess readiness on each of six dimensions and were adapted from the original readiness model. For example, the question “How would you describe your school’s climate for implementing PA?” was asked in order to assess school culture/climate, and “What programs or policies does your school use to promote students’ social, emotional, and behavioral health?” was asked to assess knowledge of existing school efforts, including PA. Table 1 shows the original six dimensions and those adapted to apply to the implementation of PA in the seven intervention schools. The only deviation from the model’s original language describing the stages was the change in sixth-stage language from “initiation” to “implementation” in order to better connote that moving from preparing for PA implementation (Stage 5) to attaining stable, ongoing implementation over multiple years (Stage 7) may take more effort over a longer period of time than the word initiation implies.

Table 1

Original Readiness Dimensions and Corresponding Positive Action (PA) Adaptations

Original Six Dimensions of Community Readiness	PA Readiness Assessment Adaptation
Efforts	School efforts: To what extent are there existing efforts, programs, and policies in the school that address social/emotional/behavioral (SEB) issues?	Each of the six dimensions at left is rated at one of the following nine readiness stages:
Community knowledge of efforts	School knowledge of efforts: To what extent do school personnel and administrators know about their school’s current efforts to address SEB issues and their effectiveness, and are the efforts accessible to all segments of the school?	1. No awareness 2. Denial/resistance 3. Vague awareness 4. Preplanning 5. Preparation 6. Implementation 7. Stabilization 8. Confirmation 9. Community ownership
Leadership	School leadership: To what extent are the school’s administrators (principal) supportive of PA and addressing SEB issues?
Community climate	School climate: To what extent do major segments of the school community support PA implementation? What is the prevailing attitude of the school toward addressing SEB issues?
Community knowledge of the issue	School knowledge about the issue: To what extent do school staff, teachers, and administrators know about the causes of the SEB problems in their students, the consequences, and how they affect the school?
Resources	School resources related to the issue: To what extent are school resources—people, time, money, space, training, external staff—available to support PA implementation efforts?

An adapted scoring rubric was created detailing characteristics of each readiness stage for each of the six dimensions. For example, with regard to the “leadership” dimension, Stage 3 (“denial/resistance”) was defined as “School administrators believe that social, emotional, and behavioral health issues are not an issue that must be addressed in the school.” For this same dimension, Stage 7 (“stabilization”) was defined as “Leaders are supportive of continuing basic efforts to implement PA and are considering resources available for self-sufficient PA implementation.”

A co–principal investigator, the PA implementation coordinator, and one graduate research assistant independently scored all interviews and subsequently participated in in-person discussions in order to reach consensus. As previously described, stages were slightly adapted from the original CRM, and interview data were scored using this adapted version. The team closely followed the scoring process guidelines detailed in the CRM handbook (Plested, Edwards, & Jumper-Thurman, 2006). Each school received an average readiness stage score for each dimension, based on all of that school’s interviews. The scores across the six dimensions for each school were then averaged to compute an overall stage of readiness score for the school. The scores that correspond with the readiness stages were “rounded down” rather than up, as indicated by the handbook (Plested et al., 2006). For example, a score of 6.7 would be a 6, rather than a 7. Scores were rounded down because a school that came very close to reaching a given readiness stage cannot be categorized as having already reached that stage.

Use of Assessment Data

Each fall, once the assessment process was complete, the evaluation team provided readiness assessment summaries to the schools, articulating their strengths and areas for improvement. This spurred dialogue between schools and PA team members that provided the basis for individualized technical support, such as distributing additional PA materials as needed or convening a planning committee if one was not already in place.

Assessments also informed whether support should be targeted toward the entire school, individual grades, or families. For example, in Table 2, which shows the nine readiness stages, their descriptions as they apply to PA, and the results of the CRM assessment across program years, we see that in the fall of 2008 many schools were in the “preplanning” and “preparation” stages whereas a few schools had already reached the “implementation” stage. The schools in the “preplanning” and “preparation” stages received additional school staff training in curriculum implementation so as to be assisted in moving from the planning stages to taking action in implementing the curriculum across the grades. For schools in the “implementation” stage, support included more focused efforts in working with individual teachers already implementing the curriculum to ensure that PA was delivered with consistency and integrity.

Table 2

Readiness Stages Scoring Rubric and Application to Positive Action (PA)

Community Readiness Stages Scoring Rubric			Results of PA Readiness Assessment
Readiness Stages	PA Readiness Assessment Interpretation		School Code	Fall 2008 Stage	Fall 2009 Stage	Fall 2010 Stage
No awareness	School has no knowledge of the need to implement PA.	↓ Application of the scoring rubric to PA ↓	A	5 = “Preparation”	6 = “Implementation”	6 = “Implementation”
Denial/resistance	No efforts have been made to implement PA this year.
Vague awareness	Individuals recognize the need to implement PA this year but have no immediate motivation to do anything.		B	4 = “Preplanning”	6 = “Implementation”	6 = “Implementation”
Preplanning	Some school personnel have met and begun to discuss developing PA implementation strategies for this year.		C	4 = “Preplanning”	6 = “Implementation”	6 = “Implementation”
Preparation	PA implementation is being planned for this year.		D	6 = “Implementation”	6 = “Implementation”	7 = “Stabilization”
Implementation	PA implementation has begun this year, or remains ongoing.
Stabilization	PA implementation has been running uninterrupted for several years.		E	4 = “Preplanning”	6 = “Implementation”	6 = “Implementation”
Confirmation	Several different PA components and activities are being implemented.		F	6 = “Implementation”	7 = “Stabilization”	7 = “Stabilization”
Community ownership	Evaluation plans are used to test the effectiveness of many different PA components		G	5 = “Preparation”	6 = “Implementation”	6 = “Implementation”

Stages on the individual readiness dimensions were also taken into consideration when applying tailored program support. For example, a school that had just experienced a leadership transition to a new principal received additional focused communication efforts with the school’s new leader. In another school, where bullying was considered to be a high-priority need on the dimension of school climate, tailored assistance focused on applying PA activities to address bullying. In a school where PA efforts varied widely among teachers in the same school, efforts were made to target at least one teacher from each grade to be a PA representative and encourage colleagues in implementation. Each school reached the “implementation” level by fall 2009 and at least maintained that level of readiness into fall 2010 (Table 2).

Conclusion

The CRM proved to be easily adaptable to a new setting, population, and health problem. The model’s guidelines are straightforward, and the scoring tools are easily employed. A reasonable amount of staff time was dedicated to in-school interviews and consensus scoring. However, interviews were quite short at only 30 minutes, and only a handful of informants were necessary. This is actually less resource time than is typical of most qualitative data collection efforts.

The results of the readiness assessment allowed for tailored technical assistance to each school. The movement from mostly “preplanning” and “preparation” stages in 2008 to the attainment of “implementation” stages in 2009 likely resulted from these tailored efforts. Arguably, without this assessment approach, the tailored implementation support would not have been possible.

Although CRM was utilized to assess readiness after a gap year in programming, the model may also be useful if employed before program implementation begins. In this case example, all of the schools had already recognized the importance of addressing social, emotional, and behavioral issues in their schools. If these levels of awareness were not already attained, the readiness assessment could inform education and awareness activities. Conducting the assessment earlier may reveal additional contextual factors that can further inform program design and delivery. These contextual factors are integral to a formative evaluation approach, as it allows for greater understanding of the context as early as possible.

Finally, previous applications of this model relied more heavily on community members’ involvement (Thurman et al., 2007). If applied as a tool in conjunction with a community-based participatory research approach, the assessment could involve members of the school community as interviewers. Although important considerations such as a compromise in the rigor of the methodology or the introduction of bias into the process must be taken into account, this type of approach may yield increased ownership of the process that may translate into further advancement through the readiness stages. This has been true of other formative evaluation approaches as well—more community involvement has led to increased community empowerment and improved use of evaluation results (Patton, 2008).

The possibilities of using the CRM approach are numerous. More research is needed to better understand the value of different model applications and how it could be best utilized across multiple settings. However, evidence suggests it has value in informing program implementation and understanding the variability across multiple program implementation sites.

Footnotes

The findings reported here are based on research funded by the following grants from the Institute of Education Sciences, U.S. Department of Education: R305L030072, R305L030004, and R305A080253 to the University of Illinois at Chicago (2003-2005) and Oregon State University (2005-2012).

The research described here was done using the program, training, and technical support of Positive Action, Inc., in which Dr. Flay’s spouse holds a significant financial interest.

Issues regarding conflict of interest were reported to the relevant institutions and appropriately managed following the institutional guidelines.

Notes

Associate Editors,Evaluation and Practice Department

Michael Fagen, PhD, MPH, is Clinical Assistant Professor and Director of the MidAmerica Center for Public Health Practice, Community Health Sciences, in the School of Public Health at the University of Illinois, Chicago.

Brad L. Neiger, PhD, MCHES, is a Professor and Associate Dean in the College of Life Sciences at Brigham Young University, Utah.

References

Beal

G. M.

(1965). Social action: Instigated social change in large social systems. Ames: Department of Economics and Sociology, Iowa State University.

Donnermeyer

J. F.

Plested

B. A.

Edwards

R. W.

Oetting

Littlethunder

(1997). Community readiness and prevention programs. Journal of the Community Development Society, 28, 65-83.

Edwards

R. W.

Jumper-Thurman

Plested

B. A.

Oetting

E. R.

Swanson

(2000). Community readiness: Research to practice. Journal of Community Psychology, 28, 291-307.

Flay

B. R.

Allred

C. G.

(2010). The Positive Action program: Improving academics, behavior and character by teaching comprehensive skills for successful learning and living. In Lovat

Toomey

(Eds.), International research handbook on values education and student wellbeing (pp. 471-501). Dordrecht, Netherlands: Springer.

K.-K.

Washburn

DuBois

D. L.

Vuchinich

Brechling

Flay

B. R.

(2011). Effects of the Positive Action programme on problem behaviors in elementary school students: A matched-pair, randomized control trial in Chicago. Psychology & Health, 26, 187-204.

Oetting

E. R.

Donnermeyer

J. F.

Plested

B. A.

Edwards

R. W.

Kelly

Beauvais

(1995). Assessing community readiness for prevention. Substance Use & Misuse, 30, 659-683.

Patton

M. Q.

(2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Plested

B. A.

Edwards

R. W.

Jumper-Thurman

(2006). Community readiness: A handbook for successful change. Fort Collins, CO: Tri-Ethnic Center for Prevention Research.

Prochaska

J. O.

DiClemente

C. C.

(1992). The transtheoretical approach. In Norcross

J. C.

Goldfried

M.R.

(Eds.), Handbook of psychotherapy integration (pp. 147-171). New York, NY: Oxford University Press.

10.

Purkey

W. W.

Novak

J. M.

(1996). Inviting school success: A self-concept approach to teaching, learning, and democratic practice. New York, NY: Wadsworth.

11.

Rogers

E. M.

(1994). Diffusion of innovations (4th ed.). New York, NY: Simon & Schuster.

12.

Thurman

P. J.

Vernon

I. S.

Plested

(2007). Advancing HIV/AIDS prevention among American Indians through capacity building and the community readiness model. Journal of Public Health Management and Practice, 13(Suppl.), S49-S54.