Abstract
Purpose:
Research shows that many evidence-based school programs are not sustained after the demonstration period is complete. This article outlines the Welcoming Empowerment Monitoring Approach and builds on data at each school—to address school safety and reduce substance use.
Method:
The study used California Healthy Kids Survey data across 145 schools in Southern California at five points in time over an 8-year period.
Results:
From project inception through 3 years after the project completed, all manifestations of school victimization dropped and were sustained. Lifetime alcohol and marijuana use also declined for all students during the overall period assessed.
Discussion:
Schools, districts, and regions tailored interventions according to each school’s and region’s specific needs. Combinations of ground-up solutions, evidence-based programs, building internal capacity, and connecting the school to resources helped reduce victimization and substance use.
The empirical school safety, bullying, and adolescent substance use literatures have grown exponentially during the past 25 years (Astor & Benbenishty, 2019). To a large extent, the school safety and substance use intervention literatures have focused on the creation and implementation of psychologically oriented evidence-based programs (EBPs) and to a much smaller extent, on the roles of organizational systems, support structures, regional resources, community partnerships, or policies (e.g., American Educational Research Association, 2013; Astor et al., 2009, 2010; Gaffney et al., 2019; Hodder et al., 2017; Li et al., 2017; Onrust et al., 2016). Currently, there are multiple clearinghouses that list and rate the scientific quality of substance use reduction, bullying, and school safety programs that are almost entirely psychologically focused (Fixsen et al., 2013; Midwest Comprehensive Center, n.d.). The federal government also provides a list of approved programs that schools can choose from when applying for grants (see http://schoolsafety.gov). Most of the programs listed in EBP clearinghouses are for elementary school populations of students (see U.S. Department of Education, n.d.). Very few school safety EBPs exist for or have been successfully sustained at the middle and high school levels (for exceptions, see Hawkins et al., 2014; Sugai & Horner, 2019).
Use of Evidence-Based Safety and Substance Use Programs Outside of Randomized Trials
Multiple meta-analyses and reviews have reported on the effectiveness of evidence-based school safety programs in diverse countries across the globe (Gaffney et al., 2019; Hahn et al., 2007). Although recent reviews have generally supported the effectiveness of antibullying programs, outcomes are heterogeneous for perpetration and victimization (Gaffney et al., 2019). Some reviews, descriptions, and analyses show very mixed short- and long-term results (e.g., Della Cioppa et al., 2015). Alternatively, a few scaled programs implemented in entire countries, cities, and states have touted strong and sustained reductions (e.g., see Hawkins et al., 2012); however, very few programs implemented at scale have been carefully monitored beyond an initial research period to examine the sustainability of school-based victimization or substance use outcomes (Hahn et al., 2007). School safety and substance use studies rarely include a careful examination of long-term postintervention outcomes based on changes in community, school district, or regional human and support resources (Nadeem & Ringle, 2016; Pinkelman et al., 2015).
Many issues related to organizational, social, and climate contexts of schools are dealt with in the implementation science literatures and only secondarily as part of the core philosophy or scientific underpinnings of a program or approach (Horner et al., 2017). By contrast, “ground-up” approaches and empowerment methods have long been part of social work philosophy and interventions (National Association of Social Workers, 2017). Ground-up and researcher–participant partnerships have also gained popularity in public health and with youth empowerment approaches (e.g., Israel et al., 1998; Wilson et al., 2007). Such approaches allow researchers, students, parents, educators, and community members to be equal partners in the research and solution identification. However, few approaches incorporating ground-up processes as part of the intervention have been examined at scale. In part, this may stem from the practical reality that each school in a district or region may choose to implement different strategies in different ways, with differing degrees of program fidelity. From one viewpoint, allowing for wide latitude in community or ground-up implementation goes against many of the criteria for fidelity of programs (Gearing et al., 2011). From another equally valid scientific perspective, it could allow for greater fitting of the interventions to the multiple local variables that are germane to the setting in that point in time (Chambers & Norton, 2016).
Only a handful of programs (namely, Communities that Care and Positive Behavioral Interventions and Supports) allow for significant ground-up community-, teacher-, youth-, or district-generated solutions as part of their youth or educational empowerment interventions (Benbenishty & Astor, 2007, 2008; Benbenishty et al., 2003; Hawkins et al., 2012, 2014; Sugai & Horner, 2006, 2019). The present study is based on a similar strategy, the Welcoming Empowerment Monitoring Approach (WEMA; Astor & Benbenishty, 2018, 2019; Astor et al., 2018; Benbenishty & Astor, 2012).
Sustainability, Transferability, and Implementation of School Safety Approaches
Reports in the policy and clinical literatures indicate that many top-down EBPs are not sustained after the demonstration period is complete (Fixsen et al., 2013, 2005; Nadeem & Ringle, 2016; Pinkelman et al., 2015). Research also commonly indicates that programs are not easily adapted to different demographic, cultural, and geographic regions (Baumann et al., 2015; Nese et al., 2016). Some scholars suggest that many state- and clearinghouse-endorsed “EBPs” are not easily scalable organizationally beyond the smaller sets of schools normally recruited in convenience samples where the students or classes are then randomized (Astor et al., 2016).
The implementation science literature has made a major effort to address the gaps in application of EBPs to real-life settings (Chambers & Norton, 2016). However, these efforts are often viewed as a form of adaptation designed to increase the original fidelity of the program rather than starting with the views of students, teachers, and the local community. Evaluations commonly focus on the extent to which a program is implemented with fidelity rather than the long-term outcomes (Fixsen et al., 2013). The goodness of fit between EBPs and different contexts has been widely discussed in many empirical literatures including medicine, nursing, psychology, social work, and public health (Fixsen et al., 2005). However, goodness of fit over time has been discussed to a far less extent in the school safety literature (Cross & Barnes, 2014).
Multiple Co-Occurring School Safety and Substance Use Risks in Schools
Schools wanting to initiate a school safety program face a challenge of selecting from myriad programs for different types of safety problems and various vulnerable groups experiencing safety issues in each school (U.S. Department of Education, n.d.). Hence, a school that has overlapping problems with substance use, bullying, and multiple forms of physical or emotional victimization might need to implement several types of programs to address its issues. Moreover, if schools or regions have many vulnerable groups (e.g., children experiencing homelessness, foster children, refugees, LGBTQ children, or children with disabilities), they may need to select from a variety of programs that are specifically approved for those groups. Needing to purchase and train for so many programs can become unscalable and unsustainable over time.
Basic epidemiological research has long shown that substance use and victimization on school grounds are highly associated with each other (Espelage et al., 2012; Tharp-Taylor et al., 2009). In fact, many state and national organizations combine the two as part of their mission (e.g., Office of Safe and Drug-Free Schools). Yet solutions to multiple risks on campus are rarely addressed by intervention programs. With a few exceptions, these literatures and interventions tend to be siloed by topic and discipline. Very often, the same research studies publish separate topics in different disciplines and content-focused journals, even when those findings involve the same groups of students from the same schools and with the same teachers (e.g., separately publishing in substance use, school violence, or mental health literatures depending on the dependent variable and focus of the journal).
In the school-based EBP literatures, implementation science has responded to weaknesses of the traditional, strict fidelity required of EBPs (Forman et al., 2013) and acknowledged that school organization, processes, and contexts differ greatly and should not be standardized. Even so, very little research has been done with school safety or substance use prevention efforts based on the quality of procedures or organizational systems. How schools use data organizationally and in an ongoing manner and how school districts build infrastructure so that scaled programs can be adapted for each school are areas not widely researched in the school safety or school substance use literatures.
This article focuses on how a place-based and school-wide social work approach responds to many interrelated challenges in schools, creating systemic changes, scaling up, and sustaining safety and substance use reduction interventions. “Place-based” refers to the assumption that every school has a unique ecological context with specific needs, risks, and characteristics that interventions must address. “School-wide” refers to our assumption that interventions must be integrated into all aspects of the school, including space, time, and mission, and address the needs of all groups affiliated with the school (e.g., students, teachers, parents, and community members). A whole school approach is different, for example, than one class implementing one intervention and another class using a different one, or a sole focus on one vulnerable group to the exclusion of all other students. Our approach caters resources and programs to each school through an inclusive, ground-up, and participatory process. Rather than a sole reliance on implementing EBPs that have been tested in other contexts, WEMA builds on a data-rich infrastructure—generated by the voices and reports of students, parents, teachers, and administrators at each school—that then is used as a feedback system to build human, organizational, and community capacity to address school safety and reduce substance use (Astor & Benbenishty, 2018, 2019; Astor et al., 2004). WEMA has been sustained successfully at scale in Israel for more than 22 years (Astor et al., 2011; Astor, Meyer-Reynolds, et al., 2012; Benbenishty & Astor, 2003; Benbenishty et al., 2020) and more recently in Chile (López et al., 2018), the United States, parts of Canada, and China as part of social work thinking and methods but has not yet been examined empirically in U.S. secondary schools.
Building Capacity and Monitoring
WEMA is a social work model aligned with our profession and research assumptions. The processes are the same in each context and culture. Because WEMA is a place-based and school-wide intervention, variations in culture and traditions of each school site are reflected in the local intervention. WEMA differs from other approaches in several respects and has six key assumptions that undergird its methods and strategy (Astor & Benbenishty, 2018; Astor et al., 2004). These assumptions reflect the tenets of our theory of schools in evolving contexts (Astor & Benbenishty, 2019).
Key Assumptions and Principles of WEMA
The school is in the center of the conceptual model
We start with a focus on the school as a system rather than on individual students as clients. This entails a school-wide approach, focusing on making changes at the school system and organizational level to address challenges and vulnerable groups. The system focus encompasses the whole school community: students, staff, and parents. As such, it is essential to address the needs and desires of each of these groups and deal with potential conflicts and mismatches.
Schools are nested in multiple contexts that affect every school differently
Schools vary significantly in the challenges they face and the resources they can muster. Many schools are embedded in contexts that provide few resources and have discriminatory and inadequate opportunity structures (Astor et al., 2021). There are several implications of this conceptualization, namely that external contexts are a target for change. Faulty opportunity structures, inadequate resources, and discriminatory resource allocation policies are addressed through multiple means, including advocacy, coalition building, and changes in legislation. Furthermore, a major component of WEMA is to mobilize sustainable resources from multiple social institutions and provide them to schools that need them, such as social work and psychology interns from local universities and colleges, and helping schools take advantage of existing and stable funding streams. Given the large variations among schools, it is clear that a one-size program will not fit all schools, even those in the same district. We, therefore, do not attempt to implement one EBP in all district schools.
Schools create internal organizational, social, and academic climates that buffer, moderate, and mediate external influences
Although the external contexts have important impacts on a school, they do not predetermine outcomes. The internal context that the school creates plays an important role. Consequently, in WEMA, a major emphasis is on helping schools develop effective internal mechanisms to face external challenges and creating a welcoming and positive climate for the whole school community. This can be accomplished by helping the school make organizational changes and enhance its ability to provide academic supports to students. Here again, the focus is on sustainability of positive internal contexts, which requires building and enhancing the school’s long-term capacity rather than relying on short-term intervention funding.
The emphasis on supporting the school’s internal mechanisms to deal with external challenges is a ground-up approach. Each local community and school may have intrinsic and important knowledge unique to that school, including experiences of local community oppression, racial disparities, cultural strengths, and areas of pride and celebration. Based on this local knowledge, those who live and learn there may have solutions that work in their context. In WEMA, a ground-up approach is used to identify such local solutions and test them empirically. Because both external and internal contexts vary so widely across schools, solutions generated in one school are not automatically assumed to generalize to other settings. Instead, locally driven solutions are used as case examples that other settings could modify, test, and adopt.
Although we strongly support the scaling up of locally developed successful practices and interventions, we also see the important role of EBPs that have been tested in other contexts. We emphasize, however, the need for a careful analysis of their relevance to a particular context and encourage modifications that make them more compatible with the local context. Furthermore, as with ground-up solutions, effectiveness cannot be assumed and must be tested in the local context.
External and internal contexts are continuously changing
Each school evolves over time and is likely different in the types of risks and behaviors present at any given time. By design, there is constant transition in each school because students are continually moving between grades, entering, or exiting by graduating or attending different schools. Teachers and administrators turn over regularly, as does district leadership and organization (Astor & Benbenishty, 2019). Challenges that a school faces today are not the same as yesterday’s challenges, and interventions that worked today may not work tomorrow. Instead of thinking “pre–post” intervention, we promote a continuous “finger on the pulse” and practices and interventions that are responsive to changes.
Continuous monitoring processes underlie an evidence-informed approach
Our understanding of external and internal school contexts, challenges and needs, and the progress made toward achieving a school’s goals is based on systematic and ongoing data gathering, analysis, and interpretation. WEMA assumes the data used to solve safety and school climate problems must reflect the multiple voices of students, teachers, school staff, administrators, and parents (Astor & Benbenishty, 2018; Astor et al., 2012). As such, school data and voices must be assessed and used at regular and frequent intervals (Sullivan et al., 2015). A data system is developed to serve as the primary engine to assess needs, apply needed programs, build supports, create relationships with community and partnerships with universities, and use district-level and community infrastructure to address the localized problems in each school. Such data systems are most effective when they include quantitative, qualitative, mixed, and participatory methods (Astor & Benbenishty, 2018).
WEMA is rooted deeply in the ideology, practices, and ethics of social work
WEMA puts a major emphasis on listening to the voices of the school community and empowering the various members of the community to be part of decision making, planning, and interventions. Our approach seeks to learn from what social workers and other local practitioners are doing, assess these practices empirically, and scale up practices and programs that work. Hence, although we think that practitioners should have access to relevant EBPs developed elsewhere, we seek to empower practitioners first. Our social work focus is also evident in how we target changes to socioecological systems in which the school is embedded, our advocacy for vulnerable populations on a state and national level, and efforts to engage legislators in addressing school and community needs.
Building Capacity and Welcoming Schools Project
Using WEMA, a large-scale school safety, positive school climate, and risk reduction consortium was initiated in fall 2010 for civilian public schools with a large number of military-connected students. Preparations and preintervention baseline data collection began in fall 2009. The aims of the building capacity and welcoming schools consortium were to address school safety, risk behaviors, and mental health by targeting systemic changes across whole schools and districts. Our intervention approach reflected the WEMA principles outlined here and involved providing sustainable resources, building and enhancing internal capacity, implementing structured interventions, and attempting to change the external contexts in which these military-connected schools were embedded. It should be noted that although these activities are described separately, they were interconnected and mutually reinforcing.
Providing Resources to Enhance and Build Capacity
The consortium project received generous funding. Nonetheless, following WEMA principles, we did not provide direct funding to consortium members from the project budget. Instead, we leveraged project funding to help connect the consortium to stable sources of future funding and build sustainable capacity in districts and schools, including by making organizational changes that helped attract additional external funding.
Building capacity through placement of graduate interns
The consortium developed an internship program that placed graduate-level interns from social work, school counseling, and school psychology in consortium schools. This intervention reflects two components of our approach: a focus on providing resources to underresourced schools and building sustainable internal capacity. During the first 3 years, 172 graduate-level interns from local universities were placed in consortium schools. Interns reported spending more than 81,000 hours working with 4,137 students, 1,538 student groups, 1,000 parents, and an additional 1,107 students who participated in school-based Families OverComing Under Stress (FOCUS) groups (Cederbaum et al., 2014; Esqueda et al., 2014). Interns also participated in or led other practice-related activities, including social behavioral assessments, team meetings, and individualized educational plan (IEP) and special education meetings. They shared examples of inspirational programs, ideas, activities, events, and partnerships across the consortium. Importantly, through this process, many schools learned the value of such service providers and submitted grants and changed financial priorities to help recruit more social workers.
Connecting to university service-learning programs
Most universities have service-learning programs designed to provide their students with experiences in their areas of study including social work, psychology, and teacher training. Partners at Learning (PAL), a program through University of California, San Diego’s education studies department, provides opportunities for undergraduates to work with underserved schools and students. Each year, PAL undergraduates worked with hundreds of students across more than 20 consortium schools, providing mentoring and tutoring and contributing more than 20,000 volunteer hours (Capp, Benbenishty, et al., 2017).
Promoting organizational changes that increase resources
Schools and districts need organizational structures and processes to acquire resources. Although multiple grants and funding sources may be available, districts and schools need to have the organizational capacity to identify potential resources, apply for them, and manage them in ways that increase the sustainability of their cash flow. An important part of the intervention was to consult with schools on funding opportunities and make suggestions, showing successful examples from neighboring districts of effective ways to acquire and successfully manage external funding. In some cases, all the school had to do was count the number of military-connected students to receive military aid funds. In other schools, obtaining additional funding required helping them establish a department to specialize in submitting state and federal grants. Such organizational changes helped schools increase their resources manyfold.
Informing about available resources
We created a smartphone application that included information about all military-connected resources available in the community. The purpose was to ease the use of services and better link schools, students, parents, and educators to local resources. All school employees, parents, students, and nongovernmental organizations were encouraged to download the app and use it to find resources (Benbenishty, 2017).
Building Capacity Through Professional Development
The consortium created several professional development opportunities for staff in various roles (e.g., teachers, principals, pupil personnel, district-level officials, and interns placed in consortium schools). Professional development topics were selected in consultation with district leadership and in response to findings from our monitoring system (detailed below). A series of workshops were provided: threat assessment, psychological first aid, and bullying prevention.
Four practice guides were also created, published, and distributed by Columbia University Teachers College Press (Astor et al., 2012a, 2012b; Astor, Jacobson, Benbenishty, Cederbaum, et al., 2012; Astor, Jacobson, Benbenishty, Pineda, et al., 2012). These guides highlight best practices, as identified in the research literature, and those implemented in the Building Capacity Consortium. To support the vulnerable group of military-connected students, these guides aimed to help educators, administrators, and families of military children. They included an introduction to military culture and school-based interventions and strategies to support military-connected students in schools (Castillo et al., 2017).
Implementing Both Grassroots and Established EBPs
The consortium aimed to identify interventions that could be disseminated as EBPs for military-connected schools. Some of the programs implemented were grassroots practices developed locally (see Because Nice Matters). Others were programs with prior evidence of effectiveness but had not yet been tested in military-connected schools (e.g., Learning Together). Existing programs were also modified to address the needs of consortium schools (see FOCUS Skill Building Groups). In a series of articles, we detailed our separate evaluations of these programs (Berkowitz et al., 2014; Capp, Astor, et al., 2017; Capp, Benbenishty, et al., 2017; Capp et al., 2018; De Pedro, Esqueda, et al., 2014; De Pedro et al., 2017; Garcia et al., 2015).
Because Nice Matters
One school district previously implemented a district-wide antibullying program, Because Nice Matters. The monitoring process was used to determine whether there was any evidence for the effectiveness of the program as a potential grassroots program that could be scaled up for other schools and possibly school districts. The program had been developed by local stakeholders and did not require many resources, suggesting it would be more easily sustained over time. Because Nice Matters encourages and recognizes kind behavior and involves symbolic activities, such as wearing purple and black to remind everyone that bullying can cause physical and psychological damage (De Pedro et al., 2017).
Learning Together
A program developed by The Learning Together Company features two components: reading together and doing math together. Learning Together is a peer-to-peer tutoring program that immerses students into a collaborative learning experience and stresses social and emotional experiences as an important part of learning (for a review of Learning Together in consortium schools, see Capp et al., 2018). Here, the monitoring system examined how an existing EBP fit in specific schools in the consortium districts.
FOCUS school-based skill-building groups
A family-level resilience training program designed to assist families experiencing high levels of stress, FOCUS is based on evidence-based family intervention models for at-risk families and aims to enhance family cohesion and strengthen parent–child, marital, and coparenting relationships by helping families develop a shared understanding of past experiences and build skills such as emotional regulation, communication, problem-solving, goal setting, and managing trauma and loss reminders (https://focusproject.org; see Garcia et al., 2015).
Impacting Proximal and Distal External Contexts
The conceptual framework and social work orientation underlying WEMA support an ecological approach and encourage interventions that target the contexts in which schools are embedded. Many of our efforts, therefore, were aimed outside the schools in the consortium.
The proximal contexts included many institutions (e.g., public universities) and nongovernmental organizations, many of them unaware of the underserved group of military-connected students in consortium schools. We employed advocacy and community activities aimed to increase awareness of these students and build closer collaborations with and in support of these schools. This included making presentations about the consortium in nongovernmental organizations’ coalition meetings to encourage resource sharing, meeting with key officials in higher education institutions to promote placement of interns, implementing service-learning programs in consortium schools, and developing programs to facilitate college acceptance of high school students.
Other awareness-building activities were carried out for the larger public. These included, for instance, a ceremonial day devoted to military students during a local Major League Baseball game and special public activities to commemorate Veteran’s Day and Military Child Month.
Change efforts were also directed at the state and national levels. The authors provided information and advocated on behalf of military students in meetings with state and federal congressional representatives and joined the Military Children Education Coalition in nationwide efforts to require that all school information systems have a “military children identifier” to ensure that such students become more visible. Furthermore, the authors took part in activities sponsored by then-Second Lady Jill Biden and then-First Lady Michelle Obama on behalf of military students.
A Monitoring System to Support WEMA
WEMA is evidence informed. As such, it requires a data system to inform the multiple interventions and outcomes relevant to each location and context. The system, which includes two components, reflects the assumptions and principles underlying WEMA.
California Healthy Kids Survey (CHKS)
Administered to all schools in the state biannually by the California Department of Education in cooperation with WestEd, a nonprofit research and development agency working at a national, state, and local level (Gilreath et al., 2014), the CHKS was selected as the centerpiece of the feedback-empowerment-monitoring system because it was an existing survey administered to most California schools and included surveys for students, parents, and school staff (also known as CalSCHLS; https://calschls.org). The choice of this survey was based to a large extent on the importance we put on sustainability. Although we could use grant money to develop a survey that may have been more in line with the goals of this particular project, we preferred to adopt an existing system that would be sustained over time without additional funding.
Based on the emphasis on local contexts, the consortium worked with the California Department of Education (through WestEd, the subcontractor) to modify existing survey modules and develop a military-connected schools module for students, staff, and parents (Gilreath et al., 2014). The military module, once created for our consortium, was opened for use by all schools in California. Schools in the consortium could choose from many modules and compare results over time to themselves, their local district schools, and similar schools in the state (Capp, Astor, et al., 2017).
Although the CHKS had been in use for many years, the consortium made multiple efforts to improve the dissemination and use of survey information to support school improvement. For example, instead of district-level reports, we advocated for school-level reports that would be more sensitive to school-level variations. Meetings were conducted with district and school leadership teams to demonstrate how reports could be interpreted and used for local decision making.
A local monitoring system
Given the myriad activities implemented during the project, we created a flexible system of online surveys. These surveys were used to solicit feedback from multiple constituents on professional development events, the many programs employed, and any other activities carried out during the project (see Cederbaum et al., 2014; Esqueda et al., 2014). Such surveys were administered to interns, field instructors, and school representatives approximately twice per year (see Cederbaum et al., 2014; De Pedro, Atuel, et al., 2014). Information gathered through this local monitoring system was disseminated to all districts and schools in real time to support local decision making.
Method
This article reports partial findings from a WEMA-based intervention project aimed at reducing multiple forms of bullying, school-based victimization, weapon involvement, gang affiliation, and risk behaviors including substance use and mental health issues. We focus on monitoring across five points (before, during, and after the intervention), multiple aspects of school violence (victimization, weapon-related behaviors, and gang membership), climate, safety, and substance use in all secondary schools, based on the CHKS. We have presented other outcomes and aspects of the intervention in research reports and published manuscripts (Benbenishty, 2014, 2017).
Sample
The Building Capacity Consortium was composed of eight public school districts and 145 schools serving approximately 117,000 K–12 students (about 10% of whom were military connected) in California. This study included 73,415 students (51.5% male) from Grades 7 (36.7%), 9 (32.1%), and 11 (29.2%; Tables 1 and 2).
Student Number and Percentage of Gender per Year.
Note. Gender was unknown for 1,010 participants.
Student Number and Percentage of Grade Level by Year.
Note. Grade level was unknown for 2,064 participants.
Data Source
We analyzed data from the CHKS administered by the California Department of Education and available to California secondary schools for 3 decades. The CHKS is a survey administered biannually to fifth-, seventh-, ninth-, and 11th-grade students. The survey is conducted in a census-like manner among all school districts, schools, and students in relevant grades. Each 2-year wave provides a representative sample of the state of California. Prior statewide studies using this database reported that approximately 85% of school districts in California participate (Benbenishty et al., 2018). Student-level response rates in the districts of this study were about 87%. Seventh, ninth, and 11th graders in our sample received CHKS questionnaires examining substance use and school victimization in 2009, 2011, 2013, 2015, and 2017.
Measures and Analytic Plan
Victimization and involvement in school violence
School victimization
Students reported whether they were victimized in the last year on a scale of 0, 1, 2 or 3, or 4+ times. Seven victimization types were included (e.g., “You were pushed, shoved, kicked, someone spread mean rumors and lies about you”; “You were made fun of”). Each item was dichotomized (0 = did not happen, 1 = happened at least once). We also computed a mean as an index of victimization (α = .77).
Involvement with weapons
Four behaviors were used to describe weapon involvement in the past year: “Did you carry a gun to school?” “Did you carry another weapon (such as a knife or a club) to school?” “Have you been threatened or injured with a gun?” and “Have you seen another student with a gun or knife at school?” We also computed a mean as an index of weapon involvement (α = .73).
Gang membership
Gang membership was assessed using one yes–no question, “Do you consider yourself a gang member?”
Substance use
Lifetime use
Participants were asked questions regarding their lifetime use of substances. The scale was 0, 1, 2, 3, 4–6, or 7 or more times. Specifically, they were asked about their use of cigarettes, marijuana (pot, weed, grass, hash, and bud), and inhalants. They were also asked about having at least one drink of alcohol and binge drinking (i.e., five or more drinks of alcohol within a couple of hours). We dichotomized each of the items (0 = no, 1 = at least once). We also created indexes that were the means of the relevant items.
Use in last 30 days
Participants were asked the same set of questions regarding how many days in the last month they used substances. The scale was 0, 1 day, 2 days, 3–9 days, 10–19 days, or 20–30 days. Additionally, participants were asked about substance use on school property; specifically, how many days in the last month they smoked cigarettes, smoked marijuana, or had at least one drink of alcohol. We dichotomized each of the items (0 = no, 1 = at least once). We also created indexes that were the means of the relevant items.
School climate
Students were asked whether they agree or disagree with statements about their school using a 5-point scale: 1 (strongly disagree), 2 (disagree), 3 (neither agree nor disagree), 4 (agree), and 5 (strongly agree).
Belongingness
Belongingness (α = .82) was computed as the mean of three items: “Do you feel close to the people at school?” “Are you happy to be at this school?” and “Do you feel like you are part of this school?”
Teacher support
This variable (α = .89) was measured as the mean of six items (e.g., “Do teachers treat students fairly at school?” “Do teachers and other adults care about you?” “Do teachers and other adults tell you when you do a good job?”).
Meaningful participation
This variable (α = .77) was computed as the mean of three items: “At school, you help make class rules or choose things to do?” “How well do you do in your school work?” and “At school, I do things to be helpful.”)
Safety
Safety (α = .71) was assessed as a mean of two items: “I feel safe at school” and “How safe do you feel in school?”
Analytic Plan
We assessed the percentage, means, and standard deviations of each item and scales in each of the 5 years of the study. We conducted one-way analyses of variance for each variable with post hoc comparisons (Scheffe’s significance test). Given the large sample size, we used p < .001 as the significance level.
Findings
Violence and Victimization
We found that secondary students in consortium schools had strong, sustained, and statistically significant reductions in moderate and severe forms of victimization. The reductions for severe victimization and weapons use are quite dramatic. Table 3 shows the number and percentage of students in each wave reporting different forms of victimization on school grounds. The pattern of reductions and sustainability was evident in almost all moderate, severe, and weapon use items tracked. The table shows that the most prevalent behaviors were associated with verbal or social victimization: “Had sexual jokes, comments, or gestures made to you at school”; “Had mean rumors or lies spread about you at school”; and “Been made fun of because of your looks or the way you talk at school.” Less prevalent behaviors were associated with physical victimization, for example, “Been in a physical fight at school” and “Been afraid of being beaten up at school.”
School Victimization: Number (Percentage) of “At Least Once” in Past 12 Months by Year.
a Significantly (p < .001) lower than 2009.b Significantly (p < .001) lower than 2011.c Significantly (p < .001) lower than 2013.d Significantly (p < .001) lower than 2015.
The pattern of change was consistent and quite strong—all manifestations of school victimization dropped. For example, in 2009, 9.6% of students reported carrying a weapon on school grounds. This percentage dropped significantly in each wave and was sustained after the intervention finished. In 2017, 5.1% students reported bringing a weapon on school grounds (a 47% reduction).
Lifetime and 30-Day Substance Use
Reductions were even stronger for substance use. Table 4 shows lifetime use and Table 5 shows 30-day use of substances. The most common substance used was alcohol. There was a decline in alcohol use for all students during the five time periods assessed. The use of marijuana also declined across the five time periods. The pattern was similar for other substances.
Number (Percentage) of Students Reporting Lifetime Smoking or Substance Use by Year.
a Significantly (p < .001) lower than 2009. b Significantly (p < .001) lower than 2011. c Significantly (p < .001) lower than 2013. d Significantly (p < .001) lower than 2015.
Number (Percentage) of Students Reporting Smoking or Substance Use at Least Once in Past 30 Days.
a Significantly (p < .001) lower than 2009. b Significantly (p < .001) lower than 2011. c Significantly (p < .001) lower than 2013. d Significantly (p < .001) lower than 2015.
We present both tables not only to show consistency but also because lifetime substance use can be an indirect sign that the intervention was a primary prevention. In other words, the climate and overall response of the school and community actually may have contributed to students never trying the substances. This is important to examine in projects that span long periods because the culture of the school, peer group, parents, and educators may have changed to such a degree that substance use is not only reduced but also a considerable number of students never try drugs. For both 30-day substance use and lifetime (prevention), we see many types of substances with significant reductions between 2009 and 2017. Reductions in substance use and trying drugs were sustained by the schools after the program completed, for at least 3 years.
Table 5 presents findings on use of substances in the past 30 days. Here again, students show impressive general reductions in 30-day substance use. Students were also asked to report whether they used substances on school grounds. Table 5 shows that few students reported such behavior, and when they did, they were more likely to use alcohol or marijuana on school property than other substances. Here again, the reduction trend of school-based substance use is consistent.
Summary indices are presented in Table 6. These show similar trends. There was a decrease in victimization, weapon involvement, lifetime substance use, and 30-day substance use over the years. As with prior analyses, the findings surrounding violence and substance use indicate rather larger reductions over time. Changes regarding other aspects studied, such as school climate, were less consistent and showed much smaller increases over time.
Means (Standard Deviations [SDs]) of Summary Indexes by Year.
a Significantly (p < .001) higher than 2009. b Significantly (p < .001) higher than 2011. c Significantly (p < .001) higher than 2013. d Significantly (p < .001) higher than 2015. e Significantly (p < .001) lower than 2009. f Significantly (p < .001) lower than 2011. g Significantly (p < .001) lower than 2013. h Significantly (p < .001) lower than 2015.
Discussion and Applications to Practice
This article reports partial findings from a large-scale school intervention in California secondary schools (middle and high schools) using WEMA. Overall, the findings show rather deep and consistent reductions for most forms of school victimization and substance use during an 8-year period for the 145 schools in eight school districts that were part of the consortium. Victimization behaviors showing reductions ranged from severe (e.g., being threatened by weapons) to verbal and psychological (e.g., name-calling, rumors, and social exclusion). Substance use reductions were strong for an array of substances. Reductions of substance use were exhibited both for the past 30 days and lifetime.
The outcomes of this study were sustained over time. Furthermore, most of the substance use and victimization reductions were sustained over time and in most cases reduced even further—even up to 3 years after the initial intervention ceased. We believe this provides initial and tentative evidence for the sustainability of the approach after researchers, grants, and funding supports stopped. Very few school safety research programs or approaches have examined the impact of an intervention on the school setting to see whether student outcomes are sustained 3 years after an intervention. It is important to note that based on our theoretical framework, we followed up on schools rather than tracking individual students, as do other studies (e.g., Espelage et al., 2012). Hence, any sustained changes in students’ behaviors and perceptions are not the result of the direct intervention. Rather, they suggest that the original intervention changed the school’s internal and external contexts in ways that had a positive impact on students.
Each wave of student outcomes in our study represents different students in different grades. They may have different teachers. Superintendents had 150% turnover, and there was more than 50% turnover of administrators in the district during the 8 years studied. Yet different students in the same schools showed reductions wave to wave and multiple years after the approach was implemented. This suggests that something organizational, cultural, or systemic has changed in the consortium schools during the time frame.
The study provides evidence that schools, districts, and regions can tailor interventions according to each school’s and region’s specific needs and that they do not need a “one size fits all” EBP with limited implementation flexibility and a major focus on fidelity. It also suggests that combinations of ground-up solutions, EBPs, building internal capacity, and connecting the school to stable resources can work positively to reduce victimization and substance use outcomes for each school and community.
Based on our theoretical framework, we expected the change mechanism would be that our interventions would lead to improvement in school climate that would change student outcomes. Surprisingly, school climate showed inconsistent and only mild to moderate improvements at different points. Given the literature on school climate, we expected improvements to be significant and strong. The findings indicate, however, that only the subjective dimension of feeling safe at school seemed to improve in the long run.
Subsequent research indicates that the way school staff experienced climate was not always similar to the way students experienced it, which may have affected the outcomes in unanticipated ways. This was a blind spot in our theoretical framework that has prompted our team to explore the issue further. Clearly, a school is not likely to have a positive climate when the work conditions for educators and staff are not supportive of their positive development (Capp et al., 2020a, 2020b). This consideration needs to be incorporated into not only future studies but also school-based violence and substance use interventions. Although almost all programs rely on educators and staff for climate interventions, few focus on the staff climate directly (Capp et al., 2020a, 2020b).
What contributed to the deep and sustained changes? Our design does not allow direct causal inferences. Nonetheless, we think our approach empowered schools, administrators, and communities to work on problems they prioritized. Because WEMA is an empowerment approach, the school and community decided which problems they were interested in working on and what approach they wanted to take. Each district and school was encouraged to discuss, deliberate, and organize on issues that they felt were most pressing to them. Often, the researchers would point out other areas the schools could work on. However, what became clear to us was that schools and communities did not have the capacity or resources to work on many issues simultaneously. They selected issues they cared about the most and felt would galvanize the staff, parents, students, school board, and community at large. This is a very important and qualitative difference between WEMA and other approaches.
Furthermore, because data were collected over time, schools were encouraged to add or switch focuses to areas that were deemed more important to them based on the data. For example, bullying and substance abuse were initially selected as issues districts wanted to address. However, over time, suicide and weapons use became more important to many of the schools, in part based on national events and the media, but also based on trends they saw in the data. WEMA processes allow and encourage communities to fit their goals and interventions to what they want and have the capacity to work on over long periods. Schools did change their foci and priorities during the project based on local events such as a suicide, outside events such as a mass shooting, media reports, and policy changes (e.g., changes in funding priorities). WEMA enables schools to adapt quickly to changing realities.
We think local monitoring is essential to support school empowerment. When we started the project, CHKS reports were provided only on a district level and as we were told time and time again, were not really used. Based on WEMA, we generated school-level reports and conducted many meetings to help schools interpret and make use of the data. Each school could then use its own data and feedback from the staff, parents, students, or the community to track, monitor, and keep a finger on the pulse of how the interventions they chose were progressing. The process was designed to be empowering, constructive (not punitive), welcoming, and evolving. Members of the school community could prioritize their outcome goals and alter or adjust their strategies in real time. For greater engagement, the nature of the local data needed to be recent and representative and incorporate the voices of different groups in the school (Astor et al., 1999; Benbenishty & Astor, 2007, 2008; Benbenishty et al., 2003). Individual schools and school districts, thus, could experiment with combinations of approaches and adjust them more sensitively over time. Such capability may have also evoked greater ownership over the interventions chosen because they were free to engage the school community in their creation and maintenance. In many of the districts and schools, we witnessed this dynamic. We believe having multiple perspectives in reporting and providing feedback on behaviors and responses was critical.
Our overall evidence-informed belief, based on numerous observations and other qualitative data that could not be presented in this article (see Benbenishty, 2014, 2017), is that organizational changes contributed to improvements. Many of our organizational consultations aimed to change the ways the district and school leadership organized personnel relevant to changes in climate and developed more resources. When the project started, only six of the eight school districts in the consortium had pupil personnel departments or staff that were responsible for maintaining staff in-services, human resources, community supports, communication strategies, or data management. Working with schools and school districts to reorganize so there were district-level personnel responsible to oversee the processes and programs chosen was particularly important and helped districts own and evolve to address the changing needs in their schools.
Schools also hired more social workers due to the popularity of the Master of Social Work internship program introduced as part of the intervention. There was similar excitement about the master’s-level counseling, school psychology, and military counseling programs we helped coordinate. By the end of the intervention, school districts had hired more than 100 full-time equivalent (FTE) pupil personnel service (PPS) workers to oversee approximately 200 graduate students in psychology, social work, and counseling programs and another 500 undergraduates. Three years after the end of the WEMA, there were more than 200 FTE PPS workers in the district, with many more university partnership linkages involving hundreds of university students and staff. From discussions with former superintendents, it appears that other neighboring school districts saw the utility and impact of the PPS workers and interns and also expanded dramatically their numbers, often in places that previously had no PPS workers. It appears the growth may have been regional as well as in our consortium after WEMA concluded.
Similar, if not more impressive, changes took place in most (but not all) districts in terms of enhancing their interest and capacity to apply successfully for more grants and funding from multiple sources. This was also supported by the connections made with community nongovernmental organizations, county services and departments, military family services, and youth development services such as sports, recreation, arts, day care, and afterschool care. We created formal relationships with more than 400 local organizations that were then linked to the school districts, schools, and families based on how their services matched the needs and goals of the schools. Many of these relationships expanded and grew dramatically during the 8-year intervention and postintervention period. Programs focused on day care, sports, art, employment for parents, and health support were connected to each school district and then to the schools with the greatest needs in the consortium. Many of these school–organization relationships continue without any memory of their origin.
Similarly, formal relationships with six local universities and academic departments created ongoing internships at the undergraduate level (more than 1,000 per year now), at the graduate level (more than 200 per year now), and through mentoring programs (more than 500 per year). Such relationships also linked the schools to researchers, evaluators, and graduates who partnered with and volunteered to support the schools. Several universities created special partnerships with the schools for automatic undergraduate admissions for graduates with a B-average grade point average. Other organizations provided tutoring and mentoring programs. Still others have been critical in helping districts manage data. These relationships are still continuing and growing.
The relationships have helped generate greater partnerships, but they have also enhanced the schools’ and districts’ capacity to take advantage of funding sources such as California Local Control Accountability Program (https://www.cde.ca.gov/re/lc/); military impact aid (https://oese.ed.gov/offices/office-of-formula-grants/impact-aid-program/); state, county, and national grants; state-level homeless and foster care funding; and funding related to mental health, school climate, and socioemotional learning programs. We estimate the growth in funding to be in the hundreds of millions of dollars for the consortium during and since the program completed. Developing large-scale funding was not part of district and school thinking before WEMA began. Except for two districts, none had applied in the past for ongoing grants or state funding programs.
We experienced multiple methodological challenges. Randomized controlled trials are the gold standard in terms of research designs to assess the causal effects of a program (Shadish et al., 2002). However, they are inadequate for a system-wide intervention that addresses multiple areas and utilizes such a wide range of interventions. Instead, to be evidence informed, we developed a rich tapestry of data gathering, analysis, and reporting mechanisms. We think our monitoring approach had an important role not only in basing the interventions on evidence but also in empowering districts and schools to use their local data with their own discretion in terms of priorities and choices of interventions, whether formal EBPs or locally developed interventions.
Although rich in data, our system cannot fully support causal claims or protect well against threats to internal validity. In fact, we found it difficult to always make connections between our interventions and the outcomes on the ground. First, so many of the interventions were conducted together that it would have been impossible to separate the multiple contributions to change. Furthermore, quite often, our role was to initiate a change and then this initial step took on a life of its own and led to other changes, such that the origin of the change could not be traced to our efforts. In fact, after a short while and given the staff turnover, many in the district and school leadership did not know of us or the original project. In our social work approach, this local ownership is a strength of our WEMA.
Nonetheless, as researchers, we need to continue to find ways to help evaluate the extent to which a project like ours has made a difference. As mentioned, a randomized controlled design would not have worked. We also found it impossible to base the study on a quasi-experimental design comparing our consortium with other districts and schools. We could not replicate the massive data collection conducted in our consortium in other locations that had not committed to be part of the project. It would also be against WEMA to try to control what other schools were doing to help make valid comparisons. We attempted post hoc comparisons using propensity score matching to identify districts that were similar in background and comparable on their CHKS data. We met with major difficulties in matching district- and school-level variables, however, and became aware of major variations among these control districts and between them and our districts and schools, which are very diverse in many characteristics.
One of the largest problems we encountered during the implementation of this project was the massive turnover of educators, administrators, district superintendents, school board members, and support staff. In some of our districts, we had 100%–200% turnover of major leadership over this time frame. Such massive turnover affects the organizational memory and priorities over time. This is an important challenge for our approach as it requires ongoing modifications to address the priorities and emphases of the district and school leadership. The impact of significant turnover should be studied further.
According to our methodological approach, we think that instead of trying to control variations (by using experimental design) in order to establish causality, we need to study naturally occurring variability and use long-term monitoring to test specific hypotheses of factors that explain this variability. Interventions may be only some of the myriad other important contextual and time factors that explain changes. Consider, for instance, the relative importance of an intervention implemented by researchers and the effects of external forces such as the current pandemic. An ongoing monitoring system to assess changes and their correlates may be the only feasible approach to being evidence informed, not only during a short research grant, but as a way of life, as part of ongoing practice.
Future research is needed to answer the many questions raised by this study. We think that WEMA offers a distinctly social work-based alternative for school safety and substance use intervention. Research shows that schools, even those in the same communities, have wide variation in school safety and substance use outcomes. Schools also do not have the personnel capacity to work on dozens of EBPs that are strict in implementation and fidelity guidelines. Many schools do not have the resources or trained personnel to sustain or grow these programs. Further, schools change over time demographically, in staff and administration, and district wide. WEMA provides a flexible process based on sound local data and an integrative process to include all the opinions, data, and voices of school constituents. It also allows for ground-up ideas that are normally not accepted in EBPs.
It is possible for future research to have “control” school districts or consortiums to test these ideas further. However, the conceptualization of a “control district or region” should be thought out very carefully. For example, districts vary in size of students, resources, culture, political affiliation of school boards, and academic outcomes. There are strong implications for districts over time regarding each of these variables. Organizational strategies also vary widely over time from district to district. In the case of this study, the interventions were so widespread, publicized (by government agencies, the White House, and congressional staff) that many similar districts began buying our guides, materials, using the websites, and visiting our consortium. In fact, many of our measurements and tools were integrated into the CHKS and used statewide for all schools. The meaning of a control district in this context needs further conceptualization and consideration.
We do think it is possible, nonetheless. More complex mixed methods, with propensity score matching and documentation of funding, grants, and the totality of school and district changes over time, may be useful. But such a study would be complex, and it’s unclear whether it would be worth the scientific and policy efforts, given how variable each school is over time.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research and/or authorship of this article: This work was supported by Defense Education Activity Educational Partnership Program Grants: HE1254-10-1-0041 and HE1254-13-1-0017.
