Effectiveness of the ViSC Social Competence Program in Turkish Elementary Schools

Abstract

This study examined whether participating in the ViSC social competence program is able to reduce different types of reported problem behaviors including bullying perpetration and victimization as well as promoting various social competencies in elementary school children. Two wave longitudinal self-report and teacher report data was available from 822 students (546 intervention group, 276 control group, 50% girls, M_age = 9.30) from five intervention (23 classes) and four control (14 classes) schools. Multilevel growth models revealed a more favorable development of the ViSC intervention group compared with the control group in three student-reported and all teacher reported variables. Participating in the program reduced conduct problems, reactive and proactive aggression and promoted school bonding, relationships with classmates, emotional regulation, and prosocial skills. However, the program was not effective regarding the reduction of bullying and victimization. Findings are discussed with regard to the implementation of future bullying prevention programs and social policies in Turkish schools.

Keywords

Bullying victimization bullying prevention elementary school-aged students teachers

Introduction

Although researchers and school psychologists stress the importance of early bullying prevention that should ideally start in kindergarten (Levine & Tamburrino, 2014), the majority of evidence-based anti-bullying programs have been implemented in secondary schools (Gaffney, Ttofi, & Farrington, 2019a, 2019b; Ttofi & Farrington, 2011). Implementing anti-bullying programs in secondary schools might be too late, because preschoolers as young as four years of age already show bullying behavior (Alsaker & Valkanover, 2012; Hanish et al., 2004) that stabilizes during the elementary school years (Boulton & Smith, 1994; Camodeca, Goossens, Terwogt, & Schuengel, 2002; Kochenderfer-Ladd, 2003). Chronic bullying perpetration and victimization has strong adverse effects on the psychosocial and academic development of children in later life (Arseneault, 2017; Kochenderfer & Ladd, 1996; Smith, Talamelli, Cowie, Naylor, & Chauhan, 2004); therefore, implementing anti-bullying programs in elementary schools is invaluable.

Bullying is a widespread problem in Turkey as 7 to 35% of Turkish children and adolescents are involved in bullying as a bully, victim, or both (Alikasifoglu et al., 2004; Doğan-Ates & Yağmurlu, 2010; Pekel-Uludagli & Uçanok, 2005; Yurtal & Cenkseven, 2016). However, there is dearth of information with regard to how to design effective anti-bullying programs in schools. Thus, it is imperative to implement existing evidence-based bullying prevention programs in Turkey and to evaluate their effectiveness. Doğan and colleagues (2017) adapted, implemented, and evaluated the Viennese Social Competence (ViSC) program, originally developed in Austria (Strohmeier, Hoffmann, Schiller, Stefanek, & Spiel, 2012), in Turkish middle schools. Building on this experience, the present research had two main aims: (1) to adapt and implement the ViSC program in elementary schools, and (2) to evaluate the effectiveness of the program by using a quasi-experimental longitudinal control group design. Extending the previous evaluation study (Doğan et al., 2017), data was collected from students and teachers and covered a large range of psychosocial competencies in addition to bullying perpetration and bullying victimization.

Bullying in Elementary School Children

Although bullying, by definition, implies a certain degree of chronicity and repetition (Olweus, 1993; Roland, 1989; Smith & Sharp, 1994), limited number of longitudinal studies on the stability of bullying in elementary school children showed that bullying stabilizes when children are between six and ten years old. Schäfer and Albrecht (2004) investigated the stability of bullying in nine to ten-year-old children over a period of three months by using self-reports and found moderate stability for both bullying and victimization. Using cut-off scores for classifications, 17% of perpetrators, 10% of victims, 35% of bully-victims, and 73% of uninvolved children had the same role at both time points. Furthermore, Kochenderfer-Ladd (2003) followed six-year-old children annually over a period of four years. According to teacher ratings, aggression was moderately stable (r = 0.58); however, based on self-report, victimization turned out to be rather unstable (r = 0.16) over a period of four years. At age 6, 17% of the children could be identified as victims and 12% of the children could be identified as perpetrators. Thirty-two percent of these victims and 28% of these perpetrators remained in their roles four years later. Similarly, Camodeca and colleagues (2002) investigated seven-year-old children over a period of one year. Based on peer nominations, children were first classified as bullies and victims using cut-off scores. Overall, bully and victim roles were moderately stable (r = 0.69 vs. 0.54) and 55% of bully-victims, 40% of bullies, and 17% of victims remained in their roles at the second of the two measurements. Moreover, Boulton and Smith (1994) followed eighth to nine-year-old children over a period of one year. In boys, both victim and bully nominations were moderately stable (r = 0.66 vs. r = 0.74). In girls, low stability for victim (r = 0.15) and moderate stability for bully (r = 0.46) nominations were found.

To summarize, all studies found at least some stability for both bullying and victimization in elementary school children indicating that these behaviours stabilize when children are six to ten years old. In the majority of studies, bullying appeared to be more stable than victimization. Furthermore, results indicate that the stability of victimization increases with age and that gender is an important variable to examine. Thus, to prevent chronic bullying and victimization, the implementation of bullying prevention programs in elementary schools is very important.

The ViSC Program

The ViSC program was designed for Austrian middle school children between the ages of 11 and 14 (Strohmeier et al., 2012; Yanagida, Strohmeier, & Spiel, 2019). Stemming from a socioecological perspective, the program views bullying prevention as a whole school task; therefore, it aims to change the behavior of individual students (i.e., bully, victim or bully/victim), students in classes, and the school environment. To create this multi-level change, the ViSC program integrates main ideas from social information processing theory (Crick & Dodge, 1996), social learning theory (Bandura, 1973), and bullying as a group phenomenon (Salmivalli, Lagerspetz, Björkqvist, Österman, & Kaukiainen, 1996).

The program simultaneously operates on the school, class, and individual level. On the school level, teachers are the main change agents because they share the responsibility to create a common understanding of bullying behaviors and implement preventive measures in schools (Schultes, Stefanek, van de Schoot, Strohmeier, & Spiel, 2014). ViSC coaches, these are trained experts, organize a series of in-school trainings for all teachers in one school. On the class level, teachers implement a 13-unit class project by using a wide variety of methods such as role play, interactive games, small group work, and whole class discussion. The class project is fully manualized and comprise single work sheets, group activities, and a summary sheet. During the class project, students are trained in several competences including recognizing their own emotions and the emotions of others, and coping with these emotions in a non-aggressive way. Furthermore, they are encouraged to feel responsible when someone in the school encounters a negative situation and they learn how to react in a way that is likely to improve the situation. Most importantly, students learn strategies how to deal with bullying incidents in the role of a bystander or a victim. On the individual level, teachers and school counselors are trained to differentiate two underlying mechanisms of aggressive behaviors (i.e., reactive and instrumental) (Card & Little, 2006; Vitaro, Brendgen, & Barker, 2006), to recognize and differentiate bullies, victims, and bully-victims; and to organize structured conversations with them when bullying occurs (Roland & Vaaland, 2006).

Evaluation Results

The ViSC program has been implemented in Austria, Cyprus, Germany, Kosovo, Romania, and Turkey during the last decade. In Austria, the program has been implemented in more than 100 middle schools with 5^th, 6^th, and 7^th graders (M_age = 11.7). Evaluation results demonstrated that the program is effective in reducing victimization, cyber-victimization, and cyberbullying (Gradinger, Yanagida, Strohmeier, & Spiel, 2015, 2016; Yanagida et al., 2019). In Cyprus, Solomontos-Kountouri and colleagues (2016) examined the effectiveness of the program in middle schools over two school years and found that the program effects differed depending on the age of the participants (M_age = 12.6). It was found that younger students (grade 7) benefited from the program more compared to older students (grade 8), especially in terms of the reduction of physical aggression and victimization. In Germany, the explicit and implicit aggression reducing effects of the program has been shown in 6^th and 8^th graders by using both qualitative and quantitative data in longitudinal studies (Gollwitzer, Banse, Eisenbach, & Naumann, 2007; Gollwitzer, Eisenbach, Atria, Strohmeier, & Banse, 2006). In Kosovo, Arënliu and colleagues (2020) developed a short (6 units) and ultra-short (4 units) version of the program and tested the effectiveness of these programs in middle schools in 7^th and 8^th grade students (M_age = 13.4). It was shown that students who participated in the ultra-short version of the program reported a significant decrease for physical victimization from pretest to posttest compared to the control group. In Romania, Trip and colleagues (2015) combined the ViSC program with a REBE (rational emotive behavioral education) program and examined the effectiveness of this combined program over one school year in 6^th grade students (M_age = 11.8). Results indicated that the program was effective in reducing dysfunctional cognitions and emotions as well as the overt expression of anger; however, there was no intervention effect on reducing victimization or bullying. In Turkey, the effectiveness of two program dosages (school and class level intervention vs. school level intervention only) was examined in middle schools in 5^th graders (M_age = 10.1) (Doğan et al., 2017). Over a period of one year, perpetration and victimization increased in the two intervention groups compared to the control group between pre- and post-test, but decreased again between post-test and follow-up indicating a sensitizing effect of the program.

To date, the ViSC program has been only implemented in middle schools including 5^th to 8^th graders. Most of these studies exclusively relied on student self-report bullying and victimization measures to examine program effectiveness. Furthermore, the existing studies did not include the perspectives of teachers and did not investigate possible changes in social-emotional developmental domains such as prosocial skills or peer relations.

As the most recent meta-analytic reviews show (Gaffney et al., 2019a; 2019b), the ViSC program is one of the few evidence-based programs developed and implemented during the last decade. Because the existing programs differ on various aspects, it is difficult to compare them with each other. However, previous studies examining the effects of the same program across different age groups showed that these programs have often more success in elementary schools compared to middle schools (Smith, 2010). For example, researchers investigated the effectiveness of KiVa program across different age groups in Finland and found that the largest effects were in elementary school (grade 1–6) and the smallest effects were in the lower secondary grade levels (grades 7–9) (Kärnä, Voeten, Little, Poskiparta, Alanen, & Salmivalli, 2011). Especially, they reported the largest effect sizes for grade 4 students.

The Present Study

ViSC program was slightly adapted to be suitable for implementation in elementary schools (see below for details). The program was implemented by graduate students in 23 classes located in five schools in the period of two months. Data were collected by other research assistants from students and teachers before and after the program implementation in these five intervention schools and in four different schools who served as the untreated control group. Since the ViSC program aims to modify a broad spectrum of social emotional competences (e.g., empathy, modeling positive behavior for peers), data were collected not only on bullying and victimization, but also on other variables such as emotional problems, school attachment, and peer relations (from students), as well as emotion regulation and prosocial skills (from teachers). Using multi-informants including teachers – instead of just relying on self-report measures – has often been suggested in previous evaluation studies, but only rarely realized. Teachers have a crucial role in observing the peer ecology and recognizing bullying incidents in their classrooms since they spent a lot of time with the students (Gest & Rodkin, 2011; Pepler, 2006). Therefore, teacher reports are considered to be ecologically valid when examining students’ aggressive behaviors and conduct problems (O’Brennan, Bradshaw, & Furlong, 2014).

The following two main hypotheses were tested in the present study.

Hypothesis 1: We expected that the program would be effective in reducing different types of student reported problem behaviors as well as promoting different social competencies in the intervention group compared to the control group. Program effectiveness is indicated (a) by a steeper decrease in bullying and victimization as well as conduct problems, emotional symptoms, and peer problems when comparing the intervention group to the control group, and (b) by a steeper increase in peer relations, school attachment, attachment to teachers as well as prosocial behaviors when comparing the intervention group to the control group.

Hypothesis 2: We expected that the program would be effective in reducing different types of teacher reported problem behaviors as well as promoting different social competencies the intervention group compared to the control group. Program effectiveness is indicated (a) by a steeper decrease in reactive and proactive aggression when comparing the intervention group to the control group, and (b) by a steeper increase in emotional regulation and prosocial skills when comparing the intervention group to the control group.

Method

Developing an Elementary School Program Version

We already culturally adapted and implemented the ViSC program in Turkey in middle schools (for details see Doğan et al., 2017). This adaptation was based on the guidelines suggested for the deep structure and surface structure of culturally relevant interventions (Castro, Barrera, & Martinez, 2004; Nocentini & Menesini, 2016; Resnikow, Soler, Braithwait, Ahluwalia, & Butler, 2000). During this adaptation, deep structure (e.g., theory, core values, significant aspects) of the ViSC program remained unchanged; however, minor modifications involving the surface structure (e.g., language, names in the vignettes) were made. For the present study, it was necessary to adapt the ViSC program to be able to implement in Turkish elementary schools. These adaptations were guided primarily by available resources to overcome several practical obstacles (for more details see Strohmeier, Solomontos-Kountouri, Trip, Doğan, & Arënliu, 2020). For this version, we made no changes on the surface structure of the ViSC middle school version; however, modifications were made on the deep structure in the implementation model while keeping the core theory and values of the program intact. For example, instead of training class teachers who would then implement the program on the school, class and individual level, we trained graduate psychology students as ViSC coaches. We developed a shortened version of the class project consisting of seven units and then the class project was implemented by these trained graduate students in the classroom. We only implemented the class level of the program in order not to overburden the schools and graduate students. A similar approach has been realized in Kosovo (for more details see Arënliu et al., 2020). The original ViSC class project consists of 13 units of which during unit 1–8 students are trained on social competences to prevent bullying; unit 9 focuses on the decision of the common activity as a class (e.g., theatre, puppet show, making a film), and during units 10–13 all students in the class work together to complete this common activity (Strohmeier et al., 2012). In this elementary school version, the common activity of the class project was removed and the content of the first eight units were condensed without removing any topic; therefore, the new version of the class project consists of seven units (see Table 1). Because the target group of the implementation were 4^th graders, the original materials were developmentally appropriate and it was not necessary to change them.

Table 1

ViSC Class Project – Elementary School Version

Content
Unit 1	What is the class project and why are we participating in?
	Why rules are important in our lives and what rules do we want in our class?
Unit 2	How can we recognize critical social situations and what can we do to help improve the situation?
Unit 3	How can we recognize the emotions of others and what can we do to help them feel better?
Unit 4	How can we recognize our own emotions and what can we do to cope with them to feel better?
Unit 5-6	What can we do if we are treated in a mean and unfair way by others? What is the best thing to do in such situations, and why?
Unit 7	What can we do if we don’t understand the behavior of our classmates who are from a different background? What is the best thing to do in such situations and why?

Program Implementation

The implementation process of the program in Turkey was described in detail previously (Doğan et al., 2017). After obtaining official approvals from the Ethics Committee and local branch of the Ministry of Education, the researchers visited schools to meet principals and counselors for discussing the implementation process. Teachers were given a brief information about the study and informed consent was obtained. Active parent and child consent forms were also received. Graduate psychology students received an intensive training and they had weekly supervision meetings with the first author; thus, the program was implemented with high fidelity. The program implementation started two to three months after the beginning of the school year when students were already familiar with each other. The class project was implemented in a weekly interval during the same two-class period (a total of 90 minutes). Given the high number of students in some classrooms and using various methods (i.e., role play, small group work) in the program, four to five undergraduate research assistants served as helpers during the program implementation in each classroom.

Study Design

We used a quasi-experimental longitudinal control group design with two measurement points. The ViSC program was implemented in five intervention schools (23 classes) and four control schools (14 classes). Pretest (Time 1: T1) data was collected one week before, and post-test (Time 2: T2) data was collected within two to three weeks after the program implementation in both intervention and control schools. Students completed paper and pencil surveys that lasted about 25–35 minutes in their classrooms during regular class hours. The class teachers completed the questionnaire out of the class hours in the teachers’ room in schools.

Participants

The sample consists of 822 4^th grade students (50% girls) with a mean age of 9.30 years (SD = 0.70) who participated in the two waves of measurement. In addition, 37 class teachers (78% women) with a mean age of 45.4 years (SD = 6.6) rated the behaviors of 842 students. Twenty students were absent either in T1 or T2 of data collection even though they participated in the program; thus, these students were excluded from the data analysis. When we compared the intervention and control group in terms of the demographic characteristics, we found no statistically significant differences between the groups (See Table 2).

Table 2

Sample Description at Pretest

	Intervention group (n = 546)	Control group (n = 276)
	M (SD)	M (SD)
Age	9.46 (0.75)	8.99 (0.42)
	%	%
Gender (female)	50.2	50.7
Married parents	93.0	91.3
Perceived financial situation
Very bad or bad	3.9	1.1
Neither bad nor good	14.3	12.7
Good	28.1	35.3
Very good	53.7	50.9
Parent’s working status
Working father	94.9	96.7
Working mother	26.9	32.6
Father’s education level
Elementary school or less	34.8	33.0
Middle school	26.5	22.8
High school or higher	38.6	44.2
Mother’s education level
Elementary school or less	43.0	39.4
Middle school	26.4	23.0
High school or higher	30.6	37.6

Note. There were no statistically significant differences between groups.

Child Measurements

Demographic information. Children reported their age and gender as well as the educational level, marital status, and occupation of their parents.

Bullying perpetration. Eleven validated items covering physical, relational, and verbal aspects of bullying perpetration were used (e.g., ‘How often did you hit one or more classmates?’). Cronbach’s α coefficients were.90/.91 (T1/T2). Exactly the same items were used in all ViSC evaluation studies (for more details see Doğan et al., 2017; Yanagida et al., 2019).

Bullying victimization. Eleven validated items covering physical, relational, and verbal aspects of bullying victimization were used (e.g., ‘How often did other classmates make fun of you?’). Cronbach’s α coefficients were.91/.92 (T1/T2) (for more details see Doğan et al., 2017; Yanagida et al., 2019).

Strengths and Difficulties Questionnaire (SDQ). The SDQ is a brief screening instrument for assessing children’s positive and negative aspects of behavior (Goodman, 1997). In this study, four subscales of the SDQ (i.e., emotional symptoms, peer problems, conduct problems, and prosocial behavior) were used. Cronbach’s α coefficients were.66,.41,.48,.85/.71,.46,.45,.84, respectively (T1/T2). Children responded on a four-point Likert scale (0 = strongly disagree; 3 = strongly agree).

School Attachment Scale (SAS). This measure was developed by Hill and Werner (2006) to measure children and adolescents’ attachment to school and validated in Turkish by Savi (2011). The scale consists of three subscales: attachment to school (e.g., ‘I like my school.’), attachment to classmates (e.g., ‘My friends care about me.’), and attachment to teachers (e.g., ‘I like my teachers.’). Cronbach’s α coefficients were.90,.82,.75/.90,.86,.79, respectively (T1/T2).

Peer relations. This measure was developed by Kaner (2000) to measure peer relations in schools. In this study, the subscale ‘friendship loyalty (attachment)’ consisting of 8 items was used (e.g., ‘My friends like me.’). Cronbach’s α coefficients were.92/.93 (T1/T2).

Teacher Measurements

Demographic information. Teachers were asked to indicate their gender and age.

Reactive and instrumental aggression. This scale was developed by Dodge and Coie (1987) and previously validated in Turkish (Akgün, Araz, & Karadağ, 2007). Three items measure reactive aggression (e.g., ‘When this child is bullied or threatened, he or she gets angry easily.’) and three items measure instrumental aggression (e.g., ‘This child uses physical force in order to dominate other kids.’). Teachers responded on a five-point Likert scale (1 = never; 5 = always). Cronbach’s α coefficients for reactive aggression were.96/.97 (T1/T2) and for proactive aggression were.95/.96 (T1/T2).

Emotion regulation and prosocial skills. This measure was developed by the Conduct Problem Prevention Research Group (2004) and previously validated in Turkish (Akgün & Araz, 2014). Ten items measure emotion regulation (e.g., ‘This child expresses needs and feelings appropriately.’) and nine items measure prosocial skills (e.g., ‘This child helps others.’). Teachers responded on a five-point Likert scale (1 = not at all, 5 = very much). Cronbach’s α coefficients for prosocial skills were.97/.98 (T1/T2) and emotional regulation skills were.96/.97 (T1/T2).

Statistical Analyses

Multilevel growth modelling (level 1: time, level 2: student, level 3: class) was conducted with SPSS Version 26 to test program effectiveness. Maximum likelihood was used as estimation procedure. This analysis adequately considers the nested data structure, time is nested in students, and students are nested in classes taking into account the dependencies between observations (i.e., design effect, see Snijders & Bosker, 1999). The data met all necessary conditions for all the analyses that we conducted.

We investigated whether the control group differs from the intervention group in terms of different outcome variables. Program effectiveness was investigated based on the cross-level interactions Time × Control vs . Intervention. We also computed standardized estimates for each effect to be able to estimate effect sizes. In order to compute these standardized estimates, we re-run all analyses with standardized dependent variables.

Results

Descriptive Statistics for Outcome Variables

As a first step, the means and standard deviations of all variables for intervention and control group are reported by two waves of data collection (T1 and T2) in Table 3.

Table 3

Descriptive Statistics for the Outcome Variables: Means and Standard Deviations

	Intervention group		Control group
	Time 1	Time 2	Time 1	Time 2
	(n = 546)	(n = 546)	(n = 276)	(n = 276)
Child measures
Bullying
Perpetration	0.80 (0.83)	0.81 (0.83)	0.84 (0.83)	0.84 (0.92)
Victimization	0.44 (0.63)	0.43 (0.61)	0.42 (0.57)	0.46 (0.68)
SDQ
Emotional symptoms	1.06 (0.67)	1.03 (0.74)	1.09 (0.70)	1.03 (0.76)
Conduct problems	0.79 (0.80)	0.71 (0.79)	0.64 (0.73)	0.71 (0.77)
Peer problems	0.90 (0.54)	0.92 (0.59)	0.89 (0.54)	0.88 (0.58)
Prosocial behavior	2.42 (0.66)	2.40 (0.74)	2.42 (0.74)	2.41 (0.69)
School attachment
School	4.53 (0.91)	4.52 (0.90)	4.52 (0.90)	4.31 (1.10)
Classmates	4.30 (0.89)	4.40 (0.88)	4.39 (0.79)	4.31 (0.97)
Teachers	4.61 (0.70)	4.64 (0.69)	4.60 (0.66)	4.55 (0.74)
Peer relations	3.86 (1.04)	3.99 (1.04)	4.07 (0.93)	4.07 (1.04)
Teacher measures
Reactive aggression	2.51 (1.25)	2.30 (1.23)	2.59 (1.20)	2.70 (1.17)
Instrumental aggression	1.74 (1.06)	1.66 (1.06)	1.87 (1.11)	1.91 (1.16)
Emotional regulation	3.67 (0.91)	3.85 (0.96)	3.53 (0.88)	3.50 (0.86)
Prosocial skills	3.85 (0.92)	3.96 (0.98)	3.78 (0.92)	3.71 (0.90)

Note. Sample size slightly differs by scale and wave due to missing values.

Baseline Effects

As shown in Table 4, the ViSC intervention group differed from the control group regarding student-reported conduct problems and peer relations. There were no baseline effects for teacher-reported variables. Boys and girls differed in all outcome variables.

Table 4

Multilevel Growth Curve Model Results for Child Measures: Comparison of the Control and the Intervention Group

Predictor	Bullying		SDQ				SAS-school	SAS-class-mates	SAS-teachers	Peer relations
	Aggression	Victimization	ES	CP	PP	PB
Level 1: Measurement: Unstandardized estimates
Intercept	0.99^*** (0.09)	0.55^*** (0.05)	1.05^*** (0.05)	0.82^*** (0.05)	0.95^*** (0.04)	2.32^*** (0.05)	4.40^*** (0.09)	4.32^*** (0.08)	4.52^*** (0.06)	3.90^*** (0.08)
Time	– 0.01 (0.05)	0.04 (0.04)	– 0.07 (0.05)	0.06 (0.05)	0.00 (0.04)	– 0.00 (0.05)	– 0.21^*** (0.05)	– 0.07 (0.05)	– 0.06 (0.05)	0.01 (0.05)
Gender	– 0.29^*** (0.05)	– 0.26^*** (0.04)	0.08^* (0.04)	– 0.34^*** (0.04)	– 0.12^*** (0.03)	0.19^*** (0.04)	0.23^*** (0.06)	0.13^* (0.05)	0.19^*** (0.04)	0.31^*** (0.06)
Level 3: Class: Unstandardized estimates
Baseline effects
Control vs. Intervention	– 0.03 (0.10)	0.03 (0.05)	– 0.03 (0.06)	0.15^* (0.06)	0.02 (0.05)	0.01 (0.06)	0.00 (0.11)	– 0.09 (0.09)	– 0.01 (0.74)	– 0.20^* (0.10)
Intervention effects
Time×Control vs. Intervention	0.02 (0.06)	– 0.06 (0.04)	0.04 (0.06)	– 0.14^* (0.07)	0.03 (0.04)	– 0.02 (0.06)	0.20^** (0.07)	0.17^** (0.06)	0.09 (0.06)	0.12 (0.07)
Level 3: Class: Standardized estimates
Baseline effects
Control vs. Intervention	– 0.04	0.04	– 0.05	0.19	0.04	0.01	0.00	– 0.11	– 0.02	– 0.19
Intervention effects
Time×Control vs. Intervention	0.02	– 0.10	0.06	– 0.18	0.05	– 0.03	0.21	0.20	0.13	0.12

Note. Unstandardized and standardized estimates (Standard error); Gender coded as 0 = male, 1 = female; ES = Emotional Symptoms; CP = Conduct Problems; PP = Peer Problems; PB = Prosocial Behavior; SAS = School Attachment Scale; ^*p < .05; ^**p < .01; ^***p < .001.

Intervention Effects

As shown in Table 5, the overall pattern of results revealed a more favorable development of the ViSC intervention group compared to the control group.

Table 5

Multilevel Growth Curve Model Results for Teacher Measures: Comparison of the Control and the Intervention Group

Predictor	Reactive aggression	Instrumental aggression	Emotional regulation	Prosocial skills
Level 1: Measurement: Unstandardized estimates
Intercept	3.08 ^*** (0.15)	2.22^*** (0.14)	3.25^*** (0.10)	3.53^*** (0.12)
Time	0.11^* (0.04)	0.04 (0.04)	– 0.04 (0.03)	– 0.07^* (0.03)
Gender	– 0.86^*** (0.07)	– 0.62^*** (0.06)	0.55^*** (0.05)	0.50^*** (0.05)
Level 3: Class: Unstandardized estimates
Baseline effects
Control vs. Intervention	– 0.12 (0.19)	– 0.16 (0.17)	0.14 (0.12)	0.08 (0.15)
Intervention effects
Time×Control vs. Intervention	– 0.32^*** (0.05)	– 0.13^* (0.05)	0.22^*** (0.04)	0.18^*** (0.04)
Level 3: Class: Standardized estimates
Baseline effects
Control vs. Intervention	– 0.10	– 0.14	0.16	0.08
Intervention effects
Time×Control vs. Intervention	– 0.26	– 0.12	0.24	0.20

Note. Unstandardized and standardized estimates (Standard error); Gender coded as 0 = male, 1 = female; ^*p < .05; ^**p < .01; ^***p < .001.

Bullying perpetration and victimization. No intervention effect was found.

SDQ. Regarding conduct problems, there was a steeper decrease in the ViSC intervention group compared to the control group (Time×Control vs. Intervention, b= 0.15, p< .01). No other intervention effects were found.

School attachment. There was a steeper increase in the ViSC intervention group compared with the control group for attachment to school (Time × Control vs . Intervention, b = 0.20, p < .01), for attachment to classmates (Time × Control vs . Intervention, b = 0.17, p < .01), and for attachment to teachers (Time × Control vs . Intervention, b = 0.09, p = .094).

Peer relations. There was a steeper increase in the ViSC intervention group compared to the control group that was significant when tested one-sided (Time × Control vs . Intervention, b = 0.12, p = .07).

Reactive and instrumental aggression. There was an increase in the control group, while the intervention group had a decrease for reactive aggression (Time × Control vs . Intervention, b = -0.32, p < .001). Similarly, there was no change in the control group, while the intervention group had a decrease for instrumental aggression (Time × Control vs . Intervention, b = -0.13, p < .05).

Emotion regulation and prosocial skills. There was no change in the control group, while intervention group increased for emotion regulation (Time × Control vs . Intervention, b = 0.22, p < .001). Similarly, there was no change in the control group, while intervention group increased for prosocial skills (Time × Control vs . Intervention, b = 0.18, p < .001).

Discussion

Longitudinal studies on the stability of bullying in elementary school children indicate that a substantial number of children are trapped in stable bullying roles (e.g. Kochenderfer-Ladd, 2003; Schäfer & Albrecht, 2004). Thus, the implementation of bullying prevention programs in elementary schools is invaluable to prevent chronic bullying and victimization. The present research was conducted in Turkey, a country in which knowledge on how to design effective anti-bullying programs in schools is scarce. Building on the experience of the previous implementation of the ViSC program in Turkish middle schools (Doğan et al., 2017), an elementary school version of the ViSC program was developed and implemented in five elementary schools (23 classes) for the present study. The effectiveness of the program was examined within a quasi-experimental longitudinal control group design and a large range of student and teacher-rated problem behaviors and psychosocial competencies were measured in addition to bullying perpetration and bullying victimization. Although it was necessary to shorten the original program, multilevel growth curve analyses revealed that the elementary school version of the program was effective in reducing several aspects of problem behaviors and in promoting social competencies among 4^th graders.

Since the ViSC program focuses on social emotional competences (e.g., empathy, modeling positive behavior for peers), we collected data not only on bullying and victimization, but also on other variables such as emotional problems, prosocial behavior, and peer relations (from students) and emotion regulation, prosocial behavior (from teachers). Moreover, we also wanted to study potential ‘side-effects’ (i.e., unintended but beneficial outcomes) (Salmivalli, Garandeau, & Veenstra, 2012) of the ViSC program. Several studies of the KiVa program showed side-effects such as reducing students’ internalizing symptoms (anxiety and depression), improving peer-group perceptions (Williford et al., 2012), and increasing academic motivation and academic performance (Salmivalli et al., 2012).

To overcome a number of practical obstacles, trained graduate students implemented a shortened version of the class project on a weekly basis over a period of two months, instead of teachers implementing the multi-level whole-school program. By implementing the class level component only, essential program components have not been utilized and core elements of the original program have not been included. From a socio-ecological anti-bullying perspective, implementing class units by external assistants and not delivering teacher trainings is not an optimal strategy. However, in developing countries like Turkey, schools have to deal with a multitude of structural obstacles and the system is often not ready to implement a whole school anti-bullying program (Sivaraman, Nye, & Bowes, 2019; Strohmeier et al., 2020).

System readiness is related to many factors like educational policy, money spent on education, high quality teacher education, and adequate school infrastructure. Thus, in constrained settings it is unrealistic to implement whole school programs because of structural obstacles, even if evidence shows that multi-level programs are the most effective in reducing school bullying (Gaffney et al., 2019b). Therefore, it is necessary to develop different implementation models that can be potentially sustained even with no or limited resources. Training graduate psychology students instead of training teachers is such a low cost model, especially when the implementation of the anti-bullying program can be added as a requirement in their graduate education curriculum (Doğan, 2019). Another model would be delivering materials to teachers when trainings are not possible or using existing community organizations to help implementing anti-bullying measures in schools. Future studies should investigate the feasibility of such implementation models and ideally compare different implementation models to find the most effective model in the case of no or very limited resources.

The present study showed that the program was not effective in changing bullying perpetration and bullying victimization. Although the class component of the ViSC program includes a number of exercises to empower the peer group to recognize and intervene in bullying situations, two-months implementation period might have been too short to cause a sustainable change and to reduce bullying rates. Furthermore, the status and expertise of the volunteer students might have played a role. It is possible that graduate students might not have the same authority as teachers to change dysfunctional class norms. Moreover, external trainers with more practical experience with working students might have been more effective in bringing about positive change. Another relevant factor might be the management of ongoing bulling situations. Unfortunately, indicated actions (e.g., talks with bullies, victims, and bully-victims) were not implemented. It is reasonable to assume that the class project is not a strong enough measure to deal with an actual bullying case. Very similar results were found when the ViSC class project was implemented in Romania (Trip et al., 2015) and Kosovo (Arënliu et al., 2020). Although it might be more resource-intensive, future implementations of the ViSC program could benefit from a wider social and administrative support by focusing on the whole-school approach. Future efforts should strive to get principals of the schools, teachers, other school staff including school psychologists, and parents on board for the implementation of the bullying intervention.

Limitations

The evaluation of the ViSC program in elementary schools in Turkey provides important insights despite several limitations that should be noted. First, we were not able to collect follow-up data. However, the importance of follow-up data has been shown in previous ViSC evaluation studies. For example, Gollwitzer and colleagues (2007) found that effects were present at follow-up rather than post-test because the ViSC program foster social competences which need some time to transfer and become observable in a behavior. Therefore, it is suggested that future studies should include follow-up data to detect transfer effect rather than immediate training effects. We only collected data at pre-test and post-test, because at the time of an ideal follow-up measure (4 to 6 months after T2) our 4^th graders would already have started a new middle school (grade 5). Thus, it would have been very difficult to relocate them in their new schools. Because the importance of collecting follow-up data has been shown in previous ViSC evaluation studies (Solomontos-Kountouri et al., 2016), the present research design worked against possible program effects. Future studies should include 3rd graders, because they can be reexamined when they are in the 4^th grade. Moreover, several researchers emphasized the value of using multi-informant and multi-method strategies when examining the effectiveness of anti-bullying programs in schools. Even though we collected data from children and teachers, future studies should also include peer nominations as well as observational assessments.

Despite these limitations, this study adds to the current literature by showing that different types of bullying and victimization are observed among elementary school-aged students in Turkey. Based on the previous literature (Smith, Salmivalli, & Cowie, 2012) and our results, we emphasize the importance of implementing evidence-based bullying prevention programs in elementary schools and communicating with school officials about effective bullying policies. Thus, we would be able to intervene and protect children against short and long-term negative effects of bullying as early as possible.

Bio Sketches

Aysun Doğan is full professor of developmental pychology and the director of the Child and Adolescent Research Lab at Ege University, Turkey. She ran numerous national and international research projects focusing on friendship relations of young adults, immigrant children’s psychological well-being, and adolescent-parent relations. Her current research interests include bullying, victimization, friendships, immigration, and preventive intervention programs.

Dagmar Strohmeier is full professor at the University of Applied Sciences Upper Austria, Austria and professor II at the University of Stavanger, Norway. Her research area is on peer relations with a cross-cultural and cross-national perspective, especially among immigrant youths. She has developed and implemented the ViSC program to foster social competencies in Austrian, Romanian, Cypriot, Turkish, and Kosovar schools.

Rukiye Kızıltepe is a Ph.D. student and research assistant at the psychology department at Ege University, Izmir, Turkey. Her research interests include preventive intervention programs, intergenerational transmission of child maltreatment, and victimization in different contexts such as school, home, and neighborhood.

Durdane Gümüşten is an assistant professor at the psychology department at Kahramanmaraş İstiklal University, Kahramanmaraş, Turkey. Her research focuses on parent-adolescent relationships and psychosocial adjustment during adolescence.

Takuya Yanagida received his Ph.D. in psychology at the University of Vienna, Austria in 2017. He holds a position as senior research associate at the Faculty of Psychology, University of Vienna. His interests are missing data analyses, longitudinal data, psychometrics, and evaluation.

Footnotes

Acknowledgments

We would like to thank our ViSC project team including our graduate and undergraduate students at EGE University for their invaluable work during the implementation of the project. We are also grateful to the school principals, teachers, and students who participated in this study.

References

Akgün,

, & Araz,

(2014). The effects of conflict resolution education on conflict resolution skills, social competence, and aggression in Turkish elementary school students. Journal of Peace Education11, 30–45. doi: 10.1080/17400201.2013.777898

Akgün,

, Araz,

, & Karadağ,

(2007). We can resolve our conflicts”: A conflict resolution education program and its psycho-social effects. Turkish Journal of Psychology22, 43–62.

Alikasifoglu,

, Erginoz,

, Ercan,

, Uysal,

, Kaymak,

D. A.

, & Ilter,

(2004). Violent behaviour among Turkish high school students and correlates of physical fighting. The European Journal of Public Health14, 173–177. doi: 10.1093/eurpub/14.2.173

Alsaker,

F. D.

, & Valkanover,

(2012). The Bernese program against victimization in kindergarten and elementary school. New Directions for Youth Development133, 15–28. doi: 10.1002/yd.20004.

Arënliu,

, Strohmeier,

, Konjufca,

, Yanagida,

, & Burger,

(2020). Empowering the peer group to prevent school bullying in Kosovo: Effectiveness of a short and ultra-short version of the ViSC social competence program. International Journal of Bullying Prevention2, 65–78. doi: 10.1007/s42380-019-00052-4

Arseneault,

(2017). The long-term impact of bullying victimization on mental health. World Psychiatry16, 27–28. doi: 10.1002/wps.20399

Bandura,

(1973). Aggression: A social learning analysis. Englewood Cliffs, NJ: Prentice Hall.

Boulton,

M. J.

, & Smith,

P. K.

(1994). Bully/victim problems in middle-school children: Stability, self-perceived competence, peer perceptions and peer acceptance. British Journal of Developmental Psychology12(3), 315–329. doi: 10.1111/j.2044-835X.1994.tb00637.x

Camodeca,

, Goossens,

F. A.

, Terwogt,

M. M.

, & Schuengel,

(2002). Bullying and victimization among school-age children: Stability and links to proactive and reactive aggression. Social Development11, 332–345. doi: 10.1111/1467-9507.00203

10.

Card,

N. A.

, & Little,

T. D.

(2006). Proactive and reactive aggression in childhood and adolescence: A meta-analysis of differential relations with psychosocial adjustment. International Journal of Behavioral Development30, 466–480. doi: 10.1177/0165025406071904

11.

Castro,

F. G.

, Barrera,

, & Martinez,

C. R.

(2004). The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit. Prevention Science5, 41–45. doi: 10.1023/B:PREV.0000013980.12412.cd

12.

Conduct Problem Prevention Research Group (2004). A social competence scale-teacher version. Fast Track project technical report. Retrieved October 10, 2019, from https://fasttrackproject.org/techrept/s/sct/

13.

Crick,

N. R.

, & Dodge,

K. A.

(1996). Social information-processing mechanisms in reactive and proactive aggression. Child Development67, 993–1002. doi: 10.1111/j.1467-8624.1996.tb01778.x

14.

Dodge,

K. A.

, & Coie,

J. D.

(1987). Social-information processing factors in reactive and proactive aggression in children’s peer groups. Journal of Personality and Social Psychology53, 1146–1158. doi: 10.1037//0022-3514.53.6.1146

15.

Doğan

, (2019, November). Promoting inclusion and preventing bullying in Turkish schools. Keynote speech presented at the 24^th Workshop on Aggression, Stavanger, Norway.

16.

Doğan,

, Keser,

, Şen,

, Yanagida,

, Gradinger,

, & Strohmeier,

(2017). Evidence based bullying prevention in Turkey: Implementation of the ViSC social competence program. International Journal of Developmental Science11, 93–108. doi: 10.3233/DEV-170223

17.

Doğan-Ates,

, & Yağmurlu,

(2010). Examining victimization in Turkish schools. European Journal of Educational Studies2, 31–37.

18.

Gaffney

Ttofi

M.M.

, & Farrington

D.P.

(2019a). Evaluating the effectiveness of school-bullying prevention programs: An updated meta-analytical review. Aggression and Violent Behavior45, 111–133. doi: 10.1016/j.avb.2018.07.001

19.

Gaffney,

, Ttofi,

M. M.

, & Farrington,

D. P.

, (2019b). Examining the effectiveness of school-bullying intervention programs globally: A meta-analysis. International Journal of Bullying Prevention1, 14–31. doi: 10.1007/s42380-019-0007-4

20.

Gest,

S. D.

, & Rodkin,

P. C.

(2011). Teaching practices and elementary classroom peer ecologies. Journal of Applied Developmental Psychology, 32, 288-296. doi: 10.1016/j.appdev.2011.02.004

21.

Gollwitzer,

, Banse,

, Eisenbach,

, & Naumann,

(2007). Effectiveness of the Vienna social competence training on implicit and explicit aggression. Evidence from an Aggressiveness IAT. European Journal of Psychological Assessment23, 150–156. doi: 10.1027/1015-5759.23.3.150.

22.

Gollwitzer,

, Eisenbach,

, Atria,

, Strohmeier,

, & Banse,

(2006). Evaluation of aggression-reducing effects of the “Viennese social competence training”. Swiss Journal of Psychology65, 125–135. doi: 10.1024/1421-0185.65.2.125

23.

Goodman,

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry38, 581–586. doi: 10.1111/j.1469-7610.1997.tb01545.x

24.

Gradinger,

, Yanagida,

, Strohmeier,

, & Spiel,

(2015). Prevention of cyberbullying and cyber victimization: Evaluation of the ViSC social competence program. Journal of School Violence14, 87–110. doi: 10.1080/15388220.2014.963231

25.

Gradinger,

, Yanagida,

, Strohmeier,

, & Spiel,

(2016). Effectiveness and sustainability of the ViSC social competence program to prevent cyberbullying and cyber-victimization: Class and individual level moderators. Aggressive Behavior42, 181–193. doi: 10.1002/ab.21631

26.

Hanish,

L. D.

, Eisenberg,

, Fabes,

R. A.

, Spinrad,

T. L.

, Ryan,

, & Schmidt,

(2004). The expression and regulation of negative emotions: Risk factors for young children’s peer victimization. Development and Psychopathology16, 335–353. doi: 10.10170S0954579404044542

27.

Hill,

, & Werner,

(2006). Affiliative motivation, school attachment, and aggression in school. Psychology in the Schools43, 231–246. doi: 10.1002/pits.20140

28.

Kaner,

(2000). Akran ilişkileri ölçeği ve akran sapması ölçeği geliştirme çalışması [The development of peer relations and peer deviance measure]. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Dergisi33, 77–89. doi: 10.1501/Egifak_0000000024

29.

Kärnä,

, Voeten,

, Little,

T. D.

, Poskiparta,

, Alanen,

, & Salmivalli,

(2011). Going to scale: A nonrandomized nationwide trial of the KiVa antibullying program for grades. Journal of Consulting and Clinical Psychology79, 796–805. doi: 10.1037/a0025740

30.

Kochenderfer,

B. J.

, & Ladd,

G. W.

(1996). Peer victimization: Manifestations and relations to school adjustment in kindergarten. Journal of School Psychology34, 267–283. doi: 10.1016/0022-4405(96)00015-5

31.

Kochenderfer-Ladd,

(2003). Identification of aggressive and asocial victims and the stability of their peer victimization. Merrill-Palmer Quarterly49, 401–425. doi: 10.1353/mpq.2003.0022

32.

Levine,

, & Tamburrino,

(2014). Bullying among young children: Strategies for prevention. Early Childhood Education Journal42, 271–278. doi: 10.1007/s10643-013-0600-y

33.

Nocentini,

, & Menesini,

(2016). KiVa anti-bullying program in Italy: Evidence of effectiveness in a randomized control trial. Prevention Science17, 1012–1023. doi: 10.1007/s11121-016-0690-z

34.

Olweus,

(1993). Bullying at school: What we know and what we can do. Oxford, UK: Blackwell.

35.

O’Brennan,

L. M.

, Bradshaw,

C. P.

, & Furlong,

M. J.

(2014). Influence of classroom and school climate on teacher perceptions of student problem behavior. School Mental Health6, 125–136. doi: 10.1007/s12310-014-9118-8

36.

Pekel-Uludagli,

, & Uçanok,

(2005). Loneliness, academic achievement and types of bullying behavior according to sociometric status in bully/victim groups. Turkish Journal of Psychology20, 77–95.

37.

Pepler,

D. J.

(2006). Bullying interventions: A binocular perspective. Journal of the Canadian Academy of Child and Adolescent Psychiatry15(1), 16–20.

38.

Resnikow,

, Soler,

, Braithwait,

R. L.

, Ahluwalia,

J. S.

, & Butler,

(2000). Cultural sensitivity in substance abuse prevention. Journal of Community Psychology28, 271–290. doi: 10.1002/(sici)1520-6629(200005)28:3<271::aid-jcop4>3.0.co;2-i

39.

Roland,

(1989). A system oriented strategy against bullying. In E. Roland & E. Munthe (Eds.), Bullying: An international perspective. London: David Fulton.

40.

Roland,

, & Vaaland,

(2006). ZERO teacher's guide to the Zero Anti-Bullying Programme. Stavanger: Centre for Behavioural Research, University of Stavanger.

41.

Salmivalli,

, Garandeau,

C. F.

, & Veenstra,

(2012). KiVa anti-bullying program: Implications for school adjustment. In A. M. Ryan & G. W. Ladd (Eds.), Adolescence and education. Peer relationships and adjustment at school (pp. 279–305). Charlotte, NC: IAP Information Age Publishing

42.

Salmivalli,

, Lagerspetz,

K. M.

, Björkqvist,

, Österman,

, & Kaukiainen,

(1996). Bullying as a group process: Participant roles and their relations to social status within the grou. Aggressive Behavior22, 1–15. doi: 10.1002/(sici)1098-2337(1996)22:1<1::aid-ab1>3.0.co;2-t

43.

Savi,

(2011). School attachment scale for children and adolescents: The study of validity and reliability. Elementary Education Online10, 80–90.

44.

Schäfer,

, & Albrecht,

(2004). Wie du mir, so ich dir?” Prävalenz und Stabilität von Bullying in Grundschulklassen [“Tit for Tat?” Prevalence and stability of bullying in primary school classes]. Psychologie in Erziehung und Unterricht51, 136–150.

45.

Schultes,

M.T.

, Stefanek,

, van de Schoot,

, Strohmeier,

, & Spiel,

(2014). Measuring implementation of a school-based violence prevention program on two levels: Fidelity and teachers’ responsiveness as predictors of proximal outcomes. Zeitschrift für Psychologie/Journal of Psychology222, 49–57. doi: 10.1027/2151 2604/a000165

46.

Sivaraman,

, Nye,

, & Bowes,

(2019). School-based anti-bullying interventions for adolescents in low- and middle-income countries: A systematic review. Aggression and Violent Behavior45, 154–162. doi: 10.1016/j.avb.2018.07.007

47.

Smith,

P. K.

(2010). Bullying in primary and secondary schools: Psychological and organizational comparisons. In S. R. Jimerson, S. M. Swearer, & D. L. Espelage (Eds.), Handbook of bullying in schools: An international perspective (pp. 137–150). New York: Routledge.

48.

Smith,

P. K.

, Salmivalli,

, & Cowie,

(2012). Effectiveness of school-based programs to reduce bullying: A commentary. Journal of Experimental Criminology8, 433–441. doi: 10.1007/s11292-012-9142-3

49.

Smith,

P. K.

, & Sharp,

(1994). School bullying: Insights and perspectives. London: Routledge.

50.

Smith,

P. K.

, Talamelli,

, Cowie,

, Naylor,

, & Chauhan,

(2004). Profiles of non-victims, escaped victims, continuing victims and new victims of school bullying. British Journal of Educational Psychology74, 565–581. doi: 10.1348/0007099042376427

51.

Snijders,

T. A. B.

, & Bosker,

T. J.

(1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London, UK: SAGE.

52.

Solomontos-Kountouri,

, Gradinger,

, Yanagida,

, & Strohmeier,

(2016). The implementation and evaluation of the ViSC program in Cyprus: Challenges of cross national dissemination and evaluation results. European Journal of Developmental Psychology13, 737–755. doi: 10.1080/17405629.2015.1136618

53.

Strohmeier,

, Hoffmann,

, Schiller

E. M.

, Stefanek,

, & Spiel,

(2012). ViSC social competence program. New Directions for Youth Development133, 71–84. doi: 10.1002/yd.20008

54.

Strohmeier,

, Solomontos-Kountouri,

, Trip,

, Doğan,

, & Arënliu,

, (in press). International implementation of the ViSC social competence program in Cyprus, Romania, Turkey, and Kosovo. In P. K. Smith & J. O. Higgins (Eds.), The Wiley-handbook of bullying. London and New York: Routledge.

55.

Trip,

, Bora,

, Sipos-Gug,

, Tocai,

, Gradinger,

, Yanagida,

, & Strohmeier,

(2015). Bullying prevention in schools by targeting cognitions, emotions and behaviour: Evaluating the effectiveness of the REBE-ViSC program. Journal of Counseling Psychology62, 732–740. doi: 10.1037/cou0000084

56.

Ttofi,

M. M.

, & Farrington,

D. P.

(2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology7, 27–56. doi: 10.1007/s11292-010-9109-1

57.

Vitaro,

, Brendgen,

, & Barker,

E. D.

(2006). Subtypes ofs: A developmental perspective. International Journal of Behavioral Development30, 12–19. doi: 10.1177/0165025406059968.

58.

Yanagida,

, Strohmeier,

, & Spiel,

(2019). Dynamic change ofand victimization among adolescents: Effectiveness of the ViSC program. Journal of Clinical Child and Adolescent Psychology48, 90–104. doi: 10.1080/15374416.2016.1233498

59.

Yurtal,

, & Cenkseven,

(2016). Bullying at primary schools: Prevalence and nature. Turkish Psychological Counseling and Guidance Journal3, 1–13.

60.

Williford,

, Boulton,

, Noland,

, Little,

T. D.

, Kärnä,

, & Salmivalli,

(2012). Effects of the KiVa anti-bullying program on adolescents’ depression, anxiety, and perception of peers. Journal of Abnormal Child Psychology40, 289–300. doi: 10.1007/s10802-011-9551-1