Challenges in the Evaluation of Cyberbullying Prevention

Abstract

Nowadays cyberbullying prevention programs such as Surf-Fair can significantly reduce cyberbullying but little is known about necessary and sufficient conditions of their effectiveness. Case Study 1 followed three successive waves of implementation of Surf-Fair in one German highest-track secondary school in a pre-post design. Results show no reduction in negative cyber incidents, but students indicated using better coping strategies afterwards and evaluated Surf-Fair positively. These effects were more pronounced in later waves conducted by teachers, confirming teachers’ ability to administer Surf-Fair. Case Study 2 explored if a stand-alone Surf-Fair Bystander Unit would successfully increase empathy and helping and reduce negative cyber incidents at German mixed-track comprehensive schools. No significant effects were detected in a pre-post waiting-control-group design, most likely due to the context. Based on these examples more general challenges of cyberbullying prevention research will be discussed, for example school volunteering and politics and different evaluation and didactic approaches.

Keywords

Cyberbullying prevention evaluation bystander teachers

Nowadays, we have substantial evidence that (cyber-) bullying can be successfully reduced. Prevention and intervention programs specifically targeting cyberbullying such as the Australian Cyber Friendly Schools (Cross et al., 2016), the Spanish ConRed (Del Rey, Casa, & Ortega, 2016), the Italian Noncardiamointrappola! (Menesini, Palladino, & Nocentini, 2016) or the German Media Heroes (Wölfer et al., 2014) as well as those programs targeting bullying and social competence more generally such as the Finnish KiVa (Williford et al., 2013) or the Austrian ViSC (Gradinger, Yanagida, Strohmeier, & Spiel, 2015) all showreductions of cyber-victimization and cyber-perpetration in the treatment groups in comparison with control groups which often go hand in hand with reductions in traditional bullying. However, these programs have very diverse theoretical backgrounds, didactical approaches, and combine multiple parts to address many risk and protective factors of cyberbullying simultaneously. Thus, it is an open question which parts are necessary and/or sufficient for these programs’ success (Cross et al., 2016).

First hints might come from recent meta-analyses about the effectiveness of (traditional) bullying prevention programs (Evans, Fraser, & Cotter, 2014; Jiménez-Barbero, Ruiz-Hernández, Llor-Zaragoza, Pérez-García, & Llor-Esteban, 2016): Overall, such programs show mixed success with on average moderate effect sizes in reducing bullying and victimization. Moderating variables such as program duration have been identified (Farrington & Ttofi, 2009) but could not be confirmed in other meta-analyses (Evans et al., 2014). However, evaluation studies of single programs show significant moderators: For example, students’ antibullying attitudes (Saarento, Boulton, & Salmivalli, 2015), bullies’ popularity (Garandeau, Lee, & Salmivalli, 2014), or the program’s implementation fidelity (Haataja et al., 2014) affected the success of the Finish KiVa program.

In this set of studies, we explored the conditions and boundaries of the German cyberbullying prevention program Surf-Fair (Pieschl & Porsch, 2012). Surf-Fair is less comprehensive and shorter than other programs and has a unique didactical approach: Based on student-centered and constructivist anchored instruction (Cognition and Technology Group at Vanderbilt, 1992) we show a fictitious cyberbullying case that ends without solution. In the subsequent modular exercises with an emphasis on group work participants are guided to define the problem and find solution themselves from the perspectives of cyber-victims, cyber-perpetrators, and cyber-bystanders (Salmivalli, 1999). We already demonstrated Surf-Fair’s success in reducing cyberbullying (Pieschl & Urbasik, 2013): In 2010 we conducted a formative evaluation case study at one German Gymnasium where we compared three classrooms (each with n = 29): one control group (CG) without treatment, one receiving 90 minutes of treatment (EX90), and the other receiving 180 minutes of treatment (EX180). One trained student member of the program development team administered all treatments. Descriptively, the results show an increase in cyber-victimization and cyber-perpetration in CG, almost equal rates in EX90, and reductions in EX180; the differences in development between CG and EX180 were significant.

Even though these results are promising, we are cautious in generalizing. The first evaluation study described above (Pieschl & Urbasik, 2013) was conducted under seemingly ideal conditions, namely a program developer administered the full comprehensive version of Surf-Fair to highly capable students from the highest-track German schools (Gymnasium). To explore the limits of Surf-Fair, we conducted two additional case studies where we varied administrators (program developers vs. teachers), scope (comprehensive anti-cyberbullying program vs. stand-alone bystander unit), and types of school (high-track school vs. mixed-track school). We will shortly review each variable before presenting the relevant case study.

Case Study 1: Can Teachers Implement Surf-Fair?

This case study explores if teachers can implement Surf-Fair just as well as members of the program development team; the success of the latter approach was already demonstrated in the original evaluation case study (see above, Pieschl & Urbasik, 2013).

On the one hand, many successful prevention programs use teachers as program administrators (for a summary see Evans et al., 2014) because teachers are highly relevant in the context of bullying (Veenstra, Lindenberg, Huitsing, Sainio, & Salmivalli, 2014). For example, students with a good relationship with their teachers engage in less bullying (Wang, Swearer, Lembeck, Collins, & Berry, 2015) and teachers’ attitudes towards bullying can moderate the success of anti-bullying programs (Saarento et al., 2015). On a practical level, having teachers administer prevention programs also grants schools low-cost and low-threshold access to bullying and cyberbullying prevention. Therefore, as default we also assumed that teachers would be able to administer Surf-Fair successfully with the help of a detailed manual (Pieschl & Porsch, 2012). On the other hand, prevention and intervention research in general often shows that programs implemented by members of the development teams are more successful than those implemented by others (Beelmann & Karing, 2014) and not all teachers are equally competent and motivated to administer anti-bullying programs with the necessary fidelity (Goncy, Sutherland, Farrell, Sullivan, & Doyle, 2015; Haataja, Ahtola, Poskiparta, & Salmivalli, 2015; Polanin & Espelage, 2015).

To explore if teachers could successfully administer Surf-Fair, we accompanied and supported the implementation of Surf-Fair at one volunteering Gymnasium (highest-track school in Germany) across three years. In the first wave (2010), trained student research assistants who were part of the program development team administered Surf-Fair; in this wave politics teachers observed the implementation and discussed their questions with the first author. In the next two waves (2011 and 2012), the politics teachers implemented Surf-Fair independently in their regular politics classrooms. In all waves Surf-Fair was administered for 180 minutes and evaluated in a pre-post design without control group; pre data was collected directly before the treatment, post data approximately two months after the end of treatment. Using similar measures as the original case study (Pieschl & Urbasik, 2013), we hypothesized that Surf-Fair would decrease cyber-perpetration and cyber-victimization prevalence rates from pre to post (H1), increase the likelihood of using appropriate coping strategies from pre to post (H2), result in positive evaluations at post measurement (H3), and that teachers would be as good at administering Surf-Fair as program developers (no significant superiority of wave 1) (H4).

Method

Sample

Out of thirteen participating classrooms with approximately N = 391 students, one classroom (8%) had to be eliminated from the analysis because less than fifty percent of students provided complete data; another seventy-two students (20%) were excluded because they did not participate in Surf-Fair, did not answer the pre or post questionnaire, or did not provide parental consent. Thus, the final sample consisted of n = 289 students (age: M = 10.51, SD = 0.53; 154 boys, 135 girls). In the first wave n = 38 (2 classrooms) participated, in the second wave n = 143 (5 classrooms), and in the third wave n = 108 (5 classrooms).

Measures

Cyber Incidents

This questionnaire (Pieschl, Kuhlmann, & Porsch, 2015) measures the frequency of cyber-perpetration and cyber-victimization in the last two months, each with five items (harassment, denigration, impersonation, outing, and exclusion; Willard, 2007) on 5-point scales from 0 = never to 4 = multiple times per week. Due to the diverse nature of these behavioral items, internal consistency is mixed (cyber-perpetration Cronbach’s α= 0.68/0.72; cyber-victimization Cronbach’s α= 0.57/0.66; values indicate pre/post results). As in the original case study (Pieschl & Urbasik, 2013), we classified participants as cyber-perpetrators or cyber-victims if they gave at least one answer other than “never”. We do not refer to these experiences as cyberbullying because they do not meet all defining criteria such as repetition. The rationale behind this is that Surf-Fair targets the reduction of all potentially harmful cyber incidents.

Coping with Cyber-Victimization

This questionnaire (Pieschl & Urbasik, 2013) asks students to imagine being a cyber-victim and to indicating the likelihood of using twelve coping strategies on 5-point scales from 1 = definitely not to 5 = definitely. The original factor structure could not be replicated. After excluding three items, an exploratory factor analysis explaining 48% of variance yielded the scales “active coping” (7 items; Cronbach’s α= 0.70/0.73) and “technical coping” (2 items; Cronbach’s α= 0.71/0.66) (values indicate pre/post results).

Subjective Evaluations

Students indicated their agreement to the statements “Surf-Fair is very interesting” and “I have learned a lot by participating in Surf-Fair” on 5-point scales from 1 = high disagreement to 5 = high agreement and evaluated the program Surf-Fair with regular school grades on a 6-point scale from 1 = very good to 6 = insufficient.

Results

We only interpret results with p < 0.05 as significant and we only report significant results.

H1: Decreased Prevalence of Cyber Incidents

Across all waves 5.6%/8.7% of students were classified as cyber-victims and 2.9%/3.6% were classified as cyber-perpetrators which corresponds to scale means of M = 0.03, SD = 0.14/M = 0.03, SD = 0.15 for cyber-victimization and M = 0.01, SD = 0.10/M = 0.01, SD = 0.10 for cyber-perpetration (values indicate pre/post results). Neither McNemar tests for the classifications nor mixed ANOVAs for the scale means (cyber-victimization and cyber-perpetration), showed significant main effects of time (within subject: pre vs. post) or wave (between-subject: 2010, 2011, 2012) or significant interactions. Hypothesis 1 was not confirmed.

H2: Increased Appropriate Coping

Across all waves the likelihood of using “active” coping (M = 4.01, SD = 0.68/M = 3.92, SD = 0.74) was higher than using “technical” coping (M = 2.78, SD = 1.23/M = 3.19, SD = 1.22). A mixed MANOVA shows significant main effects of time (within-subject: pre vs. post), F(2, 264) = 6.20, p < 0.005, $η_{p}^{2}$ = 0.05, and wave (between-subject: 2010, 2011, 2012), F(4, 530) = 3.74, p < 0.010, $η_{p}^{2}$ = 0.03, and a significant interaction, F(4, 530) = 3.26, p < 0.05, $η_{p}^{2}$ = 0.02. Post hoc analyses show univariate effects only regarding “technical” coping: In 2011 and in 2012 students indicated a higher likelihood of using “technical” coping after Surf-Fair than before, t(140) = –3.09, p < 0.005, d = –0.26 and t(94) = –5.30, p < 0.001, d = –0.55 respectively. Hypothesis 2 was partly confirmed.

H3: Positive Subjective Evaluation

Across all waves students evaluated Surf-Fair as highly interesting (M = 4.04, SD = 0.93), thought they had learned a lot (M = 4.30, SD = 0.94), and gave Surf-Fair a mean grade of “good” (M = 2.13, SD = 0.83). ANOVAs testing for effects of wave (between-subject: 2010, 2011, 2012) showed significant differences regarding “interest”, F(1,284) = 24.09, p < 0.001, $η_{p}^{2}$ = 0.15, “learning”, F(1,284) = 26.34, p < 0.001, $η_{p}^{2}$ = 0.16, and “grade”, F(1,284) = 9.72, p < 0.001, $η_{p}^{2}$ = 0.06. In each case the subjective evaluations became progressively more positive in later waves. Hypothesis 3 was confirmed.

H4: Teachers as Good Surf-Fair Administrators

Results regarding all scales (H1-H3) showed no superiority of wave 1 (trained student program developers) over waves 2/3 (politics teachers), all significant wave effects point in the reverse direction. Hypothesis 4 was confirmed.

Discussion

The results seem to indicate that Surf-Fair does not impact cyber-victimization and cyber-perpetration prevalence or the likelihood of using “active” coping, but increases the likelihood of “technical” coping and is evaluated positively on a subjective level. Furthermore, teachers seem to be at least as good as student program developers in administeringSurf-Fair.

However, in interpreting these results additional factors and the limitations of this case study have to be considered: Only students of the highest track of the German educational system (Gymnasium) participated; we do not know if the results can be generalized to other populations. For example, regarding “technical” coping we can speculate that especially students from the highest educational track might profit from Surf-Fair in this regard because such students might also possess better digital literacy (Behrens & Rathgeb, 2016). Regarding the prevalence of negative cyber incidents, we might not have found significant effects due to a lack of a control group (which could not be implemented due to school policies). Normally, cyber-perpetration and cyber-victimization prevalence increases in this age group (Pieschl & Urbasik, 2013), thus a lack of significant increase might be interpreted positively. Similarly, we might not have found an effect regarding “active” coping due to a ceiling effect (overall high likelihood of using active coping). Additionally, ample variance between classes indicates that classroom level differences might have overshadowed any treatment effects. Last but not least, we cannot conclude that teachers are significantly better administrators of Surf-Fair than members of the development team because we cannot distinguish between cohort and treatment effects (program developers vs. teachers). An alternative interpretation would be that the issue of cyberbullying became more relevant to students, parents, and teachers over the years and that this perceptional shift had an impact on students’ evaluations of Surf-Fair.

Despite these limitations we consider this case study important and relevant because it explored the implementation of Surf-Fair in a realistic school setting and showed that under these ecologically valid conditions students seem to learn something from Surf-Fair (i.e., coping), evaluated Surf-Fair positively, and that teachers can administerSurf-Fair.

Case Study 2: Does a Bystander Unit Alone Work in a Comprehensive School?

This case study goes two steps further: First, it explores if one specific perspective of Surf-Fair is sufficient for its success, namely exclusively focusing on cyberbullying bystanders. On the one hand, many students witness cyberbullying as bystanders (Behrens & Rathgeb, 2016), bystanders are highly relevant to bullying (Salmivalli, 1999), bystander interventions were already successful in reducing bullying (Polanin, Espelage, & Pigott, 2012), and empathy trainings for bystanders reduced negative cyber-bystander behavior significantly (Barlinska, Szuster, & Winiewski, 2013). On the other hand, the positive effects regarding cyber-bystanding seem only short-termed (Barlinksa, Szuster, & Winiewski, 2015). One explanation is that while most students notice cyberbullying, not all of them feel responsible (Gahagan, Vaterlaus, & Frost, 2016), and thus many fail to intervene (Dillon & Bushman, 2015). This bystanding behavior seems to depend on many contextual factors (DeSmet et al., 2014) such as the number of other bystanders (Machackova, Dedkova, & Mezulanikova, 2015).

Second, this case study explores if such a stand-alone Bystander Unit can also be successful at a comprehensive school (German: Gesamtschule) that accepts students of all ability levels. On the one hand, most prevention programs claim that they should be effective in all schools. On the other hand, one could speculate that especially a constructivist program such as Surf-Fair is highly dependent on student motivation and ability and thus might fail under these conditions. Additionally, representative surveys in Germany show that perpetration of violence in general (Schlack, Hölling, & Petermann, 2009) as well as cyber-perpetration, cyber-victimization, and cyber-bystanding (Behrens & Rathgeb, 2016; Porsch & Pieschl, 2014) are more prevalent in lower-track schools.

To explore these questions, we developed a new theory-driven stand-alone Surf-Fair Bystander Unit based on the student-centered didactic approach of Surf-Fair: A photo case story without a solution was used to introduce the different roles of bystanders (reinforcers, assistants, outsiders, and defenders; Salmivalli, 1999) and stimulate students’ empathy. In small groups, students should resolve the situation from the bystander perspective: They discussed and practiced noticing a situation as cyberbullying, interpreting it as an emergency, feeling responsible for helping, knowing adequate forms of assistance, and implementing those (Latané & Darley, 1970). For the evaluation, we cooperated with two volunteering inner-city comprehensive schools in 2015. At both schools we implemented pre-post designs with waiting control groups. Classrooms were randomly assigned to the control (CG) or the experimental group (EX). The second author who was part of the Surf-Fair Bystander Unit program development team administered all trainings (each 180 minutes) and collected all data directly before the EX treatment (pre) and approximately two months after the EX treatment (post). We hypothesized that the Surf-Fair Bystander Unit would significantly increase empathy (H1), increase helpful bystander behavior (H2), and thereby decrease cyber-perpetration, cyber-victimization, and cyber-bystanding prevalence rates (H3) in the EX group in comparison withthe CG.

Method

Sample

Of approximately N = 216 eligible students approximately sixty-six (31%) were excluded because they did not participate in Surf-Fair, did not answer the pre or post questionnaire, or did not provide parental consent; none of the participating classrooms were excluded from the analysis. Thus, the final sample consisted of n = 150 students (age: M = 11.31, SD = 0.65; 79 boys, 71 girls), 71 from school A (EX: n = 30, 2 classrooms; CG: n = 41, 2 classrooms) and 79 from school B (EX: n = 44, 3 classrooms; CG: n = 35, 2 classrooms).

Measures

Empathy

A translated version of the Adolescent Measure of Empathy and Sympathy (AMES; Vossen, Piotrowski, & Valkenburg, 2015) was used to measure students’ affective (4 items) and cognitive empathy (4 items) and their sympathy (4 items); all items were answered on 5-point scales from 0 = never to 4 = always.

Helping

We confronted students with three fictitious cyber scenarios in which they took the role of a cyber-bystander. First, they had to make a forced-choice between the alternatives (1) I help but only if I like the victim (person-dependent), (2) I help but only if the victim asks for help (situation-dependent), (3) I help but only if others help (social-dependent), (4) I help in any case (unconditional), or (5) I do not help. Second, they had to justify their choice in their own words and if they chose to help they had to elaborate their ideas for helping.

Cyber Incidents

Adapted from the Revised Olweus Bullying Questionnaire (Olweus, 2012) students indicated their involvement in cyber-perpetration, cyber-victimization, and cyber-bystanding in the last two months, each on a one-item 5-point scale from 0 = never to 4 = multiple times per week. As in Case Study 1, we classified participants as cyber-perpetrators, cyber-victims, or cyber-bystanders if answered other than “never”. We do not refer to these experiences as cyberbullying because they do not meet all defining criteria. The rationale behind this is that Surf-Fair targets the reduction of all potentially harmful cyber incidents.

Results

We only interpret results with p < 0.05 as significant and we only report significant results.

H1: Increased Empathy

Overall students scored highest on sympathy (M = 2.89, SD = 0.82/M = 2.94, SD = 0.85), followed by cognitive empathy (M = 2.10, SD = 0.91/M = 2.31, SD = 0.84), and affective empathy (M = 1.50, SD = 0.95/M = 1.48, SD = 0.84) (values indicate pre/post results). A mixed MANOVA across the three empathy scales with treatment (between-subject: EX vs. CG) and time (within-subject: pre vs. post) as factors shows no significant multivariate effects. Hypothesis 1 was notconfirmed.

H2: Increased Helping

Overall and across scenarios 91%/90% of students indicated in the forced-choice question that they would help a cyber-victim (“total help” = combined option (1) – (4)), the majority would even render “unconditional help” (60%/64%). Counting students’ ideas for helping, on average students had M = 2.31, SD = 1.66/M = 2.58, SD = 1.89 “help ideas” across the three scenarios. The most frequent ideas were to “contact the cyber-perpetrator”, “comfort the cyber-victim”, or “seek help” (from teachers, police, parents, or peers). Mixed ANOVAs, one for “total help” and one for “unconditional help”, with treatment (between-subject: EX vs. CG) and time (within-subject: pre vs. post) as factors show no significant effects. Hypothesis 2 was not confirmed.

H3: Decreased Prevalence of Cyber Incidents

Of all students 45%/43% were classified as cyber-bystanders, 33%/36% as cyber-victims, and 19%/15% as cyber-perpetrators which corresponds to means of M = 0.68, SD = 0.98/M = 0.67, SD = 0.99 for cyber-bystanders, M = 0.41, SD = 0.69/M = 0.51, SD = 0.87 for cyber-victims, and M = 0.26, SD = 0.67/M = 0.17, SD = 0.49 for cyber-perpetrators. Neither Chi-square tests for the classifications nor a mixed MANOVA across the three cyber incident scales with treatment (between-subject: EX vs. CG) and time (within-subject: pre vs. post) as factors showed any significant (univariate or multivariate) main effects or interactions. Hypothesis 1 was not confirmed.

Discussion

The results seem to indicate that the Surf-Fair Bystander Unit did not affect empathy, helping behavior, or cyberbullying, at least not at the volunteering comprehensive schools.

However, in interpreting these results additional factors and the limitations of this case study have to be considered: In comparison to the first case study about Surf-Fair (Pieschl & Urbasik, 2013) two important factors were changed, namely the program scope (comprehensive Surf-Fair vs. Surf-Fair Bystander Unit) and the type of school (high-track school vs. mixed-track school). Thus, the lack of effects cannot be clearly attributed to any one of these factors. However, there are some indicators that hint at the problematic school setting: Most students (65%) came from families with migration background and an additional reading comprehension test (LGVT 6–12, Schneider, Schlagmüller, & Ennemoser, 2007) showed above-average reading ability only for 9%, average ability for 50%, and below-average ability for 41% of the students. Additionally, the program administrator reported common student problems in understanding the content of the Surf-Fair Bystander Unit, in working on tasks in independent groups, and in comprehending the questionnaires, especially the empathy scales. Furthermore, there were sporadic behavioral problems of individual students that disrupted the Surf-Fair Bystander Unit and might have decreased its effectiveness. Thus, the Surf-Fair Bystander Unit might not be suited for mixed- or lower-track schools and the administered questionnaires might not have validly measured the intended constructs. Again, we also found ample variance between classes indicating that classroom level differences might have overshadowed any treatment effects. And we have some indicators that pre-existing individual differences might also have overshadowed treatment effects. For example, sex was significantly related to affective empathy (r_pb = 0.31 at pre; r_pb = 0.32 at post) and “help ideas” (r_pb = 0.31 at pre; r_pb = 0.33 at post); girls reported significantly higher affective empathy and generated significantly more “help ideas” than boys.

Despite these limitations and lack of significant effects we consider this case study important and relevant because it revealed difficulties of cyberbullying prevention research and implementation in more challenging schoolsettings.

General Discussion: Pitfalls and Challenges

In Germany, the federal states are responsible for education. Until now, no state has explicit bullying or cyberbullying policies and therefore there are no prevention programs that have to be implemented in all schools (Jäger, Lissman, & Arbinger, 2009). Thus, schools (and individual teachers) can independently decide if and which prevention program(s) to implement against bullying and cyberbullying and if to participate in cyberbullying research.

This situation results in many challenges for schools, for the state of cyberbullying prevention, and for cyberbullying (prevention) research. For schools, it is difficult to find the best evidence-based prevention programs as many commercial but non-evaluated alternatives exist and these are often more advertised than scientific programs (for these and other issues of implementation and acceptance also see Bauer, Damschroder, Hagedorn, Smith, & Kilbourne, 2015). However, many schools do not even look for suitable programs because they may not feel responsible as most cyberbullying happens outside of school, because implementing cyberbullying prevention programs could indicate to parents that there were cyberbullying problems at school and thus tarnish a school’s reputation, because they may think that their teachers have to be Social Media experts in order to implement cyberbullying prevention programs themselves, or because they may not have enough money to book outside cyberbullying prevention experts. Therefore, it is necessary to afford schools a low-cost and low-threshold access to evidence-based cyberbullying prevention.

Many of these issues also apply to cyberbullying prevention research that depends on schools to volunteer and cooperate. First, there might be a self-selection bias of those schools that volunteer to participate in research; schools with capable and motivated students with few problems might be more likely to volunteer. Thus, it is especially important to conduct and report research in more challenging school settings, even though it might be necessary to use different questionnaires and instruments (see Case Study 2). Second, school policies often determine the design that can be implemented; as randomized control trials are often not possible sometimes the informative value of the permitted research is limited (see Case Study 1).

Another more general issue in cyberbullying prevention research is the scale and approach of program evaluation. On the one hand, large-scale randomized control trials are the best option for the summative evaluation of the effectiveness of finished prevention programs. These studies can find even effects of small effect size by controlling for variables on student, classroom, teacher, and school levels. On the other hand, small-scale studies are the best option for the formative evaluation of the effectiveness of work-in-progress prevention programs. These studies might not find small effects if these are overshadowed by other variables, for example a large variance between classes (see Case Studies 1 and 2). But such studies constitute valuable explorations of what works in terms of program parts and/or evaluation instruments. A related general issue is the didactic approach and program fidelity of the cyberbullying prevention program at hand. In teacher-centered programs the trainer has full control of the classroom and directs all activities; thus, implementations in different classrooms should be very similar given high program fidelity. More constructivist student-centered programs such as Surf-Fair focus on group-based student activities; thus, even with high program fidelity implementations can look very differently in different classrooms and not all implementations might be equally effective. Thus, high variance between classes is the norm for such an approach and for such evaluation studies (see CaseStudies 1 and 2).

Considering these challenges of cyberbullying prevention research, we can put the two presented case studies in perspective: These case studies should be considered explorative and parts of an overall formative evaluation strategy of our work-in-progress prevention program Surf-Fair. The two case studies show very different results regarding prevalence of cyber-perpetration and cyber-victimization. One potential explanation are the different employed measures, namely multiple behavioral items for cyber-perpetration and cyber-victimization in Case Study 1 versus one global item each for cyber-bystanding, cyber-perpetration, and cyber-victimization in Case Study 2 (Menesini & Nocentini, 2009). Therefore, we should be cautious in directly comparing these results. Furthermore, we can draw no definite conclusions regarding program scope (comprehensive Surf-Fair vs. Surf-Fair Bystander Unit) or type of school (high-track vs. mixed-track); the Surf-Fair Bystander Unit might be effective in different school types and a comprehensive version of Surf-Fair might be effective in a mixed-track type of school. However, Case Study 1 showed that teachers seem to be as competent in administering Surf-Fair as members of the program development team. We can only speculate about potential moderators and the generalizability of this finding: Teachers might be more motivated to administer a program of short duration such as Surf-Fair (180 min) than programs that require longer effort and commitment. Additionally, the constructivist didactic approach of Surf-Fair might benefit teachers who do not necessarily want to become Social Media experts themselves and are willing to rely on the expertise of their students – though anecdotal evidence suggests the opposite as teachers often prefer teacher-centered methods of teaching.

This student-centered constructivist didactic approach of Surf-Fair also means that implementations may vary strongly between classes. Thus, the lack of effects, for example regarding behavior (cyber incident prevalence), should not be mistaken for evidence against the effectiveness of Surf-Fair. These studies were informative because they were implemented in a realistic school context, not only in the highest track educational track but also in comprehensive schools, and we obtained feedback about what works in terms of program parts and in terms of evaluation instruments.

References

Barlińska

, Szuster

, & Winiewski

(2013). Cyberbullying among adolescent bystanders: Role of the communication medium, form of violence, and empathy. Journal of Community & Applied Social Psychology, 23, 37–51. doi: 10.1002/cas2137

Barlińska

, Szuster

, & Winiewski

(2015). The role of short- and long-term cognitive empathy activation in preventing cyberbystander reinforcing cyberbullying behavior. Cyberpsychology, Behor and Social Networking, 18, 241–244. doi: 10.1089/cyber.2014.0412

Bauer

M. S.

, Damschroder

, Hagedorn

, Smith

, & Kilbourne

A. M.

(2015). An introduction to implementation science for the non-specialist. BMC Psychology, 3, 32. (Online Article). doi: 10.1186/s40359-015-0089-9

Beelmann

, & Karing

(2014). Implementationsfaktoren und -prozesse in der Präventionsforschung: Strategien, Probleme, Ergebnisse, Perspektiven. [Implementation factors and processes in prevention research: Strategies, problems, findings, prospects.]Psychologische Rundschau, 65, 129–139. doi: 10.1026/0033-3042/a000215

Behrens

, Rathgeb

(2016). JIM 2016. Jugend, Information, (Multi-) Media. Basisstudie zum Medienumgang 12- bis 19-Jähriger in Deutschland[Survey about the media use of 12-19 year olds in Germany]. Stuttgart: Medienpädagogischer Forschungsverbund Südwest (mpfs).

Cognition and Technology Group at Vanderbilt. (1992). The Jasper Series as an example of anchored instruction: Theory, program description, and assessment data. Educational Psychologist, 27, 291–315. doi: 10.1207/s15326985ep2703_3

Cross

, Shaw

, Hadwen

, Cardoso

, Slee

, Roberts

, & … Barnes

(2016). Longitudinal impact of the Cyber Friendly Schools program on adolescents’ cyberbullying behavior. Aggressive Behor, 42, 166–180. doi: 10.1002/ab.21609

Del Rey

, Casas

J. A.

, & Ortega

(2016). Impact of the ConRed program on different cyberbulling roles. Aggressive Behavior, 42, 123–135. doi: 10.1002/ab.21608

DeSmet

, Veldeman

, Poels

, Bastiaensens

, Van Cleemput

, Vandebosch

, & De Bourdeaudhuij

(2014). Determinants of self-reported bystander behavior in cyberbullying incidents amongst adolescents. Cyberpsychology Behavior and Social Networking, 17, 207–215. doi: 10.1089/cyber.2013.0027

10.

Dillon

K. P.

, & Bushman

B. J.

(2015). Unresponsive or un-noticed? Cyberbystander intervention in an experimental cyberbullying context. Computers in Human Behavior, 45, 144–150. doi: 10.1016/j.chb.2014.12.009

11.

Evans

C. R.

, Fraser

M. W.

, & Cotter

K. L.

(2014). The effectiveness of school-based bullying prevention programs: A systematic review. Aggression and Violent Behavior, 19, 532–544. doi: 10.1016/j.avb.2014.07.004

12.

Farrington

D. P.

, & Ttofi

M. M.

(2009). Reducing school bullying: Evidence-based implications for policy. Crime and Justice, 38, 281–345. doi: 10.1086/599198

13.

Gahagan

, Vaterlaus

J. M.

, & Frost

L. R.

(2016). College student cyberbullying on social networking sites: Conceptualization, prevalence, and perceived bystander responsibility. Computers in Human Behavior, 55((Part B)), 1097–1105. doi: 10.1016/j.chb.2015.11.019

14.

Garandeau

C. F.

, Lee

I. A.

, & Salmivalli

(2014). Differential effects of the KiVa anti-bullying program on popular and unpopular bullies. Journal of Applied Developmental Psychology, 35, 44–50. doi: 10.1016/j.appdev.2013.10.004

15.

Goncy

E. A.

, Sutherland

K. S.

, Farrell

A. D.

, Sullivan

T. N.

, & Doyle

S. T.

(2015). Measuring teacher implementation in delivery of a bullying prevention program: The impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science, 16, 440–450. doi: 10.1007/s11121-014-0508-9

16.

Gradinger

, Yanagida

, Strohmeier

, & Spiel

(2015). Prevention of cyberbullying and cyber victimization: Evaluation of the ViSC Social Competence Program. Journal of School Violence, 14, 87–110. doi: 10.1080/15388220.2014.963231

17.

Haataja

, Ahtola

, Poskiparta

, & Salmivalli

(2015). A process view on implementing an antibullying curriculum: How teachers differ and what explains the variation. School Psychology Quarterly, 30, 564–576. doi: 10.1037/spq0000121

18.

Haataja

, Voeten

, Boulton

A. J.

, Ahtola

, Poskiparta

, & Salmivalli

(2014). The KiVa antibullying curriculum and outcome: Does fidelity matter?Journal of School Psychology, 52, 479–493. doi: 10.1016/j.js2014.07.001

19.

Jäger

, Lissmann

, & Arbinger

(2009). Cyberbullying – Approaches in Germany. Country report. Zentrum für empirische pädagogische Forschung (zepf), University of Koblenz-Landau, Germany.

20.

Jiménez-Barbero

J. A.

, Ruiz-Hernández

J. A.

, Llor-Zaragoza

, Pérez-García

, & Llor-Esteban

(2016). Effectiveness of anti-bullying school programs: A meta-analysis. Children and Youth Services Review, 61, 165–175. doi: 10.1016/j.childyouth.2015.12.015

21.

Latané

, & Darley

J. M.

(1970). The unresponsive bystander: Why doesn’t he help?Englewood Cliffs, NJ: Prentice Hall.

22.

Machackova

, Dedkova

, & Mezulanikova

(2015). Brief report: The bystander effect in cyberbullying incidents. Journal of Adolescence, 43, 96–99. doi: 10.1016/j.adolescence.2015.05.010

23.

Menesini

, & Nocentini

(2009). Cyberbullying definition and measurement: Some critical considerations. Zeitschrift für Psychologie/Journal of Psychology, 217, 230–232. doi: 10.1027/0044-3409.217.4.230

24.

Menesini

, Palladino

B. E.

, & Nocentini

(2016). Noncadiamointrappola! [Let’s not fall into the trap!]: Online and school-based program to prevent cyberbullying among adolescents. In Völlink

, Dehue

, & McGuckin

(Eds.), Cyberbullying: From theory to intervention (pp. 156–175). New York, NY, US: Routledge/Taylor & Francis Group.

25.

Olweus

(2012). Cyberbullying: An overrated phenomenon?European Journal of Developmental Psychology, 9, 520–538. doi: 10.1080/17405629.2012.682358

26.

Pieschl

, Kuhlmann

, & Porsch

(2015). Beware of publicity! Perceived distress of negative cyber incidents and implications for defining cyberbullying. Journal of School Violence, 14, 111–132. doi: 10.1080/15388220.2014.971363

27.

Pieschl

, & Porsch

(2012). Schluss mit Cybermobbing! Das Trainings- und Präventionsprogramm “Surf-Fair”. [Stop cyberbullying! The training and prevention program “Surf-Fair”.]Weinheim: Beltz.

28.

Pieschl

, & Urbasik

(2013). Does the cyberbullying prevention program Surf-Fair work? - An evaluation study. In Hanewald

(Ed.), From cyber bullying to cyber safety: Issues and approaches in educational contexts (pp. 205–224). Hauppauge, NY: Nova Science Publishers.

29.

Polanin

J. R.

, & Espelage

D. L.

(2015). Using a meta-analytic technique to assess the relationship between treatment intensity and program effects in a cluster-randomized trial. Journal of Behavioral Education, 24, 133–151. doi: 10.1007/s10864-014-9205-9

30.

Polanin

J. R.

, Espelage

D. L.

, & Pigott

T. D.

(2012). A meta-analysis of school-based bulling prevention programs’ effects on bystander intervention behavior. School Psychology Review, 41, 47–65.

31.

Porsch

, & Pieschl

(2014). Cybermobbing unter deutschen Schülerinnen und Schülern: Eine repräsentative Studie zu Prävalenz, Folgen und Risikofaktoren. [Cyberbullying among German students. A representative study about prevalence, consequences, and risk factors.]Diskurs Kindheits- und Jugendforschung, 9, 7–22.

32.

Saarento

, Boulton

A. J.

, & Salmivalli

(2015). Reducing bullying and victimization: Student- and classroom-level mechanisms of change. Journal of Abnormal Child Psychology, 43, 61–76. doi: 10.1007/s10802-013-9841-x

33.

Salmivalli

(1999). Participant role approach to school bullying: Implications for intervention. Journal of Adolescence, 22, 453–459. doi: 10.1006/jado.1999.0239

34.

Schlack

, Hölling

, & Petermann

(2009). Psychosoziale Risiko- und Schutzfaktoren bei Kindern und Jugendlichen mit Gewalterfahrungen: Ergebnisse aus der KiGGS-Studie. [Psychosocial risk and protective factors in children and adolescents with experiences of violence: Results from the German Health Interview and Examination Survey for Children and Adolescents (KiGGS).]Psychologische Rundschau, 60, 137–151. doi: 10.1026/0033-3042.60.3.137

35.

Schneider

, Schlagmüller

, & Ennemoser

(2007). LGVT 6-12 - Lesegeschwindigkeits- und -verständnistest für die Klassen 6-12. Kurznachweis. [LGVT 6-12 - reading speed and comprehension test for grades 6-12.]Göttingen: Hogrefe.

36.

Veenstra

, Lindenberg

, Huitsing

, Sainio

, & Salmivalli

(2014). The role of teachers in bullying: The relation between antibullying attitudes, efficacy, and efforts to reduce bullying. Journal of Educational Psychology, 106, 1135–1143. doi: 10.1037/a0036110

37.

Vossen

H. M.

, Piotrowski

J. T.

, & Valkenburg

P. M.

(2015). Development of the Adolescent Measure of Empathy and Sympathy (AMES). Personality and Individual Differences, 74, 66–71. doi: 10.1016/j.paid.2014.09.040

38.

Wang

, Swearer

S. M.

, Lembeck

, Collins

, & Berry

(2015). Teachers matter: An examination of student-teacher relationships, attitudes toward bullying, and bullying behavior. Journal of Applied School Psychology, 31, 219–238. doi: 10.1080/15377903.2015.1056923

39.

Willard

N. E.

(2007), Cyberbullying and cyberthreats: Responding to the challenge of online social aggression, threats, and distress. Champaign, IL: Research Press.

40.

Williford

, Elledge

L. C.

, Boulton

A. J.

, DePaolis

K. J.

, Little

T. D.

, & Salmivalli

(2013). Effects of the KiVa Antibullying Program on cyberbullying and cybervictimization frequency among Finnish youth. Journal of Clinical Child and Adolescent Psychology, 42, 820–833. doi: 10.1080/15374416.2013.787623

41.

Wölfer

, Schultze-Krumbholz

, Zagorscak

, Jäkel

, Göbel

, & Scheithauer

(2014). Prevention 2.0: Targeting cyberbullying @ school. Prevention Science, 15, 879–887. doi: 10.1007/s11121-013-0438-y

Challenges in the Evaluation of Cyberbullying Prevention – Insights from Two Case Studies

Abstract

Keywords

Case Study 1: Can Teachers Implement Surf-Fair?

Method

Sample

Measures

Cyber Incidents

Coping with Cyber-Victimization

Subjective Evaluations

Results

H1: Decreased Prevalence of Cyber Incidents

H2: Increased Appropriate Coping

H3: Positive Subjective Evaluation

H4: Teachers as Good Surf-Fair Administrators

Discussion

Case Study 2: Does a Bystander Unit Alone Work in a Comprehensive School?

Method

Sample

Measures

Empathy

Helping

Cyber Incidents

Results

H1: Increased Empathy

H2: Increased Helping

H3: Decreased Prevalence of Cyber Incidents

Discussion

General Discussion: Pitfalls and Challenges

References