Psychometric Properties of the Florence CyberBullying-CyberVictimization Scales

Abstract

The present study tried to answer the research need for empirically validated and theoretically based instruments to assess cyberbullying and cybervictimization. The psychometric properties of the Florence CyberBullying-CyberVictimization Scales (FCBVSs) were analyzed in a sample of 1,142 adolescents (M_age=15.18 years; SD=1.12 years; 54.5% male). For both cybervictimization and cyberbullying, results support a gender invariant model involving 14 items and four factors covering four types of behaviors (written-verbal, visual, impersonation, and exclusion). The second-order confirmatory factor analysis confirmed that a “global,” second-order measure of cyberbullying and cybervictimization fits the data well. Overall, the scales showed good validity (construct, concurrent, and convergent) and reliability (internal consistency and test–retest). In addition, using the global key question measure as a criterion, ROC analyses, determining the ability of a test to discriminate between groups, allowed us to identify cutoff points to classify respondents as involved/not involved starting from the continuum measure derived from the scales.

Introduction

In recent times, cyberbullying has become a relevant topic in developmental psychology, given its impact on children's and adolescents' lives. The psychopathological consequences of involvement in this phenomenon are relevant: the link between cybervictimization internalizing and depressive symptoms, anxiety, and suicidal ideation, on the one hand, and cyberbullying and externalizing problems, on the other hand, is well documented in the literature.^1–6 In parallel, several instruments have been developed to assess the prevalence and characteristics of cyberbullying and to understand predictors, consequences, and related factors better, as well as to evaluate the efficacy of interventions. Despite this wide interest, there is still a lack of knowledge with regard to the psychometric properties of these measures,⁷ which can affect the quality and the accuracy of collected information and of research findings.

In line with the assessment of traditional bullying, the literature has shown two different approaches to the measurement of cyberbullying, both with advantages and disadvantages.⁸ The first approach measures cyberbullying as one form of bullying behavior: following a definition of bullying, students respond to a global question about whether they have been (cyber)victimized by others or have (cyber)bullied others in the past couple of months on a scale ranging from “never” to “many times a week,” referring to the Internet or mobile phone.⁹ This measure is used both as a continuous and a threshold measure to classify people using two levels of severity: occasional and more serious cases.⁹ Bullying is considered a first-order construct, and it is conceptually defined by different types of behaviors: physical, verbal, relational, and cyberbullying. This approach is based upon the consideration that cyberbullying is strictly related to traditional bullying, being defined by the same criteria.¹⁰

The second approach measures cyberbullying through multiple-item scales. It is considered a construct separate from, albeit correlated to, traditional bullying. Under this approach, cyberbullying is defined by different types of behaviors: relational cyberbullying, technically sophisticated attacks, mobile phone bullying, Internet bullying, and so on.⁷ Given these different approaches, more efforts are needed to analyze the multidimensional construct of cyberbullying and its possible relation with the first approach of measurement.⁸

In their recent systematic review on cyberbullying assessment instruments, Berne et al.⁷ pointed out the need to define the theoretical construct underlying different measures better and to investigate the structure, validity, and reliability better of most of the cyberbullying instruments.

In relation to the first issue, the literature shows that the definition varies widely,¹¹ as do the corresponding operational measures. For example, not all definitions use the three criteria accepted for addressing traditional bullying (intention to harm, imbalance of power, and repetition). Besides, the continuous evolution of information and communication technology (ICT) creates the need for constant updating of theoretical perspectives adopted by researchers (e.g., the advent of smart phones made the initial distinction between “mobile phone” and “Internet” cyberbullying obsolete¹²). Studies have shown that different types of cyberbullying can be classified according to specific aspects, such as the covert or overt nature of the acts or the specific types of behavior (e.g., exclusion, verbal attack, photos, taking personal information, etc.^8,12–15). Menesini et al.^16,17 defined a classification based on the nature of the attack: written-verbal behaviors (e.g., phone calls, text messages, and e-mails); visual behaviors (e.g., posting compromising, embarrassing pictures and videos); impersonation behaviors, which are more sophisticated attacks based on identity theft (e.g., revealing personal information using another person's account); and exclusion behaviors, aimed at defining those considered members of the ingroup and outgroup (e.g., intentionally excluding someone from an online group). Using both qualitative¹⁶ and quantitative¹⁷ approaches, the authors found that the relevant criteria of adolescents' cyberbullying definition were the same across the four types of behaviors and across different countries,¹⁷ but the importance of this theoretical distinction for assessment purposes is still to be confirmed.

In relation to the second issue, Berne et al.⁷ found that almost all the instruments are self-report questionnaires, and they often lack detailed statistical analyses in testing psychometric properties. The most serious shortcoming was the absence of analysis related to the construct validity (e.g., exploratory and confirmatory factor analyses). In many instruments, the subscales were determined following a theoretically based model and not on the basis of the psychometric evaluation of the questionnaire structure. Few of them demonstrate good construct validity results, and the only other type of validity reported is convergent validity. Indications about reliability are reported in fewer than half of the studies, and internal consistency is the unique psychometric analysis.

Aims

The aim of the present study is to analyze the psychometric properties of the Florence CyberBullying-CyberVictimization Scales (FCBVSs). This is the revised version of another instrument,¹⁸ which showed a monodimensional structure, although initially designed with two dimensions (visual and written-verbal behaviors). In this version, new items were included in order to detect the four subtypes of cyberbullying reported by other studies on definition.^16,17 Specifically, in the present study, the construct validity (factorial structure and measurement invariance across genders), the concurrent and convergent validity, and the reliability (both internal consistency and test/retest) of this new measure were analyzed in a sample of Italian adolescents. Finally, the relation between this continuous measure and the classification based on the global key question were analyzed in order to define specific cutoff points to classify cyberbullies and cybervictims.

Materials and Methods

Participants and procedure

Participants were 1,142 adolescents (54.5% male), enrolled in eight high schools in Tuscany, Italy, aged between 13 and 20 years (M_age=15.18 years; SD=1.12 years). They were part of the control and the experimental groups of two independent evaluation trials of the Noncadiamointrappola! program (3rd ed.) carried out during the school years 2011/2012 and 2012/2013 (9th grade). Data were also collected for other students attending the same schools involved in the program (10th and 11th grade). For the purposes of the present study, data collected at the pretest evaluation in the fall of 2012 (1st trial; n=760) and 2013 (2nd trial) were used, and they were merged into one data set that was used for the present study, after controlling for differences between the two samples (analyses of variance—preliminary analyses). To analyze test–retest reliability, data collected 3 months afterwards were also used, but only for part of the sample.

Measures

FCBVSs

This instrument consists of two scales: one for perpetration and one for victimization. Each item, describing a certain type of behavior, was specified for perpetration and victimization. Both scales included the 10 items of the previous version,¹⁸ plus eight new items, developed following the authors' theoretical approach based on four patterns of behavior.^16,17 In fact, items about exclusion and impersonation behaviors were mainly added, not considered in the previous version. Participants were asked how often they had experienced particular behaviors/events during the past couple of months. Each item was evaluated on a 5-point scale, where 1=“never,” 2=“once or twice,” 3=“one or two times at month,” 4=“once a week,” and 5=“several times a week.” The two scales were composed of seven items for written-verbal, four items for visual, three items for exclusion, and four items for impersonation. The scales were introduced with the following sentence: “Cyberbullying is a new form of bullying, which involves the use of text messages, photos and videos, phone calls, e-mails … to attack another student.” This definition followed an initial definition of bullying presented at the beginning of the traditional bullying questionnaire.

Youth Self-Report

The Youth Self-Report (YSR)¹⁹ is a self-report questionnaire for the assessment of psychological symptoms in adolescents. The response format for the 103 items is 0=“not true,” 1=“somewhat or sometimes true,” and 2=“very true or often true.” The YSR can be scored on various syndrome scales: withdrawal, somatic complaints, and anxiety/depression form the Internalizing Scale; delinquent behavior and aggressive behavior form the Externalizing Scale.

Global key questions on cyberbullying and cybervictimization

In the questionnaire package, two additional items were included on students' involvement in cyberbullying and cybervictimization starting from the Olweus global key question for bullying and victimization.²⁰ Students were asked the frequency of their involvement in cyberbullying and cybervictimization episodes. The items were rated on a 5-point Likert-type scale: 0=“not at all, 1=“once or twice,” 2=“two or three times a month,” 3=“every week,” to 4=“several times a week.”

Overview of the analyses

Construct validity

All the analyses were conducted using Mplus v7.0.²¹ Given the non-normality of the items' distribution, in the confirmatory factor analyses (CFA), the missing data MLR estimator was used to obtain robust estimates.²²

First, different dimensional models were tested for both cyberbullying and cybervictimization using separate CFAs in the entire sample: (a) five-factor models, distinguishing the four types of behaviors plus another specific factor that could be defined as “spreading false rumors” (items 9 and 14). Preliminary analyses (correlations and exploratory factor analysis [EFA]) revealed the possible presence of this additional factor. For this reason, it was decided to test whether the two items could define another kind of behavior or they could fit into the written-verbal subscale (as originally supposed); (b) four-factor models distinguishing the four types of behaviors; (c) second-order factor models (cyberbullying/cybervictimization); and (d) monodimensional models.

All of the models were evaluated by means of the following overall indices: the chi square statistic, the root mean square error of approximation (RMSEA), and the comparative fit index (CFI). Recommended cutoff points for these measures are 0.08²³ or 0.06²⁴ for RMSEA and 0.90 or 0.95 for CFI.²⁵

The set of models defined by Muthén and Muthén²¹ were followed in order to test the measurement invariance across gender of the final model, listed from the least to the most restrictive. The evidence for invariance was tested both considering the Bayesian Information Criterion (a lower BIC indicates a better trade-off between fit and complexity) and through the significance of the difference in the chi square value between the nested models.^a If the difference between test results was significant, it would be possible to proceed by applying a partial invariance test.^26,27

Concurrent, convergent validity and reliability

In order to assess the concurrent validity of the factors (both first and second order), they were correlated with the global key questions^8,9 on cyberbullying and cybervictimization. Literature on cyberbullying underscored strong negative psychological consequences for people involved.^1–6 For this reason, to assess the convergent validity of the factors found (both first and second order), they were correlated with the externalizing scale (cyberbullying scales) and the internalizing scale (cybervictimization scales).

In order to evaluate the reliability of the scales, both the internal consistency (Cronbach's α) and the test–retest reliability (Pearson's r correlations) were analyzed.

ROC analyses

A receiver operating characteristic (ROC) curve analysis²⁸ was conducted to determine cutoffs for cyberbullying and cybervictimization scales scores (second-order factors; sum of the 14 items) that optimally detect cases classified by the global key question as cyberbullies and cybervictims. Indeed, this measure is used not only as a continuous measure (ranging from 0 to 4) but also to classify people using two levels of severity⁹: first level of involvement (two or three times a month or more often: occasional cyberbullies/victims); second level of more serious cases (about once a week or more often: severe cyberbullies/victims).⁹ The area under the curve (AUC) represents the probability that scores will correctly discriminate between groups (involved/not involved and serious cases/not⁹). An area of 0.8–0.9 is defined as excellent. The cutoff score is determined by the point at which there is the best trade-off between sensitivity (percentage of cases correctly identified) and false positive rates (percentage of cases incorrectly identified), minimizing the total error in misclassification.

Results

Descriptive statistics for items and subscales are reported in the Appendix.

For both cyberbullying and cybervictimization scales, item 2 was excluded for low frequency (<3% of the sample was involved in these behaviors), and item 4 was excluded because item response theory models carried out on the previous version of the scales¹⁸ showed low discrimination indexes for such items. Therefore, all the models tested have a starting configuration based on 16 items.

Construct validity

Table 1 shows fit indices for all tested models. The first ones (a) (five factors models, 16 items) overall demonstrated adequate fit indices. However, correlations between “impersonation” and “spreading false rumors” were very high (cybervictimization: 0.745; cyberbullying: 0.879), while correlations between “written-verbal” and “spreading false rumors” were 0.604 for cyberbullying and 0.664 for cybervictimization. These results did not allow us to consider those items as written-verbal behaviors (as supposed when they were created), or consider them separately as “spreading false rumors,” or theoretically accept to insert them into the “impersonation” factor. Given these controversial results, it was decided to discard these items in the following analyses. The four-factor models (14 items) (b) for both cyberbullying and cybervictimization constructs fitted the data well, with a correlation between item 16 and item 12 residuals in the cybervictimization scale (b1). The second-order factor models (c) showed adequate fit indices for both constructs, while both monodimensional models (d₁, 16 items; d₂, 14 items) did not. Overall, considering the too high correlations between two factors of the models based on five factors (16 items) and the not acceptable fit indexes of unidimensional models, the models were considered based on 14 items and four factors the best and most parsimonious both for cybervictimization (b1) and cyberbullying (b). The final models are represented in Figures 1 and 2.

FIG. 1.

Factor loadings and factors correlations of final cybervictimization model (b1). All parameters are significant at p<0.001.

FIG. 2.

Factor loadings and factors correlations of final cyberbullying model (b). All parameters are significant at p<0.001.

Table 1.

Fit Indices of Confirmatory Factor Analysis

	χ ²	df	p	CFI	RMSEA	N
Cybervictimization						1,123
Model CV (a)	158.961	94	0.000	0.930	0.025
Model CV (b)	127.018	71	0.000	0.922	0.027
Model CV (b1)	105.104	70	0.004	0.951	0.021
Model CV (c)	131.973	72	0.002	0.944	0.022
Model CV (d₁)	400.607	104	0.000	0.682	0.050
Model CV (d₂)	339.638	77	0.000	0.635	0.055
Cyberbullying						1,115
Model CB (a)	165.041	94	0.000	0.908	0.026
Model CB (b)	123.321	71	0.000	0.917	0.026
Model CB (c)	130.112	73	0.000	0.912	0.026
Model CB (d₁)	330.445	104	0.000	0.707	0.044
Model CB (d₂)	267.570	77	0.000	0.696	0.047

Model (a): Five factors, 16 items (written-verbal: 1, 3, 4, 5, 7; visual: 2, 6, 8, 10; impersonation: 11, 13, 15, 17; exclusion: 12, 16, 18; spreading false rumors: 9, 14). Model (b): four factors, 14 items (written-verbal: 1, 3, 5, 7; visual: 6, 8, 10; impersonation: 11, 13, 15, 17; exclusion: 12, 16, 18); Model (b1) for cybervictimization: four factors, 14 items with a correlation among item16 and item 12 residuals; Model (c): second-order models (cyberbullying/cybervictimization), 14 items, four first-order factors (written-verbal: 1, 3, 5, 7; visual: 6, 8, 10; impersonation: 11, 13, 15, 17; exclusion: 12, 16, 18), a correlation among item 16 and item 12 only for cybervictimization; Model (d₁): monodimensional model, 16 items; Model (d₂): monodimensional model 14 items. The best and most parsimonious models displayed in bold.

CFI, comparative fit index; RMSEA, root mean square error of approximation; CV, cybervictimization; CB, cyberbullying.

Table 2 displays the test results for measurement invariance across gender. All models were based on Model CV (b1) and Model CB (b). For cybervictimization, a partial metric invariance with two free factor loadings across gender (Model B1 items 17 and 8) and a partial scalar invariance with two free intercepts across gender (Model C items 18 and 1) were confirmed. For cyberbullying, a full metric invariance (Model B) and a partial scalar invariance with two intercepts for items 11 and 15 free across gender (Model C1) were confirmed.

Table 2.

Test Results for Measurement Invariance Across the Two Groups for Cybervictimization (Males n=573; Females n=550) and Cyberbullying (Males n=566; Females n=549)

		Compared model	χ² (df)	Scaling correction factor for MLR	χ² difference test	RMSEA	BIC
Cybervictimization model
A	Configural invariance		299.852 (140)	2.991		0.045	14,796.060
B	Full metric invariance	A	310.968 (150)	3.415	^*	0.044	14,891.045
B1	Partial metric invariance	A	284.007 (148)	3.336	n.s.	0.040	14,790.583
C	Full scalar invariance	B1	303.355 (158)	3.185	^*	0.040	14,739.085
C1	Partial scalar invariance	B1	299.351 (156)	3.211	n.s.	0.040	14,748.197
Cyberbullying model
A	Configural invariance		306.824 (142)	4.201		0.046	3,354.079
B	Full metric invariance	A	318.338 (152)	4.775	n.s.	0.044	3,298.403
C	Full scalar invariance	B	339.544 (162)	4.519	^*	0.044	3,303.583
C1	Partial scalar invariance	B	332.147 (160)	4.593	n.s.	0.044	3,193.187

For cybervictimization B1 model: following the MI, we released the constraints for item 17 and item 8 factor loadings; C1 model: we released the constraints for item 18 and item 1 intercepts. For Cyberbullying C1 model: we released the constraints for item 11 and item 15 intercepts.

p<0.05; n.s., not significant.

BIC, Bayesian information criterion.

In conclusion, strong gender measurement invariance was found for both the cyberbullying and the cybervictimization scale.

Concurrent, convergent validity and reliability

In Tables 3 (for cybervictimization) and 4 (for cyberbullying), correlations aimed to test concurrent, convergent validity and test–retest reliability are shown, as well as the Cronbach's alpha coefficients for each factor. All the correlations are statistically significant, and overall the factors showed an adequate reliability level.

Table 3.

Cybervictimization: Pearson's r Correlations Between Factors (First and Second Order) and Global Key Question, Internalizing Symptoms (YSR Scale), and Within Factors 3 Months Later

	Cybervictimization global key question	Internalizing symptoms	Test–retest after 3 months	Internal consistency
CV written-verbal	0.47^***	0.26^***	0.35^***	α=0.72
CV visual	0.26^*	0.08^*	0.16^*	α=0.76
CV impersonation	0.27^***	0.21^***	0.22^***	α=0.63
CV exclusion	0.13^***	0.22^***	0.25^***	α=0.65
CV second-order factor	0.41^***	0.27^***	0.21^***	α=0.79

Cronbach's alphas for the first- and second-order factors are given in the last column.

p<0.05; ^***p<0.001.

YSR, Youth Self-Report.

Table 4.

Cyberbullying: Pearson's r Correlations Between Factors (First and Second Order) and Global Key Question, Externalizing Symptoms (YSR Scale), and Within Factors 3 Months Later

	Cyberbullying global key question	Externalizing symptoms	Test–retest after 3 months	Internal consistency
CB written-verbal	0.39^***	0.30^***	0.25^***	α=0.70
CB visual	0.23^*	0.15^***	0.29^***	α=0.78
CB impersonation	0.33^***	0.27^***	0.47^***	α=0.77
CB exclusion	0.40^***	0.27^***	0.16^*	α=0.75
CB second-order factor	0.48^***	0.35^***	0.27^***	α=0.85

Cronbach's alphas for the first- and second-order factors are given in the last column.

p<0.05; ^***p<0.001.

ROC analyses

Cyberbullying

The AUC was 0.82 (SE=0.049; p<0.001 [CI 0.72, 0.91]) for the first level of involvement and 0.85 (SE=0.058; p<0.001 [CI 0.73, 0.96]) for the second level, indicating an excellent discriminative accuracy for both. A score ≥16 for the first level and ≥19 for the second level obtains the best balance between sensitivity (respectively 69% and 61%) and specificity (84% and 95%). Using those cutoffs, 11.9% of the sample can be classified (first level) as more occasional cyberbullies and 5% as more severe cyberbullies.

Cybervictimization

The AUC was 0.83 (SE=0.054; p<0.001 [CI 0.73, 0.94]) for the first level of involvement and 0.82 (SE=0.082; p<0.001 [CI 0.66, 0.98]) for the second level, indicating for both levels an excellent discriminative accuracy. A score ≥16 for the first level and ≥18 for the second level obtains the best balance between sensitivity (respectively 86% and 73%) and specificity (73% and 87%). Using those cutoffs, 19.1% of the sample (first level) can be classified as occasional cybervictims and 8.9% as more severe cybervictims.

Discussion and Conclusion

Overall, the FCBVSs showed good psychometric properties, suggesting that the scale is a valid and reliable measure for cyberbullying and cybervictimization. In relation to the construct validity of the instrument, CFAs showed that the best model both for cyberbullying and cybervictimization is a four-dimensional model covering four types of behavior and describing different attacks made by peers in the cyber context: written-verbal, visual, impersonation, and exclusion behaviors. These results gave empirical and statistical support to the theoretical approach considered in previous studies.^16,17 Unfortunately, items related to “spreading false rumors” did not acceptably fit into the models. While it is known that it is an important aspect in traditional bullying,²⁹ in the current sample, this behavior is related to impersonation. It might be that some rumors are spread through one's own personal account, or that personal information is stolen from the victim, and this explains why a higher correlation was found between rumors and impersonation. Certainly, further studies are needed to clarify better the role of this specific type of behavior in the cyber context and specifically to measure it.

The second-order CFA confirmed that a “global,” second-order measure of cyberbullying and cybervictimization fits the data well. One of the most important areas in determining the accuracy of scales is measuring and understanding how different groups perform on a scale.³⁰ If the replicability of results across groups can be demonstrated, then the generalizability of the instrument and also of the theoretical construct can further bolster confidence in the scale. Strong measurement invariance was found for gender, and this means that the scales measure the same factors in the same way for both males and females. Having a measure invariant across gender is a starting point for studies that aim to explore this topic. The FCBVSs can be used to analyze specific gender-related issues such as prevalence rates, relations with other constructs, and the consequences of cyberbullying.

The instrument shows good concurrent and convergent validity and adequate reliability (both internal consistency and test–retest). Specifically, significant correlations were found with externalizing (for cyberbullying and its subscales) and internalizing symptoms (for cybervictimization and its subscales). This is in line with the literature about the negative concurrent symptoms and consequences of cyberbullying and cybervictimization,^1–6 and those results support the convergent validity of the instrument. The subscales and the general second-order factors were correlated with another common measure of the construct: the global key question.^8,9 On one hand, the strong correlation found for the second-order factors means that the cyberbullying phenomena is being measured for both approaches with a different depth of understanding.⁸ On the other hand, looking at the subscales correlations, it may be argued that in the adolescents' perception, cyberbullying is largely defined in terms of written-verbal behaviors for both victims and perpetrators. Conversely, the relationship between the general key questions and exclusion behaviors is rather weak if the victims' point of view is considered, while there is a strong association for perpetrators. These discrepant results might be better understood if the definitional criteria related to this behavior are considered. For example, “intention to harm” is clearly present for cyberbullies, while it might not be so salient for victims when they are excluded. Besides, unexpectedly, the weakest correlation for the cyberbullying global key question and single subscales is with visual behavior. In general, those behaviors perceived and recognized as cyberbullying might not be viewed in the same way by victims and perpetrators.

In order to find cutoff thresholds to classify people as involved or not and to compare the two measurement approaches (FCBVSs and global key question), ROC analyses were performed. These results have relevant implications within the field of the cyberbullying measurement due to the need of comparable measures contributing to punctual real estimates of prevalence. The findings of the present study suggest that the FCBVSs can be used both as a continuous measure and as a categorical one (involved/not) in a comparable way.⁸

Overall, the present study addresses the need expressed by Berne et al.⁷ to have a more valid and reliable measure, thus representing a step forward for the assessment of cyberbullying and cybervictimization behaviors. Furthermore, evidence was found for the legitimacy of the theoretical cyberbullying classification^16,17 as a multidimensional construct consisting of verbal, visual, exclusion, and impersonation patterns of behaviors.

Notes

a. The MLR estimator was applied, and consequently the scaling correction factors were used. See www.statmodel.com/chidiff.shtml for further details about the formula used.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Appendix

Appendix A1.

Items, Descriptive Statistics, and Labels for Both Cybervictimization and Cyberbullying

		Cybervictimization			Cyberbullying
Item	Subscale	N	M	SD	N	M	SD
1. Threatening and insulting text message	WV	1,121	1.14	0.50	1,114	1.07	0.31
2. Violent videos/photos/pictures by mobile phone	V	1,119	1.04	0.27	1,112	1.02	0.16
3. Threats and insult on the Internet (Web sites, chatrooms, blogs, MSN, Facebook, Twitter, MySpace)	WV	1,119	1.22	0.56	1,110	1.10	0.40
4. Silent/prank phone calls	WV	1,115	1.59	0.91	1,114	1.18	0.50
5. Threatening and insulting e-mails	WV	1,118	1.07	0.38	1,110	1.04	0.26
6. Videos/photos/pictures of embarrassing or personal situations by mobile phone	V	1,117	1.11	0.44	1,111	1.05	0.27
7. Threatening and insulting phone calls	WV	1,115	1.09	0.40	1,112	1.07	0.32
8. Violent videos/photos/pictures shared on the Internet	V	1,116	1.09	0.42	1,112	1.04	0.25
9. Phone calls with rumors about me	WV	1,117	1.26	0.65	1,111	1.07	0.30
10. Videos/photos/pictures of embarrassing or personal situations on the Internet (e-mail, Web sites, YouTube, Facebook)	V	1,114	1.11	0.44	1,111	1.04	0.24
11. Manipulating private personal data in order to reuse them	I	1,115	1.09	0.36	1,107	1.05	0.31
12. Ignoring on purpose in an online group	E	1,117	1.11	0.43	1,109	1.15	0.50
13. Theft of personal information (images, photos) in order to reuse them	I	1,112	1.11	0.41	1,107	1.05	0.33
14. Rumors on the Internet	WV	1,117	1.20	0.57	1,109	1.06	0.36
15. Theft of password and account (e-mail, Facebook)	I	1,115	1.22	0.57	1,109	1.11	0.46
16. Exclusion from an online group (chats, forum, Facebook groups)	E	1,110	1.09	0.40	1,108	1.11	0.43
17. Theft and use of phone book	I	1,115	1.07	0.34	1,106	1.06	0.32
18. Block in a chatroom or on Facebook in order to exclude from the group	E	1,114	1.15	0.47	1,110	1.19	0.56

WV, written-verbal behaviors; V, visual behaviors; E, exclusion behaviors; I, impersonation behaviors.

References

Menesini

, Calussi

, Nocentini

. (2012) Cyberbullying and traditional bullying: unique, additive, and synergistic effects on psychological health symptoms. In Li

, Cross

, Smith

, eds. Cyberbullying in the global playground: research from international perspectives. Chichester: Wiley-Blackwell, pp. 245–262.

Hinduja

, Patchin

. Bullying, cyberbullying, and suicide. Archives of Suicide Research, 2010; 14:206–221.

Bauman

, Toomey

, Walker

. Associations among bullying, cyberbullying, and suicide in high school students. Journal of Adolescence, 2013; 36:341–350.

Gámez-Guadix

, Oru

, Smith

, et al. Longitudinal and reciprocal relations of cyberbullying with depression, substance use, and problematic internet use among adolescents. Journal of Adolescent Health, 2013; 53:446–452.

Gradinger

, Strohmeier

, Spiel

. Traditional bullying and cyberbullying. Zeitschrift für Psychologie, 2009; 217:205–213.

Garaigordobil

. Prevalencia y consecuencias del cyberbullying: una revisión. International Journal of Psychology & Psychological Therapy, 2011; 11:233–254.

Berne

, Frisén

, Schultze-Krumbholz

, et al. Cyberbullying assessment instruments: a systematic review. Aggression & Violent Behavior, 2013; 18:320–334.

Menesini

, Nocentini

. Cyberbullying definition and measurement. Zeitschrift für Psychologie, 2009; 217:230–232.

Solberg

, Olweus

. Prevalence estimation of school bullying with the Olweus Bully/Victim Questionnaire. Aggressive Behavior, 2003; 29:239–268.

10.

Olweus

. Cyberbullying: an overrated phenomenon?. European Journal of Developmental Psychology, 2012; 9:520–538.

11.

Wingate

, Minney

, Guadagno

. Sticks and stones may break your bones, but words will always hurt you: a review of cyberbullying. Social Influence, 2013; 8:87–106.

12.

Slonje

, Smith

, Frisén

. The nature of cyberbullying, and strategies for prevention. Computers in Human Behavior, 2013; 29:26–32.

13.

Spears

, Slee

, Owens

, et al. Behind the scenes and screens. Zeitschrift für Psychologie, 2009; 217:189–196.

14.

Slonje

, Smith

. Cyberbullying: another main type of bullying?. Scandinavian Journal of Psychology, 2008; 49:147–154.

15.

Schultze-Krumbholz

, Scheithauer

. Social–behavioral correlates of cyberbullying in a German student sample. Zeitschrift für Psychologie, 2009; 217:224–226.

16.

Nocentini

, et al. Cyberbullying: labels, behaviours and definition in three European countries. Australian Journal of Guidance & Counselling, 2010; 20:129–142.

17.

Menesini

, Nocentini

, Palladino

, et al. Cyberbullying definition among adolescents: a comparison across six European countries. Cyberpsychology, Behavior, & Social Networking, 2012; 15:455–463.

18.

Menesini

, Nocentini

, Calussi

. The measurement of cyberbullying: dimensional structure and relative item severity and discrimination. Cyberpsychology, Behavior, & Social Networking, 2011; 14:267–274.

19.

Achenbach

. (1991) Manual for the youth self-report and 1991 profile. Burlington, VT: University of Vermont Department of Psychiatry.

20.

Olweus

. (1996) The Revised Olweus Bully/Victim Questionnaire. Bergen: Research Center for Health Promotion, University of Bergen.

21.

Muthén

, Muthén

. (2012) Mplus User's Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.

22.

Yuan

, Bentler

. Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 2000; 30:165–200.

23.

Browne

, Cudeck

. (1993) Alternative ways of assessing model fit. In Bollen

, Long

, eds. Testing structural equation models. Beverly Hills, CA: Sage, pp. 136–162.

24.

, Bentler

. Fit indices in covariance structure analysis: sensitivity to underparameterized model misspecification. Psychological Methods, 1998; 3:424–453.

25.

Bollen

. (1989) Structural equations with latent variables. New York: John Wiley.

26.

Vandenberg

, Lance

. A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 2000; 3:4–70.

27.

Byrne

, Shavelson

, Muthén

. Testing for the equivalence of factor covariance and mean structures: the issue of partial measurement invariance. Psychological Bulletin, 1989; 105:456–466.

28.

Pintea

, Moldovan

. The receiver-operating characteristic (ROC) analysis: fundamentals and applications in clinical psychology. Journal of Cognitive & Behavioral Psychotherapies, 2009; 9:49–66.

29.

Olweus

. (1993) Bullying at school: what we know and what we can do. Cambridge, MA: Blackwell.

30.

Collins

, Raju

, Edwards

. Assessing differential functioning in a satisfaction scale. Journal of Applied Psychology, 2000; 85:451–461.