Development and Validation of an Instrument to Assess the Impact of Cyberbullying: The Cybervictimization Emotional Impact Scale

Abstract

Cyberbullying is a phenomenon with important adverse consequences on victims. The emotional impact of this phenomenon has been well established. However, there is to date no instrument with good psychometric properties tested to assess such impact. The objective of this study was developing and testing the psychometric properties of an instrument to assess the emotional impact of cyberbullying: the “Cybervictimization Emotional Impact Scale, CVEIS.” The sample included 1,016 Compulsory Secondary Education students (52.9 percent female) aged between 12 and 18 (M = 13.86, DT = 1.33) from three schools in southern Spain. The study used Confirmatory Factor Analyses to test the structure of the questionnaire and robustness of the scale. Internal consistency was also tested. The results supported the suitability of a three-factor model: active, depressed, and annoyed. This model showed an optimal adjustment, which was better than its competing models. It also demonstrated strong invariance among cybervictims and non-cybervictims and also among gender. The internal consistency of each factor, and the total scale, was also appropriate. The article concludes by discussing research and practical implications of the scale.

Introduction

Cyberbullying is today a sufficiently serious issue to be considered a public health problem.¹ This kind of aggression, defined by Smith et al.² as “an aggressive, intentional act carried out by a group or individual, using electronic forms of contact, repeatedly and over time against a victim who cannot easily defend him or herself” (p. 376), could be considered as a new form of bullying although with specific characteristics because of the peculiarities of the medium, the technological environment.

The estimated prevalence of this problem varies across studies depending, among other reasons, on the assessment instrument, time framework considered, ages included, and the definition used. Roughly speaking, the percentage of students involved in the phenomenon (as victims, perpetrators or spectators) across the world ranges between 40 percent and 55 percent.³ As for the estimated prevalence of cybervictimization, a recent meta-analysis conducted by Kowalski et al.,⁴ set this in a range between 10 percent and 40 percent. These authors also pointed out that neither was it clear whether this phenomenon was on the rise or not. Some authors, such as Patchin and Hinduja⁵ claim that the phenomenon is increasing as changes in technology take place, whereas others, such as Olweus,⁶ argue that the prevalence has not increased over the last few years.

However, regardless of its prevalence, the scientific literature leaves little doubt as far as its adverse consequences are concerned. All students, whatever their role (perpetrator, victims, victimized-aggressors or spectators), appear to be affected by being involved in this phenomenon, as occurs in traditional bullying, although each role affects people in a different way.^3,4 Specifically, cyberbullying victimization has been associated with different negative consequences such as depressive symptomatology^7,8; anxiety symptomatology⁹; psychosomatic problems^10,11; low self-esteem and negative self-concept¹²; alcohol and drug use¹³; an increased likelihood of self-harm¹⁴; and even suicidal thoughts and attempts.^15–17

Emotional impact of cybervictimization

Regarding the emotional impact of the phenomenon, cybervictimization has been associated with feelings of fear, offence, rejection, defenselessness, sadness, shame, guilt, loneliness, and helplessness.^18–21 However, nowadays there is enough scientific evidence showing that not all victims are affected in the same way or to the same degree of intensity,^22–24 with some studies having shown different profiles of emotions in cybervictims depending on the type of cyberbullying.^20,21

On the other hand, the emotions felt by victims as a result of cyberbullying could be relevant given that they could influence the coping strategy adopted and the impact that aggression episodes are going to have on victims. In this sense, some studies have found that emotion-focused coping strategies that aim to regulate emotional distress, such as alcohol use or aggressive or passive reactions seem to be ineffective and the online victimization tends to persist.²⁵ Kochenderfer-Ladd²⁶ found that victims reported more intense negative emotion than non-victims in hypothetical situations. In addition, fear was a predictor of advice seeking, whereas anger and embarrassment predicted revenge seeking, which was associated with increases in victimization. In a similar sense, Hunter et al.²⁷ reported that discrete emotional reactions to bullying (anger, fear, and sadness) predicted the extent to which young people reported using specific coping strategies.

In addition, although the definitions of cyberbullying, and that of bullying, highlight the imbalance of power perceived by victims (e.g., in the definition by Smith using here, “against a victim who cannot easily defend him or herself”), the instruments to assess these phenomena do not include items to evaluate this aspect. The assessment of the impact of aggression could be a way to approach that imbalance of power, relevant to distinguish these phenomena from other aggressive episodes.

Furthermore, the perceived emotional impact of cybervictimization is not just important among the victims but also among the other people involved. Several studies have shown that perpetrators usually underestimate the effects of their aggression on victims,^26,28 although the opposite pattern of results has been found in other studies.¹⁸

Therefore, there is a need to gauge the actual or perceived emotional impact of cyberbullying on victims. Given that there is no psychometrically validated instrument for measuring this impact, an instrument to do this task has been developed.

To do so, it was taken into account that although it is supposed that emotions felt by victims as a consequence of cyberbullying are always negative, there is no scientific evidence proving this point. To fill this void, a questionnaire including negative and also positive emotions was elaborated by authors. It was based on the Positive Affect Negative Affect Scale (PANAS)²⁹ that is widely used to assess affect, and consists of 20 items distributed in two factors: positive and negative affect. The above authors understand these factors not as highly correlated dimensions, but as relatively independent, orthogonal, dimensions. They understand that positive affect would reflect the extent to which a person feels enthusiastic, active, and alert, whereas negative affect would be characterized by sadness and lethargy. The good psychometric properties of the scale and the bifactorial structure of affect proposed by the authors have been tested in different cultures and countries.^29–31

Thus, to elaborate the scale, these dimensions and adjectives were used as a base but adapting it so that all the emotions included made sense as an answer to the question: “What emotions did you feel if you have suffered cyberbullying or do you think you would feel if you suffered it?” So, in the first version we used 17 emotions, based on the adaptation of the PANAS into Spanish.³¹ This version, showed acceptable psychometric properties in a three-factor model.³² We called the factors invigoration, dejection, and annoyance. However, there was an important imbalance in the amount of items that composed each factor, dejection factor being overrepresented in comparison to the other two factors. Thus, to try getting three more solid factors in the version testing here, three more emotions, “theoretically” related to the other two factors, were added: ready, clear-headed; choleric, enraged; fed up.

Hence, the general objective of this study was to test the psychometric appropriateness of the “Cybervictimization Emotional Impact Scale” in Secondary school students. Specifically, the aims were to analyze construct validity, the invariance across cybervictims and non-cybervictims, and across gender, and the reliability of the scale and its factors.

Materials and Methods

Participants

The participants were 1,016 Compulsory Secondary Education students (52.9 percent female) aged between 12 and 18 (M = 13.86, DT = 1.33). They were recruited from three state secondary schools in the Spanish cities of Seville, Córdoba and Huelva (Andalusia). Intentional sampling was developed using as selection criteria that the schools had state ownership and the students were at Secondary level.

Instruments

We used self-report questionnaires with Likert-type multiple-choice scales.

The emotional impact scale tested was the “Cybervictimization Emotional Impact Scale, CVEIS.” This scale lists a series of emotions and asks subjects to grade the extent to which they would feel those emotions if they had been or were a cybervictim, on a scale of 1 to 5 (Not at all [1] to A lot [5]). The emotions listed were the following: Indifferent, not bothered; Tense, nervous; Animated; Upset, bothered; Guilty; Energetic, lively; Scared, Afraid; Angry, Annoyed; Satisfied, Proud; Lonely; Irritable, In a bad mood; Ready, Clear-headed; Choleric, enraged; Ashamed; Defenseless, helpless; Determined, Daring; Depressed, sad; Fed up; Active, Alert; Jittery, Worried.

Cyberbullying was assessed using the Spanish version of the “European Cyberbullying Intervention Project Questionnaire, ECIPQ.”³³ This questionnaire has 22 items covering cyberbullying over the previous 2 months, with one subscale for cybervictimization (11 items), which was the used in the study, and another for cyber-aggression (11 items). Answers were entered on a scale of 1 to 5: 1 Never; 2 Once or twice; 3 Once or twice a month; 4 About once a week; 5 More than once a week. The forms of cyber-behavior included were as follows: Insults said to me; Insults about me said to others; Threats; Identity theft; Use of personal identity without permission; Private information theft; Display private information; Embarrassing videos or pictures; Manipulation of pictures; Social exclusion; Spreading of rumors. This scale has displayed good psychometric properties in the studies carried out to date.^33,34 The internal consistency of the cybervictimization subscale in this study, Cronbach's alpha, was 0.77.

Procedure and data analysis

This research was a cross-sectional, retrospective, ex-post-facto, one group, and multiple measures design.³⁵ The instruments were administered in a 60-minute session by trained researchers during school hours. Consent was obtained from the minors' parents, and anonymity and confidentiality of information was assured. A researcher read out each item one-by-one to clarify any misunderstanding. The class teacher was also in the classroom while students completed the questionnaires. This procedure was approved by the Andalusian Ethical Commitee for Biomedical Research.

An exploratory factor analysis (EFA) was run to explore the factors covered by included emotions. Then a Confirmatory Factor Analysis (CFA) was conducted to test the factor structure. To do this, the sample was divided randomly into two halves, using EFA (principal component method and varimax rotation) to assess the factor structure from sample A (n = 495) and then using CFA to confirm the factor structure obtained using Sample B (n = 521). For the rest of the analyses the sample total was used.

Given the ordinal character of the variables, polychoric matrixes were used to estimate the models.³⁶ Besides, given the absence of multivariate normality, showed according to Bentler by a Mardias' coefficient >5,³⁷ the Maximum Likelihood Robust method, Satorra-Bentler scaled Chi-square test, and Robust Comparative Fit Index (RCFI), which are recommended for these kind of data,³⁷ were used. The fit index used to compare the models was a combination of several with cutoffs recommended by Hu and Bentler³⁸: Non-Normed Bentler-Bonnet Index; RCFI with a cutoff >0.90; and Root Mean Square Error Of Approximation (RMSEA) with cutoff values <0.08. In addition, to compare the competitive models the Akaike's Information Criterion (AIC) was used considering the model that produces the minimum AIC a potentially useful model.³⁷

To assess the robustness of the obtained model, a double strategy was used: First, separated model by gender and by cybervictim versus non-cybervictims were run. Second, a multi-group analyses across gender and across cybervictims versus non-cybervictims were developed using a hierarchical strategy to test the invariance of these models. First of all, we tested a model without any constraint (configural model); second, a model in which equal factor loadings from items to factors were imposed (measurement model); and third, a model in which, besides equal factor loadings, factor variances and covariance were imposed. The scaled difference Chi-square test by Satorra and Bentler³⁹ to assess non-invariance was used.

To examine the internal consistency of each factor, and the total scale, the Rho coefficient and the Composite Reliability (CR) were calculated. In addition, to assess the Average Variance Extracted (AVE) was calculated. The AVE is the amount of common variance among latent construct indicators and values greater than 0.50 indicates a high validity of the construct and the individual variables.⁴⁰

The statistical analyses were run using the SPSS 20 statistical package and EQS 6.2 software.³⁷

Results

Considering cybervictims those students that reported to have received any of the 11 cyber-aggressions at least once or twice a month, according to the authors of the questionnaire recommendations, a total of 201 of them reported to have suffered cybervictimization in the last 2 months, reporting almost 7 percent to have suffered at least one of this aggressions more than once a week (Fig. 1).

FIG. 1.

Prevalence of cybervictimization.

There was no significant association between cybervictimization and gender (χ² [3, N = 1,016] = 4.98, p = 0.17).

Construct validity

The EFA showed that the “indifferent, not bothered” emotion had a very low communality (0.126), so we removed this item and ran a new EFA. The results showed an adequate three-factor model, which explained 59.57 percent of the variance. These factors were named, using the most representative emotion contained in each of them, depressed, active, and annoyed. As can be seen in the Table 1, depressed was made up of 9 emotions related to sadness, defenselessness, and so on; active included 6 items related to being vibrant; and annoyed consisted of 4 items related to rage.

Table 1.

Exploratory Factor Analysis: Rotated Components Matrix (Sample A = 495)

	Depressed	Active	Annoyed
Depressed, sad	0.760	−0.267	0.299
Scared, afraid	0.735	−0.084	0.169
Ashamed	0.734	−0.068	0.098
Defenseless, helpless	0.726	−0.154	0.215
Lonely	0.704	−0.210	0.123
Jittery, Worried	0.646	−0.105	0.382
Fed up	0.593	−0.160	0.500
Guilty	0.531	0.069	0.000
Tense, nervous	0.515	−0.018	0.433
Energetic, lively	−0.154	0.812	−0.111
Satisfied, proud	−0.013	0.797	−0.201
Animated	−0.102	0.761	−0.283
Ready, clear-headed	−0.076	0.756	−0.055
Determined, daring	−0.261	0.747	0.215
Active, alert	−0.022	0.667	0.310
Angry, annoyed	0.135	0.060	0.807
Choleric, enraged	0.172	−0.001	0.778
Irritable, in a bad mood	0.221	−0.070	0.768
Upset, bothered	0.374	−0.155	0.639

Then a CFA with the three-factor model was run (Table 2). The Mardias' coefficient = 25.26 showed an absence of multivariate normality, so robust indexes were obtained. Although the indexes were good enough, an analysis of the Langrage Multiplier Test (LMTEST) results showed that a change in the “upset, bothered” item from factor 3 (annoyed) to factor 1 (depressed) could improve the fit (standardized change = 1.273). However, given that from a theoretical point of view, this item seemed different to the others included in factor 1, and given the higher number of items in this factor compared to the rest, we decided to remove it and run the analysis with the remaining 18 items. Again, Mardias' coefficient = 27.97 showed the needs of using robust indexes. The solution (Table 2) showed an even better fit. In addition, the AIC index suggests this last model as the best option.

Table 2.

Model Fit Indicators (Sample B, n = 521)

	χ²_S-B	df	NNBBFI	RCFI	RMSEA (90% CI)	AIC
Three-factor (9-6-4 items) model	504.0866	149	0.964	0.968	0.076 (0.069, 0.084)	206.087
Three-factor (9-6-3 items) model	420.6865	132	0.967	0.972	0.072 (0.064, 0.080)	156.687

AIC, Akaike's information criterion; df, degrees of Freedom; NNBBFI, non-normed Bentler-Bonnet fit index; RCFI, robust comparative fit index; RMSEA, root mean-square error approximation (confident interval, 90%); χ², Chi-square Satorra-Bentler statistic.

Figure 2 shows that the polychoric correlations among the factors in the scale were from moderate to high, showing that although factors were related they were clearly different. The negative emotions, the depressed and annoyed factors, were directly related, whereas the active factor, consisting of emotions related to feeling energetic, determined, and so on, was inversely related to the others.

FIG. 2.

Three correlated components model. *p < 0.05.

Models by gender and by cybervictimization

The three-factor model, consisting of 9, 6, and 3 items, was run separately for gender, cybervictims, and non-cybervictims. In all cases Mardias' coefficient was higher than 5 (girls = 30.19, boys = 23.35; non-cybervictims = 37.73, cybervictims = 10.81). As we can see in Table 3, the model showed optimal indexes in all cases proving to being a good model in these subgroups. We should point out that in the case of the cybervictims group, the RMSEA index was at the limit of the cutoff criteria. However, given the small number of students in this group and that the rest of the indexes were in the optimal range, it could be considered an adequate model (Table 3).

Table 3.

Fit Statistics for Each Group, Boys and Girls, and Cybervictims and Non-Cybervictims, for the Three-Factor Model

Model	χ²_S-B	df	NNBBFI	RCFI	RMSEA (90% CI)
Girls (n = 537)	456.6527	132	0.960	0.965	0.075 (0.067, 0.082)
Boys (n = 479)	392.5043	132	0.957	0.963	0.072 (0.063, 0.080)
N-CV (n = 815)	534.7964	132	0.969	0.973	0.068 (0.062, 0.073)
CV (n = 201)	263.0362	132	0.949	0.956	0.081 (0.066, 0.095)

CV, cybervictims; N-CV, non-cybervictimization.

Invariance of the models by gender

We then went on to assess configural, measurement, and structural factorial invariance across gender.

Robust support for the configural, measurement, and structural model invariance of the three-factor model across gender was found. Table 4 shows how the fit indices suggested that all models, from the least to the most restrictive, had optimal fit. Furthermore, the scaled Satorra-Bentler test showed the absence of significant differences among the configural (model 1) and measurement (model 2) models, and among equal factor loadings, factor variances and covariances model (model 3) and configural model (model 1), which supported the three types of invariance (Table 4).

Table 4.

Fit Statistics for the Models to Testing Invariance Across Gender (Sample Total)

Model	χ² (Δ χ²)	df (Δdf)	p	NNBBFI	RCFI	RMSEA (90% CI)
Model 1	1840.666	264		0.959	0.964	0.074 (0.068, 0.074)
Model 2	1857.590 (8.12)	279 (15)	0.92	0.961	0.965	0.071 (0.066, 0.077)
Model 3	1909.836 (31.63)	285 (21)	0.06	0.961	0.964	0.071 (0.066, 0.077)

Model 1: Configural invariance. Model 2: Measurement invariance (equal factor loadings). Model 3: Structural model invariance (equal factor loadings, factor variances and covariances).

χ², Chi-square statistic.

Invariance of the models by cybervictimization (cybervictims vs. non-cybervictims)

The same kinds of analyses were run to analyze configural, measurement, and structural factorial invariance across cybervictimization comparing cybervictims to non-cybervictims.

Once again, robust support was found for the three types of invariance: configural, measurement, and structural model, across cybervictimization (Table 5).

Table 5.

Fit Statistics for the Models to Testing Invariance Across Cybervictims and Non-Cybervictims (Sample Total)

Model	χ² (Δ χ²)	df (Δdf)	p	NNBBFI	RCFI	RMSEA (90% CI)
Model 1	1690.261	264		0.964	0.969	0.071 (0.066, 0.077)
Model 2	1701.420 (5.96)	279 (15)	0.98	0.966	0.969	0.069 (0.064, 0.074)
Model 3	1735.492 (17.22)	285 (21)	0.70	0.968	0.970	0.068 (0.062, 0.073)

Model 1: Configural invariance. Model 2: Measurement invariance (equal factor loadings). Model 3: Structural model invariance (equal factor loadings, factor variances and covariances).

Internal consistency

The internal reliability of the total scale and each subscale, assessed using the Rho coefficient and CR, was strong enough with the following index: 0.89 for the total scale; 0.92 and CR = 0.90 for depressed; 0.89 for active and the same CR; and 0.81 and CR = 0.84 for annoyed. Regarding the AVE, the values were from “close” in the case of depressed = 0.49 to good in the factors active = 0.56 and annoyed = 0.64.

Discussion

The assessment of the emotional impact of cyberbullying on victims is a relevant topic that has not been dealt with previously in research. Despite the proved relevance of emotions in victims' response to aggressions, which could influence the course of the cyberbullying,^26,27 there is still no instrument to assess this issue with adequate and validated psychometric properties. In this study, we have tested a scale to assess this impact, not only in cybervictims but also in the perception of the other people involved.

The results of the psychometric analyses show the validity of the instrument and its robustness. It proved useful not just for assessing the emotion felt by victims but also for perceiving the emotions in non-victims and it was equally valid for boys and girls. In addition, the self-perception of emotional impact of the people that have suffered cyberagression could be a relevant criterion to establish what is (or not) cybervictimization.

The instrument also broadens the scope of assessment of emotional impact by not only including negative emotions but also other emotions more related to activation. The three factors obtained cover a range of emotions that have usually been related to different kinds of coping strategies. Emotions related to anger, the annoyance factor in this instrument, seem to facilitate strategies such as taking revenge; emotions related to fear, the so-called depressed factor in this instrument, facilitate emotions related to help-seeking.²⁶ In addition, we could hypothesize that the other factor, active, may be related to a more problem-focused strategy given that it consists of emotions that help people to mobilize their resources. In fact, the depressed and annoyed factors are directly related to each other, whereas the active factor is inversely related, which points to not just different but “antagonistic” emotions. Although this hypothesis has not been proved, as it was not the focus of this study, the results of the previously mentioned studies suggest that this could be a useful research pathway for prevention and intervention with cybervictims.

As in all research, there were strengths and weaknesses in the research process and methodology used. Overall, the limitations stemmed from the intentional sampling and the kind of assessment instrument, a self-reported scale. Regarding the former, although from a strictly statistical point of view we cannot say that the results are generalizable, in practice, the fact of having included several state schools from several cities, and having collected data from complete classrooms of students reinforce the generalizable character of the data. As for the self-reported scale, the subjectivity of the self-report instrument is necessary because to intervene we need to know the subjective perception of the person involved beyond the “objective fact” that has happened. Another limitation to mention is the lack of use of other instruments to evaluate the validity of the construct “emotional impact.” Although this would be desirable, to date, according to our knowledge, there are no specific instruments to assess this construct. So, this points out to a future research area. In this sense, we need to remark that although almost all indexes obtained support the good psychometric properties of the scale, the close value of the AVE regarding the depressed factor, makes us cautious. However, we agree with Borsboom et al.⁴¹ about the importance of differentiating between validation, which could be tested analyzing the scores of the attributes measured, and validity, which implies a deeper, theoretical and even ontological analysis.

Future research should take into account other variables that might be relevant for the topic we are dealing with. It would be interesting to assess the combined role of traditional bullying and cyberbullying, given that some studies have identified considerable overlapping between these phenomena,^42–44 with some finding an even greater impact when victims are involved in both phenomena.^18,44

This research contributes significantly to the study of the emotional consequences of cyberbullying on victims, offering an instrument that assesses not only the cybervictims' perceptions but also the perceptions of other people involved. This could be extremely useful for prevention strategies. An adjusted perception about the suffering of the victims could be used to reinforce the empathy of the cyber perpetrators and also that of the spectators, which might move people to get involved to stop these aggressions or, at least, not take part in them (e.g., resending offending messages, such as humiliating images). In turn, a clear understanding of the emotional impact on current victims may help them to manage it adequately. Besides, it could be interesting to study the emotional impact on victims as useful criterion to distinguish what is cyberbullying from what is not.

Footnotes

Acknowledgments

This work was supported by the National Research Plan (Government of Spain) (Project “Sexting, ciberbullying y riesgos emergentes en la red: claves para su comprensión y respuesta educativa” [Sexting, cyberbullying and emerging risks on the internet: keys for their understanding and educational response] [EDU2013-44627-P]); the Science, Innovation, Economy and Employment Office (Consejería de Economía, Innovación, Ciencia y Empleo) of the Andalusia Government [Project “Coping with cyberbullying: analysis of strategies used and evaluation of their impact (SEJ-6156)”]. L.N.R. has a grant for the Ministry of Education, Culture and Sport for University teacher trainning-FPU 2012. The authors are grateful for the support received.

Author Disclosure Statement

No competing financial interests exist.

References

David-Ferdon

, Feldman Hertz

. Electronic media, violence and adolescents: an emerging public health problem. Journal of Adolescent Health, 2007; 41:S1–S5.

Smith

, Mahdavi

, Carvalho

, et al. Cyberbullying: its nature and impact in secondary school pupils. Journal of Child Psychology and Psychiatry, 2008; 49:376–385.

Garaigordobil

Prevalencia y consecuencias del cyberbullying: una revision. International Journal of Psychology and Psychological Therapy, 2011; 11:233–254.

Kowalski

, Giumetti

, Schroeder

, et al. Bullying in the digital age: a critical review and meta-analysis of cyberbullying research among youth. Psychological Bulletin, 2014; 140:1073–1137.

Patchin

, Hinduja

Summary of our cyberbullying research (2004–2016). Cyberbullying Research Center. https://cyberbullying.org/summary-of-our-cyberbullying-research (accessed November 2016 ).

Olweus

School bullying: development and some important challenges. Annual Review of Clinical Psychology, 2013; 9:751–780.

Gámez-Guadix

, Gini

, Calvete

. Stability of cyberbullying victimization among adolescents: prevalence and association with bully–victim status and psycho-social adjustment. Computers in Human Behavior, 2015; 53:140–148.

Olenik-Shemesh

, Heiman

, Eden

. Cyberbullying victimisation in adolescence: relationships with loneliness and depressive mood. Emotional and behavioural difficulties, 2012; 17:361–374.

Sontag

, Clemans

, Graber

, et al. Traditional and cyber aggressors and victims: a comparison of psychosocial characteristics. Journal of Youth and Adolescence, 2011; 40:392–404.

10.

Beckman

, Hagquist

, Hellström

. Does the association with psychosomatic health problems differ between cyberbullying and traditional bullying?. Emotional and Behavioural Difficulties, 2012; 17:421–434.

11.

Carter

JM.

Examining the relationship among physical and psychological health, parent and peer attachment, and cyberbullying in adolescents in urban and suburban environments. PhD dissertation, Wayne State University, 2011.

12.

Didden

, Scholte

RHJ

, Korzilius

, et al. Cyberbullying among students with intellectual and developmental disability in special education settings. Developmental Neurorehabilitation, 2009; 12:146–151.

13.

Vieno

, Gini

, Santinello

. Different forms of bullying and their association to smoking and drinking behavior in Italian adolescents. Journal of School Health, 2011; 81:393–399.

14.

Schneider

, O'Donnell

, Stueve

, et al. Cyberbullying, school bullying, and psychological distress: a regional census of high school students. American Journal of Public Health, 2012; 102:171–177.

15.

Bauman

, Toomey

, Walker

. Relations among bullying, cyberbullying and suicide in high school students. Journal of Adolescence, 2013; 36:341–350.

16.

Gini

, Espelage

. Peer victimization, cyberbullying, and suicide risk in children and adolescents. JAMA, 2014; 312:545–546.

17.

Van Geel

, Vedder

, Tanilon

. Relationship between peer victimization, cyberbullying, and suicide in children and adolescents: a meta-analysis. JAMA Pediatrics, 2014; 168:435–442.

18.

Giménez Gualdo

, Hunter

, Durkin

, et al. The emotional impact of cyberbullying: differences in perceptions and experiences as a function of role. Computers & Education, 2015; 82:228–235.

19.

Sahin

The relationship between the cyberbullying/cybervictimization and loneliness among adolescents. Children and Youth Services Review, 2012; 34:834–837.

20.

Ortega

, Elipe

, Mora-Merchán

, et al. The emotional impact on victims of traditional bullying and cyberbullying: a study of Spanish adolescents. Zeitschrift für Psychologie/Journal of Psychology, 2009; 217:197–204.

21.

Ortega

, Elipe

, Mora-Merchán

, et al. The emotional impact of bullying and cyberbullying on victims: a European cross-national study. Aggressive Behavior, 2012; 38:342–356.

22.

Dredge

, Gleeson

, Garcia

. Cyberbullying in social networking sites: an adolescent victim's perspective. Computers in Human Behavior, 2014; 36:13–20.

23.

McVie

The impact of bullying perpetration and victimization on later violence and psychological distress: a study of resilience among a Scottish youth cohort. Journal of School Violence, 2014; 13:39–58.

24.

Ortega

, Elipe

, Monks

. The emotional responses of victims of cyberbullying: worry and indifference. British Journal of Educational Psychology Monograph Series, 2012; 9:139–153.

25.

Jacobs

NCL

, Dehue

, Völlink

, et al. Determinants of adolescents' ineffective and improved coping with cyberbullying: a Delphi study. Journal of Adolescence, 2014; 37:373–385.

26.

Kochenderfer-Ladd

Peer victimization: the role of emotions in adaptive and maladaptive coping. Social Development, 2004; 13:329–349.

27.

Hunter

, Boyle

JME

, Warden

. (2006) Emotion and coping in young victims of peer-aggression. In Buchwald

, ed. Stress and anxiety—application to health, community, work place and education. Newcastle, UK: Cambridge Scholar Press, pp. 307–324.

28.

Ortega

, Elipe

, Calmaestra

. Emociones de agresores y víctimas de cyberbullying. Un estudio preliminar en estudiantes de secundaria. Ansiedad y Estrés, 2009; 15:151–165.

29.

Watson

, Clark

, Tellegen

. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology, 1998; 54:1063–1070.

30.

Joiner

, Sandín

, Chorot

, et al. Development and factor analytic validation of the SPANAS among women in Spain: cross-culture convergence in the structure of mood. Journal of Personality Assessment, 1997; 68:600–615.

31.

Sandin

, Chorot

, Lostao

, et al. Escalas de afecto positivo y negativo: validación factorial y convergencia transcultural. Psicothema, 1999; 11:37–51.

32.

Elipe

, Mora-Merchán

, Ortega-Ruiz

, et al. Perceived emotional intelligence as a moderator variable between cybervictimization and its emotional impact. Frontiers in Psychology, 2015; 6:486.

33.

Ortega-Ruiz

, Del Rey

, Casas

. Evaluar el bulling y el ciberbullying. Validación española del EBIP-Q y del ECIP-Q. Psicología Educativa, 2016; 22:71–79.

34.

Herrera-López

, Casas

, Romera

, et al. Validation of the European cyberbullying intervention project questionnaire for Colombian adolescent. Cyberpsychology, Behavior, and Social Networking, 2017; 20:117–125.

35.

Montero

, León

. A guide for naming research studies in Psychology. International Journal of Clinical and Health Psychology, 2007; 7:847–862.

36.

Flora

, Curran

. An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 2004; 9:466–491.

37.

Bentler

PM.

(2006) EQS 6 Structural Equations Program Manual. Encino, CA: Multivariate Software, Inc.

38.

, Bentler

. Cut-off criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 1999; 6:1–55.

39.

Satorra

, Bentler

. A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 2001; 66:507–514.

40.

Fornell

, David

. Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 1981; 28:39–45.

41.

Borsboom

, Mellenbergh

, Van Heerden

. The concept of validity. Psychological Review, 2004; 11:1061–1071.

42.

Del Rey

, Elipe

, Ortega

. Bullying and cyberbullying: overlapping and predictive value of the co-occurrence. Psicothema, 2012; 24:608–613.

43.

Gradinger

, Strohmeier

, Spiel

. Traditional bullying and cyberbullying. Zeitschrift für Psychologie/Journal of Psychology, 2009; 217:205–213.

44.

Wachs

, Junger

, Sittichai

. Traditional, cyber and combined bullying roles: differences in risky online and offline activities. Societies, 2015; 5:109–135.