Structural Validity of the HBSC Bullying Measure: Self-Report Rating Scales of Youth Victimization and Perpetration Behavior

Abstract

Bullying involvement among youth has consistently been linked to potentially serious consequences for both perpetrators and victims. To help clarify the nature and scope of youth bullying involvement, empirically validated assessment instruments measuring victimization and perpetration behaviors are needed for use in research and practice. The present study investigated the latent factor structure of the 22 victimization and perpetration items within the 2009-2010 Health Behavior in School-Aged Children (HBSC) self-report survey. Structural validity analyses were conducted using a representative sample of U.S. youth in Grades 5 to 10 (N = 11,449) obtained from the national administration of the HBSC self-report survey. Results suggested a two-factor latent structure comprised of bullying victimization and perpetration was the most theoretically and psychometrically sound measurement model for these data. In addition, multigroup measurement and structural invariance analyses showed that this model functioned equitably across student race/ethnicity, sex, and grade level, supporting the measure’s use with diverse student populations.

Keywords

bullying social and educational environment scale development/testing measurement structural equation modeling elementary school middle school high school

The issue of youth bullying involvement has received a great deal of research and media attention over the past few decades (Olweus, 2010) and remains a salient topic of study (Hymel & Swearer, 2015). Part of the draw to study this phenomenon is the well established link between bullying involvement and the numerous deleterious effects on youths’ social, academic, and psychological health (Jimerson, Nickerson, Mayer, & Furlong, 2012). For example, students who bully others—referred to hereafter as perpetrators—show more aggressive and dominant behavior patterns, exhibit fewer prosocial behaviors, and have elevated rates of externalizing behavior problems (Cook, Williams, Guerra, Kim, & Sadek, 2010; Veenstra, Lindenberg, Oldehinkel, De Winter, Verhulst, & Ormel, 2005). On the contrary, students who are the targets of bullying—referred to hereafter as victims—show higher levels of internalizing behavior problems and are markedly less likely to attend and perform well in school (Cook et al., 2010; Haynie et al., 2001; Kochenderfer & Ladd, 1996; Swearer, Espelage, Vaillancourt, & Hymel, 2010).

The many negative effects associated with bullying involvement have been found across demographics, including sex, race/ethnicity, school grade level, and nationality (Nansel, Craig, Overpeck, Saluja, & Ruan, 2004; Nansel et al., 2001; Renshaw, Roberson, & Hammons, 2016). At the same time, some longitudinal studies of prevalence rates have shown that bullying involvement has been decreasing in the United States over the last several years, perhaps due in part to increased awareness of this public health issue and rise in antibullying legislation (see stopbullying.gov). Although this may suggest reason for optimism, some estimated rates of bullying involvement are nonetheless intolerably high, with certain studies suggesting as many as 60.4% of youth endorsing involvement as a perpetrator and/or victim to some degree (Renshaw, Hammons, & Roberson, 2016).

Due to the elevated risks connected with bullying involvement, it is an important precondition to prevention and intervention that researchers and school professionals use technically adequate instruments to measure youths’ perpetration and victimization. Although some recent meta-analytic reviews of instruments for assessing bullying involvement show that more than 40 measures are currently available (Vessey, Strout, DiFazio, & Walker, 2014), these studies indicate major discrepancies in conceptual approaches to bullying measurement as well as varied evidence quality in regard to measurement validity (Vivolo-Kantor, Martell, Holland, & Westby, 2014). This variability in validity evidence makes it difficult to reach consensus about the nature and scope of youths’ bullying involvement (e.g., Renshaw, Hammons, & Roberson, 2016). Furthermore, these issues pose a challenge for practitioners in selecting appropriate assessment tools for screening or progress monitoring intervention and prevention efforts (Bradshaw, 2015). To remedy these issues, additional studies aimed at developing conceptually coherent and psychometrically robust measures of youths’ bullying victimization and perpetration are essential.

Although no single instrument has been completely adequately studied, Vessey et al. (2014) have suggested that the revised Olweus Bully/Victim Questionnaire (OBVQ; Olweus, 1996) is perhaps the most technically adequate and commonly used bullying measure to date. Whereas some other self-report bullying scales used for general prevalence estimation utilize only a scarce few items—such as the two items concerning school victimization in the Youth Risk Behavior Surveillance Survey (YRBSS; Centers for Disease Control and Prevention, 2014) or the single victimization item in the National Crime Victimization Survey (NCVS; Bureau of Justice Statistics, 2015)—the OBVQ assesses both perpetration and victimization using multiple items targeting many classes of bullying behavior (i.e., verbal, physical, relational, racial, cyber, and general). This breadth makes the OBVQ one of the most conceptually representative instruments available.

In addition, given the evidence that both bullying perpetration and victimization are associated with overlapping yet distinct negative outcomes (Nansel et al., 2004; Renshaw, Roberson, & Hammons, 2016), the ability to derive separate scores for both perpetration and victimization is an important feature of an instrument that intends to enhance the precision of risk estimates. Assessing bullying perpetration in conjunction with victimization also allows for the identification of students who experience both types of bullying (i.e., perpetrator–victims), whom research has identified as having even poorer outcomes compared with students involved in bullying as perpetrators-only or victims-only (Cook et al., 2010; Farmer et al., 2010; Veenstra et al., 2005). Although several existing measures use single, domain-general items to assess overall victimization and perpetration behavior—like those on the YRBSS or NCVS—research suggests that items targeting specific bullying behaviors yield higher prevalence rates, suggesting improved identification sensitivity (Renshaw, Hammons, & Roberson, 2016). Furthermore, it seems reasonable to suggest that measures composed of items targeting several specific bullying behaviors would have both greater usability (see Glover & Albers, 2007) and treatment utility (see Hayes, Nelson, & Jarrett, 1987) for practitioners using data obtained from these instruments to inform bullying prevention and intervention in schools.

Given the context sketched above, the purpose of the present study was to investigate the structural validity of an underresearched measure that targets both specific bullying victimization and perpetration behaviors: the self-report bullying items within the Health Behavior in School-Aged Children (HBSC) survey, sponsored by the World Health Organization (2014). These HBSC self-report items were modeled after the OBVQ and expanded in content to target parallel classes of bullying victimization and perpetration behaviors (e.g., physical victimization and physical perpetration). Although several studies investigating bullying involvement have been conducted using isolated items from HBSC self-report surveys (e.g., Nansel et al., 2004; Renshaw, Hammons, & Roberson, 2016), no study has yet to investigate the structural validity of the entire suite of bullying-related items. Thus, the purpose of the present study was to test the latent structure and invariance of responses to the HBSC self-report bullying items. Given that an item-content evaluation suggested the HBSC self-report survey targets at least two overarching classes of bullying behavior—victimization and perpetration—the following hypotheses were proposed to guide the study:

Hypothesis 1: Exploratory factor analysis (EFA) of responses to the HBSC bullying items will yield two distinct latent factors: victimization and perpetration.

Hypothesis 2: Confirmatory factor analysis (CFA) of responses to this same set of items will corroborate the two-factor measurement model yielded by EFA.

Hypothesis 3: Multigroup CFA of responses to the two-factor measurement model will demonstrate invariance across several key student demographic factors: grade level, sex, and race/ethnicity.

Method

Participants

The current study utilized the publicly available 2009-2010 HBSC self-report survey dataset, which is composed of a stratified random sample of U.S. youth. A detailed description of the student sampling process and other information concerning the design and administration of the survey can be found in the publicly available codebook (Iannotti, 2013). A summary of participant demographic characteristics for the original full sample (N = 11,449), the full sample with cases containing missing data on one or more of the bullying items removed (n = 9,979), and two random split-half subsamples derived for data analytic purposes is provided in Table 1. Participants in the original full sample identified approximately equally as male and female were enrolled in Grades 5 to 10 (grade-level representation range = 13.7%-20.3%), and approximated racial/ethnic proportions of U.S. youth, with the most prevalent identities being White (49.6%), Hispanic/Latinx (17.7%), and Black/African American (15.3%). Most respondents indicated living in a suburban area (36.2%), followed by urban (30.4%), rural (25.7%), and unclassified areas (7.8%). Respondent scores on a composite measure of family affluence ranging from 0 = low to 9 = high were mostly in the upper-middle range of the scale (M = 5.96, Mdn = 6, Q₁ = 5, Q₃ = 7). Compared with the original full sample, relative proportions of each demographic were largely maintained across all other iterations of the sample (see Table 1).

Table 1.

2009-2010 HBSC Demographic Proportions for Sex, Grade Level, and Race/Ethnicity Across All Samples.

	Full sample		Full sample after removing cases		Subsample 1		Subsample 2
	n	%	n	%	n	%	n	%
Total sample	11,449	100	9,979	100	4,989	100	4,990	100
Sex
Male	5,883	51.4	5,048	50.6	2,518	50.5	2,530	50.7
Female	5,562	48.6	4,930	49.4	2,470	49.5	2,460	49.3
Missing data	4	0.0	1	0.0	1	0.0	0	0.0
Grade level
5th	1,574	13.7	1,207	12.1	617	12.4	590	11.8
6th	1,781	15.6	1,524	15.3	750	15.0	774	15.5
7th	2,269	19.8	1,963	19.7	966	19.4	997	20.0
8th	2,327	20.3	2,061	20.7	1,010	20.2	1,051	21.1
9th	1,817	15.9	1,662	16.7	855	17.1	807	16.2
10th	1,681	14.7	1,562	15.7	791	15.9	771	15.5
Race/ethnicity
American Indian/Alaskan Native	206	1.8	176	1.8	83	1.7	93	1.9
Asian	448	3.9	406	4.1	194	3.9	212	4.2
Black/African American	1,841	16.1	1,522	15.3	771	15.5	751	15.1
Hispanic/Latinx	2,097	18.3	1,768	17.7	861	17.3	907	18.2
Native Hawaiian/other Pacific Islander	105	0.9	89	0.9	48	1.0	41	0.8
White	5,485	47.9	4,953	49.6	2,512	50.5	2,441	48.9
Two or more	757	6.6	679	6.8	334	6.7	345	6.9
Missing data	510	4.5	386	3.9	186	3.7	200	4.0

Note. HBSC = Health Behavior in School-Aged Children.

Measure

The HBSC self-report survey is administered to a nationally representative sample of U.S. students in Grades 5 to 10 every 4 years. The survey assesses a broad range of health-related behaviors among youth, including drug/alcohol use, body image, attitudes about school, peer/family relationships, physical health, bullying involvement, and more. To operationalize “bullying” for respondents, a prompt with a formal definition—adapted from the OBVQ—was provided at the beginning of the item set:

We say a student is BEING BULLIED when another student, or a group of students, say or do nasty or unpleasant things to him or her. It is also bullying when a student is teased repeatedly in a way her or she does not like or when they are deliberately left out of things. But it is NOT BULLYING when students of about the same strength or power argue or fight. It is also not bullying when a student is teased in a friendly and playful way.

The HBSC self-report survey contains 22 items targeting specific classes of bullying behaviors—11 items for victimization (see Table 2) and 11 parallel items for perpetration (see Table 3)—covering teasing, social exclusion, physical aggression, spreading lies, various kinds of harassment, and multiple kinds of cyber bullying. The victimization and perpetration items were arranged in two-item subsets and were prefaced with a similar item stem (i.e., “How often have you [been bullied/bullied another students(s)] at this school in the past couple of months in the ways listed below?”). Response options for all bullying involvement items were arranged along a 5-point, relative frequency-based scale (1 = I haven’t been bullied at school the past couple of months, 2 = it has only happened once or twice, 3 = 2 or 3 times a month, 4 = about once a week, and 5 = several times a week).

Table 2.

2009-2010 HBSC Items Targeting Bullying Victimization.

Code	Item
vVerbal	“I was called mean names, was made fun of, or teased in a hurtful way.”
vExclusion	“Other students left me out of things on purpose, excluded me from their group of friends, or completely ignored me.”
vPhysical	“I was hit, kicked, pushed, shoved around, or locked indoors.”
vRelational	“Other students told lies or spread false rumors about me and tried to make others dislike me.”
vRacial	“I was bullied with mean names and comments about my race or color.”
vReligious	“I was bullied with mean names and comments about my religion.”
vSexual	“Other students made sexual jokes, comments or gestures to me.”
vComp	“I was bullied using a computer or email messages or pictures.”
vCell	“I was bullied using a cell phone.”
vCompOut	“I was bullied outside of school using a computer or email messages or pictures.”
vCellOut	“I was bullied outside of school using a cell phone.”

Note. HBSC = Health Behavior in School-Aged Children.

Table 3.

2009-2010 HBSC Items Targeting Bullying Perpetration.

Code	Item
pVerbal	“I called another student(s) mean names, made fun of, or teased him or her in a hurtful way.”
pExclusion	“I kept another student(s) out things on purpose, excluded him or her from my group of friends or completely ignored him or her.”
pPhysical	“I hit, kicked, pushed, shoved around, or locked another student(s) indoors.”
pRelational	“I spread false rumors about another student(s) and tried to make others dislike him or her.”
pRacial	“I bullied another student(s) with mean names or comments about his or her race or color.”
pReligious	“I bullied another student(s) with mean names or comments about his or her religion.”
pSexual	“I made sexual jokes, comments or gestures to another student(s).”
pComp	“I bullied another student(s) using a computer, email messages or pictures.”
pCell	“I bullied another student(s) using a cell phone.”
pCompOut	“I bullied others outside of school using a computer, email messages or pictures.”
pCellOut	“I bullied others outside of school using a cell phone.”

Note. HBSC = Health Behavior in School-Aged Children.

Data Analyses

The structural validity of responses to the HBSC self-report bullying involvement items was analyzed in multiple stages using the R statistical environment (R Core Team, 2016). Prior to the primary analyses, data inspection and cleaning procedures were undertaken, beginning with handling missing data. First, 647 cases (5.7%) from the original full sample were deleted listwise as they had missing data across all bullying items of interest, suggesting respondents failed to complete this section of the survey entirely. Of the remaining cases, 823 (7.2%) had partially missing data for the relevant variables and were also deleted listwise. Because the number of cases with partially missing responses accounted for less than 10% of the sample, listwise deletion methods were deemed acceptable and unlikely to significantly bias statistical estimates (Langkamp, Lehman, & Lemeshow, 2010). The remaining cases were then divided into two random split-halves for analytic purposes. Subsample 1 (S1; n = 4,989) was used to conduct EFA, whereas Subsample 2 (S2; n = 4,990) was used to conduct CFA and multigroup invariance analyses. Descriptive statistics were calculated for both subsamples to investigate the distributionality of the target items and are provided in Table 4. Skewness and kurtosis estimates were consistently greater than |2.0| for all items, indicating significant nonnormality of the distribution. Due to the relative-frequency nature of the response options for the bullying items, responses were treated as ordered categorical data for all primary analyses.

Table 4.

Descriptive Statistics for All Bullying Items in Subsamples 1 and 2.

	Subsample 1						Subsample 2
	M	SD	Mdn	Q₁, Q₃	Skew.	Kurt.	M	SD	Mdn	Q₁, Q₃	Skew.	Kurt.
vVerbal	1.56	1.09	1	1, 2	2.13	3.59	1.56	1.07	1	1, 2	2.14	3.70
vExclusion	1.44	0.97	1	1, 1	2.50	5.57	1.44	0.96	1	1, 1	2.51	5.70
vPhysical	1.26	0.76	1	1, 1	3.50	12.26	1.25	0.78	1	1, 1	3.57	12.66
vRelational	1.53	1.03	1	1, 2	2.22	4.18	1.51	1.02	1	1, 2	2.25	4.37
vRacial	1.25	0.81	1	1, 1	3.62	12.66	1.24	0.78	1	1, 1	3.71	13.52
vReligious	1.17	0.65	1	1, 1	4.52	20.94	1.17	0.67	1	1, 1	4.54	20.91
vSexual	1.42	0.98	1	1, 1	2.57	5.77	1.42	0.99	1	1, 1	2.60	5.85
vComp	1.13	0.56	1	1, 1	5.25	28.97	1.15	0.61	1	1, 1	4.87	24.53
vCell	1.13	0.56	1	1, 1	5.34	30.11	1.13	0.58	1	1, 1	5.09	26.95
vCompOut	1.13	0.56	1	1, 1	5.20	28.65	1.14	0.61	1	1, 1	4.99	25.68
vCellOut	1.12	0.55	1	1, 1	5.41	30.75	1.14	0.60	1	1, 1	5.04	26.10
pVerbal	1.35	0.80	1	1, 1	2.94	9.19	1.37	0.83	1	1, 1	2.90	8.72
pExclusion	1.24	0.69	1	1, 1	3.67	14.70	1.26	0.73	1	1, 1	3.51	13.02
pPhysical	1.19	0.66	1	1, 1	4.21	18.61	1.19	0.67	1	1, 1	4.15	17.91
pRelational	1.15	0.59	1	1, 1	4.71	23.66	1.15	0.59	1	1, 1	4.83	25.04
pRacial	1.13	0.57	1	1, 1	5.17	28.20	1.13	0.58	1	1, 1	5.22	28.65
pReligious	1.11	0.53	1	1, 1	5.74	34.64	1.11	0.55	1	1, 1	5.66	33.34
pSexual	1.17	0.64	1	1, 1	4.45	20.65	1.17	0.65	1	1, 1	4.51	20.96
pComp	1.10	0.51	1	1, 1	6.03	38.22	1.10	0.51	1	1, 1	6.20	40.22
pCell	1.10	0.51	1	1, 1	5.89	36.69	1.10	0.52	1	1, 1	5.98	37.82
pCompOut	1.10	0.51	1	1, 1	5.93	37.08	1.11	0.54	1	1, 1	5.84	35.44
pCellOut	1.10	0.51	1	1, 1	6.15	39.88	1.11	0.54	1	1, 1	5.85	35.82

Note. Q₁, Q₃ = first and third quartile score; Skew. = skewness; Kurt. = kurtosis.

EFA were conducted using the psych (Revelle, 2016) and nFactors (Raiche, 2010) R packages. Rather than analyzing the more traditional Pearson correlation matrix, the polychoric correlation matrix was analyzed, as simulation studies have demonstrated the superiority of this approach in estimating the population Pearson values when data are ordinal (Holgado-Tello, Chacón-Moscoso, Barbero-García, & Vila-Abad, 2010). Furthermore, given the nonnormality of these data, a principal axis factoring extraction method was selected, as it is more robust to violations of normality than maximum likelihood. To enhance factor interpretation and account for association among extracted factors, an oblique factor rotation method (promax) was used. The number of factors to retain for the preferred measurement model was determined based on an array of analyses and decision rules, beginning with scree plot visualization of eigenvalues and parallel analysis (Garrido, Abad, & Ponsoda, 2013). Both analyses indicated that a two-factor solution was the best description of these data. In line with this result, a two-factor solution was constrained for the first model but additional three- and four-factor solutions were also tested to explore the potential of alternative structures. Statistical assumptions of the EFA were evaluated in terms of sampling adequacy (Kaiser–Meyer–Olkin [KMO] > .90), sphericity (χ² p < .05), and extracted item communalities (h² ≥ .50). In addition, rotated pattern matrix output for each EFA was inspected for strong factor loadings ≥ 0.3, small cross-factor loadings less than 0.3, and theoretical interpretability.

CFA and multigroup invariance analyses were conducted using the lavaan (Rosseel, 2012), semTools (semTools Contributors, 2016), and psych (Revelle, 2016) R packages. Following all EFA, the preferred measurement model was analyzed with a CFA using a robust diagonal weighted least squares estimator (DWLS; DiStefano & Morgan, 2014). Data–model fit was assessed using a variety of indices, including scaled-shifted variants of model chi-square (χ²; Asparouhov & Muthén, 2010), comparative fit index (CFI; Brosseau-Liard & Savalei, 2014), and root mean square error of approximation (RMSEA; Brosseau-Liard, Savalei, & Li, 2012), as well as the standardized root mean square residual (SRMR). The following criterion values were used to indicate at least adequate data–model fit: χ² with an associated p > .05 (Kline, 2016), CFI ≥ .90, RMSEA with 90% confidence interval (CI) ≤ .08, and SRMR ≤ .08 (Hu & Bentler, 1999). Of these, the χ² index is perhaps the least useful for determining fit quality, as it may be susceptible to bias from even minor misspecification when data are nonnormal and the sample size is large, as in the present study. However, this index is still reported herein for completeness.

In addition, it is noteworthy that research supporting the use of these decision rules has mostly been conducted using maximum likelihood estimation methods with normally distributed, continuous data. Research on appropriate cutoff values for robust DWLS estimation for nonnormal ordinal data is limited, but some evidence suggests that more conservative criteria of CFI ≥ .95 and RMSEA ≤ .05 may be more appropriate (DiStefano & Morgan, 2014). Due to the lack of consensus regarding fit index cutoff values for the conditions of this study, the traditional criteria were used to identify adequate fit and the stricter criteria were used to indicate strong fit, but conclusions should be regarded as tentative. Scale reliability was evaluated at the latent level using the H statistic (Mueller & Hancock, 2008) and with categorical ω for the observed level (Dunn, Baguley, & Brunsden, 2014; Kelley & Pornprasertmanit, 2016), where values ≥ .70 were considered adequate for both. H was calculated using the formula provided by Mueller and Hancock (Mueller & Hancock, 2008) whereas categorical ω was computed using the MBESS package in R (Kelley, 2016).

Once a good-fitting CFA model was identified, multigroup CFA was then conducted with the same subsample (S2) to investigate measurement and structural invariance across three demographic factors: grade level, sex, and race/ethnicity. Tests of measurement and structural invariance involved adding increasingly restrictive parameter constraints to the model. The order of invariance constraint level proceeded as follows: (a) configural/baseline (equal latent structure), (b) weak (adds equal factor loadings), (c) strong (adds equal intercepts), (d) homogeneity of latent variance (adds equal residuals and latent variances), and (e) homogeneity of latent means (adds equal latent means). Levels (a) through (c) refer properly to measurement invariance while levels (d) and (e) concern structural invariance (Vandenberg & Lance, 2000). Relative invariance from less to more constrained models was determined based on change in CFI ≤ .01 (Vandenberg & Lance, 2000). However, it is noteworthy that little research is available offering guidance on invariance evaluation criteria when using a robust DWLS estimator for nonnormal ordinal data. Another common test for invariance is the change in model χ² from less to more restrictive models. However, given that there is disagreement in the literature regarding the appropriateness of this test when using estimation methods similar to robust DWLS (e.g., WLSMV; Asparouhov & Muthén, 2006; cf. French & Finch, 2008), we have decided to forego this approach in favor of the change in CFI for the current investigation. Invariance results should therefore also be interpreted with appropriate caution and considered tentative. Finally, once an appropriate factor structure was established with S2, subscale scores were calculated at the observed level to investigate their descriptive properties.

Results

Latent Factor Structure

EFA (S1)

Results of the first analysis, which constrained a two-factor solution, showed strong KMO sampling adequacy (.94), lack of singularity—χ²(231) = 79,308, p < .001—and adequate extracted item communalities (h² range = .56-.94). Table 5 summarizes the factor loading output and psychometric characteristics of this measurement model. Eigenvalues for the two factors collectively accounted for 75.8% of the variance in these data. Pattern matrix factor loadings for all bullying victimization (λ range = .74-.89) and bullying perpetration (λ range = .77-.95) items were greater than the minimum threshold of .30, with no significant cross-loadings. In addition, observed internal consistency estimates were strong for both factors (ω > .90), while also showing a strong latent factor correlation (ϕ = .53).

Table 5.

EFA Pattern Matrix Results for the Two-Factor Measurement Model.

Item	Factor loadings (λ)
Item	Perpetration (ξ₁)	Victimization (ξ₂)
pReligious	0.946	0.003
pComp	0.944	0.045
pCompOut	0.935	0.045
pRacial	0.931	−0.029
pCell	0.931	0.062
pCellOut	0.910	0.065
pSexual	0.869	0.011
pPhysical	0.862	−0.003
pRelational	0.859	0.084
pExclusion	0.831	−0.028
pVerbal	0.772	−0.042
vRelational	−0.162	0.889
vVerbal	−0.234	0.889
vExclusion	−0.148	0.872
vPhysical	−0.001	0.799
vRacial	0.044	0.782
vReligious	0.117	0.781
vSexual	−0.021	0.775
vComp	0.212	0.758
vCell	0.229	0.739
vCompOut	0.237	0.738
vCellOut	0.229	0.736
Eigenvalues	9.36	7.32
% variance	42.50	33.30
ω (95% CI)	.96 [.96, .96]	.99 [.99, .99]
r	.61	.47

Note. Bold factor loadings distinguish items in the perpetration and victimization scales.

ω =categorical omega; CI = confidence interval; r = average interitem correlation. The 95% CIs for categorical omega were calculated from 10,000 draws of a bias corrected and accelerated bootstrapping procedure.

Next, a second EFA constraining a three-factor solution was tested. Results from this model showed strong communalities among all items (h² range = .56-.94) and the three factors collectively accounted for 80.5% of the variance. Inspection of the pattern matrix revealed the three factors consisted of (a) 11 items related to perpetration (λ range = .82-.93), (b) seven items related to traditional (i.e., noncyber) victimization (λ range = .55-1.00), and (c) four items pertaining to cyber victimization (λ range = .69-.72). Although these loadings were strong, interpretation of the factors was not as clear as with the two-factor model, given that five items cross-loaded on other factors: pVerbal, pCompOut, pCellOut, vReligious, and vComp (see Tables 2 and 3 for item content).

Finally, a four-factor EFA solution was tested. As with both previous models, the four-factor solution showed uniformly strong communalities (h² range = .61-.96) and collectively accounted for 82.4% of the variance. Pattern matrix inspection revealed that the first three extracted factors represented the same factors as in the three-factor model—traditional victimization (λ range = .55-.96), perpetration (λ range = .82-.95), and cyber victimization (λ range = .72-.77)—plus a fourth factor where only the items vRacial (λ = .50) and vReligious (λ = .47) had factor loadings above .30. This model also showed three items with cross-loadings greater than .30: pCellOut, vRacial, and vReligious (see Tables 2 and 3 for item content). Considering these EFA results together, the preponderance of evidence converged in support of retaining the two-factor structure as the preferred measurement model for subsequent CFA and multigroup invariance analyses.

CFA (S2)

To corroborate the two-factor measurement model indicated by EFA, CFA was conducted with S2 by regressing each of the 22 observed bullying items onto their respective latent factors of perpetration and victimization and allowing both latent factors to freely covary. Findings showed the model was characterized by strong latent factor loadings for perpetration (λ range = .72-.97) and victimization (λ range = .74-.97), as well as a large covariance between the factors (ϕ = .69). Model fit indices also indicated adequate to strong data–model fit—χ²(208) = 3174.054, p < .001; CFI = .971; RMSEA = .053 (90% CI = [.052, .055]); SRMR = .064 (see Table 6 and Figure 1). In addition, both latent factors had strong internal reliability and average variance extracted (AVE): victimization H = .98 and AVE = .72, perpetration H = .99 and AVE = .82.

Table 6.

Factor Loading Coefficients, Latent Variance, and Latent Covariance for the Two-Factor HBSC Bullying CFA Model.

Parameter	Unstandardized	SE	Standardized	r_s
Factor loadings
Victimization
vVerbal	1.000	—	0.720	.496
vExclusion	1.011	.023	0.727	.501
vPhysical	1.103	.023	0.794	.547
vRelational	1.089	.022	0.784	.540
vRacial	1.124	.024	0.809	.557
vReligious	1.165	.025	0.838	.577
vSexual	1.088	.022	0.783	.539
vComp	1.309	.023	0.942	.649
vCell	1.338	.023	0.963	.664
vCompOut	1.321	.024	0.951	.655
vCellOut	1.342	.023	0.966	.666
Perpetration
pVerbal	1.000	—	0.735	.506
pExclusion	1.113	.021	0.818	.564
pPhysical	1.151	.020	0.846	.583
pRelational	1.221	.022	0.897	.618
pRacial	1.243	.021	0.914	.630
pReligious	1.283	.022	0.943	.650
pSexual	1.203	.020	0.884	.609
pComp	1.323	.022	0.973	.670
pCell	1.327	.022	0.975	.672
pCompOut	1.324	.022	0.973	.670
pCellOut	1.310	.022	0.963	.664
Latent factor variance and covariance
Victimization	0.518	.018	1.000	—
Perpetration	0.540	.019	1.000	—
Covariance	0.364	.015	0.689	—

Note. Unstandardized and standardized factor loadings refer to the pattern coefficients. r_s refers to the structure coefficients. HBSC = Health Behavior in School-Aged Children; CFA = confirmatory factor analysis.

Figure 1.

Full measurement model of the two-factor HBSC Bullying Measure.

Measurement and structural invariance analysis (S2)

Results from the multigroup measurement and structural invariance analyses showed good data–model fit at all levels of constraint across student race/ethnicity, grade level, and sex (see Tables 7 and 8). ΔCFI was uniformly less than .01 for every level of invariance across all demographics. In addition, the strictest invariance level, homogeneity of latent means, was the best fitting model for all three demographics, providing further evidence for measurement and structural invariance across groups.

Table 7.

Model Fit of the Two-Factor CFA for Each Demographic Group Used for Multigroup Invariance Analyses.

Demographic	χ²	df	RMSEA (90% CI)	CFI	SRMR	λ_Vict. range	λ_Perp. range	St. Cov.
Race/ethnicity
White	1,301.635	208	.046 [.044, .049]	.974	.067	.74-.96	.75-.98	.60
Hispanic/Latinx	782.303	208	.055 [.051, .059]	.980	.064	.71-.97	.71-.98	.81
Black/African American	714.946	208	.057 [.052, .062]	.973	.064	.68-.97	.72-.97	.79
Grade level
5th	430.592	208	.043 [.037, .048]	.987	.082	.70-.99	.78-.99	.73
6th	581.301	208	.048 [.044, .053]	.986	.069	.73-.98	.75-.99	.77
7th	834.013	208	.055 [.051, .059]	.974	.072	.72-.97	.69-.97	.75
8th	752.777	208	.050 [.046, .054]	.974	.071	.70-.96	.72-.98	.59
9th	549.378	208	.045 [.041, .050]	.975	.071	.74-.97	.74-.98	.56
10th	601.716	208	.050 [.045, .054]	.988	.059	.70-.98	.78-1.00	.79
Sex
Male	1,936.457	208	.057 [.055, .060]	.975	.071	.74-.97	.74-.98	.66
Female	1,469.310	208	.050 [.047, .052]	.967	.065	.65-.97	.73-.98	.75

Note. χ², RMSEA, and CFI estimates are presented with scaled-shifted corrections applied. All χ² values were significant at the p < .001 level. CFA = confirmatory factor analysis; RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index; SRMR = standardized root mean square residual; λ_Vict. = λ victimization; λ_Perp. = λ perpetration; St. Cov. = standardized covariance between the victimization and perpetration latent variables.

Table 8.

Measurement and Structural Invariance Results for the Two-Factor Model Across Demographics.

Demographic/invariance level	χ²	df	RMSEA (90% CI)	CFI	ΔCFI
Race/ethnicity
Configural/baseline	2,818.169	624	.051 [.049, .053]	.974	—
Weak	2,275.960	664	.042 [.040, .044]	.981	.007
Strong	2,780.884	792	.043 [.041, .045]	.977	.004
Latent variance	2,684.975	796	.042 [.040, .043]	.978	.001
Latent means	2,059.119	800	.034 [.032, .036]	.985	.007
Grade level
Configural/baseline	3,646.476	1,248	.048 [.046, .050]	.983	—
Weak	2,996.939	1,348	.038 [.037, .040]	.988	.005
Strong	3,818.532	1,668	.039 [.038, .041]	.984	.004
Latent variance	3,708.599	1,678	.038 [.037, .040]	.985	.001
Latent means	2,736.109	1,688	.027 [.025, .029]	.992	.007
Sex
Configural/baseline	3,401.684	416	.054 [.052, .055]	.973	—
Weak	3,219.269	436	.051 [.049, .052]	.974	.001
Strong	3,483.713	500	.049 [.047, .050]	.974	.000
Latent variance	3,367.200	502	.048 [.046, .049]	.974	.000
Latent means	2,369.732	504	.039 [.037, .040]	.983	.009

Note. χ², RMSEA, and CFI estimates are presented with scaled-shifted corrections applied. All χ² values were significant at the p < .001 level. RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index.

Observed Subscale Descriptive Statistics

In addition to the latent-level analyses described above, basic descriptive statistics were also calculated with S2 at the observed level for total scores from the victimization and perpetration subscales (see Table 9). Results indicated that these scores were characterized by strong internal consistency (ω > .90) as well as moderate (victimization r = .47) and large (perpetration r = .60) average inter-item correlations. In addition, the scores showed significant nonnormality (skewness and kurtosis > |2.0|) and relatively low bullying involvement frequency for both perpetration (Mdn = 11) and victimization (Mdn = 12).

Table 9.

Observed Descriptive Statistics of the HBSC Bullying Measure.

Subscales	Items	Minimum, maximum	M	SD	Mdn	Q₁, Q₃	Skew.	Kurt.	r	ω (95% CI)
Victimization	11	11, 55	14.15	6.23	12	11, 14	3.31	13.24	.47	.97 [.97, .97]
Perpetration	11	11, 55	12.81	5.30	11	11, 12	5.02	29.64	.60	.96 [.96, .96]

Note. Q₁, Q₃ = first and third quartile score; r = average interitem correlation; ω (95% CI) = categorical omega with 95% confidence interval calculated with 10,000 draws from a bias corrected and accelerated bootstrapping procedure. HBSC = Health Behavior in School-Aged Children; Skew. = skewness; Kurt. = kurtosis; CI = confidence interval.

Discussion

Although several instruments assessing youth bullying involvement are currently available, to date, none appear as conceptually broad or as balanced across victimization and perpetration domains as the measure included within the HBSC self-report survey. The present study was the first to investigate the structural validity of responses to the 22 items that make up this measure, using the publicly available, nationally representative sample of U.S. youth in Grades 5 to 10 from the 2009-2010 administration of the survey. We hypothesized that results from factor analyses would support a two-factor measurement model that represented bullying victimization and perpetration, and that this model would be reasonably invariant across key demographics.

After testing the tenability of two-, three-, and four-factor EFA models, the results converged on the conclusion that the hypothesized two-factor model of bullying involvement was in fact the best structure described by the data. CFA results provided additional evidence in support of this two-factor measurement model, showing strong data–model fit evidenced by a variety of absolute and incremental fit indices. Furthermore, findings from the follow-up multigroup CFA supported the measurement and structural invariance of responses across student sex, race/ethnicity, and grade level. However, due to insufficient subgroup sample sizes for several racial/ethnic groups, invariance for race/ethnicity was only established for the three largest populations in the United States (i.e., White, Hispanic/Latinx, and Black/African American students) and should therefore not be generalized beyond these groups based on this study alone.

Taken together, findings from the present study largely supported our hypotheses and suggest that responses to the HBSC self-report bullying items demonstrate structural validity for measuring student bullying perpetration and victimization behaviors. Given the measurement and structural invariance evidence derived from a representative sample of U.S. students in the present analysis, no adjustments to scoring or changes to administration procedures appear necessary, at least when using the measure to assess bullying involvement among 5th- to 10th-grade students identifying as White, Hispanic/Latinx, or Black/African American. Furthermore, because this study utilized such a large and demographically diverse dataset, the descriptive statistics derived from the observed subscales scores (see Table 9) could potentially be used as normative data for informing future research or practice, such as for identifying students at risk of bullying involvement via universal screening or for progress monitoring the effectiveness of prevention or intervention efforts at the level of classrooms, grade levels, schools, districts, regions, or nationally.

Although it is possible to administer either the perpetration or victimization scale independently, we suggest that this is not the optimal use of the measure. Given the differentiated outcome trajectories of victims, perpetrators, and perpetrator–victims, we recommend using both scales in conjunction to assess all three levels of bullying involvement, rather than only one. In addition, it is noteworthy that the covariance between the perpetration and victimization latent variables was large (see Table 6), suggesting that there was a considerable rate of perpetrator–victims in our sample who are at risk of the most detrimental outcomes. Although a thorough analysis of prevalence rates is beyond the scope of this measure development study, interested readers are directed to Renshaw, Roberson, and Hammons (2016) and Renshaw, Hammons, and Roberson (2016) for more detailed investigation of prevalence rates and risk outcomes among different bullying involvement groups using the 2009-2010 HBSC sample.

Results from this study should also be considered in light of a few notable limitations. As mentioned earlier, the research supporting the use of the robust DWLS estimator for nonnormal ordinal data under various model and sample size conditions is still nascent. Although the considerably large sample might allay some concern about the functionality of the estimates, substantive conclusions drawn from these results should be considered tentative. It is also not known to what extent the descriptive results presented herein (see Table 9) reflect contemporary bullying involvement among U.S. youth, as we are currently 6 years removed from the responses of participants from the 2009-2010 sample used in this study. In addition, as an artifact of splitting the full sample into random halves, invariance analyses could only be conducted for the three most prevalent racial/ethnic groups, which limit the generalizability of findings for most racial/ethnic minority groups within the United States. Although future administration of the HBSC in the United States is not guaranteed, this does not preclude this measure’s use in later health-behavior and bullying prevalence rate studies in other contexts. Further research is therefore warranted to replicate these analyses, whether in the context of formal national HBSC administration or not, to capture changing trends in bullying involvement and expand the validity evidence of the measure to more diverse student populations.

Future research investigating the concurrent and predictive validity of responses to the HBSC bullying measure with other physical, psychological, and academic well-being indicators is also encouraged. This would allow for a greater understanding of which classes of bullying behavior represented by this measurement model are salient indicators of risk. Relatedly, this measure could also be covalidated with preexisting bullying instruments to test the comparative validity of responses to competing measures for predicting valued student outcomes. Future studies within this line of research might also benefit from using other data collection methodologies beyond self-report (e.g., school discipline records, informant-report behavior ratings), as it will alleviate potential bias from relying on a single data collection approach (see Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Similarly, randomizing the order in which items are presented to participants may also help curb biased responding, as some evidence suggests respondents may be primed to answer differently depending on the phrasing of previous questions (Huang & Cornell, 2015). Finally, in addition to basic psychometric validation studies, research investigating the treatment utility (see Hayes et al., 1987) of scores derived from the HBSC bullying measure is needed to inform use of the instrument for actual prevention and intervention purposes.

Footnotes

Acknowledgements

The authors are grateful to Ronald J. Iannotti for making available the Health Behavior in School-Aged Children survey datasets and their codebooks for public use.

Authors’ Note

All data, R analysis code, and research report documentation for the present study are freely available at

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Asparouhov

Muthén

(2006). Robust chi square difference testing with mean and variance adjusted test statistics. Retrieved from www.statmodel.com/download/webnotes/webnote10.pdf

Asparouhov

Muthén

(2010). Simple second order chi-square correction. Retrieved from http://www.statmodel.com/download/WLSMV_new_chi21.pdf

Bradshaw

C. P.

(2015). Translating research to practice in bullying prevention. American Psychologist, 70, 322-332. doi:10.1037/a0039114

Brosseau-Liard

P. E.

Savalei

(2014). Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49, 460-470. doi:10.1080/00273171.2014.93369

Brosseau-Liard

P. E.

Savalei

(2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47, 904-930. doi:10.1080/00273171.2012.715252

Bureau of Justice Statistics. (2015). Indicators of school crime and safety. Washington, DC: Author. Retrieved from www.bjs.gov/index.cfm?ty=pbse&;sid=8

Centers for Disease Control and Prevention. (2014). Youth Risk Behavior Surveillance-United States, 2013. Retrieved from http://www.cdc.gov/mmwr/pdf/ss/ss6304.pdf

Cook

C. R.

Williams

K. R.

Guerra

N. G.

Kim

T. E.

Sadek

(2010). Predictors of bullying and victimization in childhood and adolescence: A meta-analytic investigation. School Psychology Quarterly, 25, 65-83. doi:10.1037/a0020149

DiStefano

Morgan

G. B.

(2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 21, 425-438. doi:10.1080/10705511.2014.915373

10.

Dunn

T. J.

Baguley

Brunsden

(2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412. doi:10.1111/bjop.12046

11.

Farmer

T. W.

Petrin

R. A.

Robertson

D. L.

Fraser

M. W.

Hall

C. M.

Day

S. H.

Dadisman

(2010). Peer relations of bullies, bully-victims, and victims: The two social worlds of bullying in second-grade classrooms. Elementary School Journal, 110, 364-392. doi:10.1086/648983

12.

French

B. F.

Finch

W. H.

(2008). Multigroup confirmatory factor analysis: Locating the invariant referent sets. Structural Equation Modeling: A Multidisciplinary Journal, 15, 96-113. doi:10.1080/10705510701758349

13.

Garrido

L. E.

Abad

F. J.

Ponsoda

(2013). A new look at Horn’s parallel analysis with ordinal variables. Psychological Methods, 18, 454-474. doi:10.1037/a0030005

14.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117-135. doi:10.1016/j.jsp.2006.05.005

15.

Hayes

S. C.

Nelson

R. O.

Jarrett

R. B.

(1987). The treatment utility of assessment: A functional approach to evaluating assessment quality. American Psychologist, 42, 963-974. doi:10.1037/0003-066X.42.11.963

16.

Haynie

D. L.

Nansel

Eitel

Crump

A. D.

Saylor

Simons-Morton

(2001). Bullies, victims, and bully/victims: Distinct groups of at-risk youth. The Journal of Early Adolescence, 21, 29-49. doi:10.1177/0272431601021001002

17.

Holgado-Tello

F. P.

Chacón-Moscoso

Barbero-García

Vila-Abad

(2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality & Quantity, 44, 153-166. doi:10.1007/s11135-008-9190-y

18.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55. doi:10.1080/10705519909540118

19.

Huang

F. L.

Cornell

D. G.

(2015). The impact of definition and question order on the prevalence of bullying victimization using student self-reports. Psychological Assessment, 27, 1484-1493. doi:10.1037/pas0000149

20.

Hymel

Swearer

S. M.

(2015). Four decades of research on school bullying: An introduction. American Psychologist, 70, 293-299. doi:10.1037/a0038928

21.

Iannotti

R. J.

(2013). Health Behavior in School-Aged Children (HBSC), 2009-2010 (ICPSR34792-v1). Ann Arbor, MI: Inter-university Consortium for Political and Social Research. doi:10.3886/ICPSR34792.v1

22.

Jimerson

S. R.

Nickerson

A. B.

Mayer

M. J.

Furlong

M. J.

(Eds.). (2012). Handbook of school violence and school safety: International research and practice (2nd ed.). New York, NY: Routledge.

23.

Kelley

(2016). MBESS: The MBESS R Package (R Package Version 4.1.0). Retrieved from https://CRAN.R-project.org/package=MBESS

24.

Kelley

Pornprasertmanit

(2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21, 69-92. doi:10.1037/a0040086

25.

Kline

R. B.

(2016). Principles and practice of structural equation modeling. New York, NY: Guilford Press.

26.

Kochenderfer

B. J.

Ladd

G. W.

(1996). Peer victimization: Cause or consequence of school maladjustment? Child Development, 67, 1305-1317. doi:10.1111/j.1467-8624.1996.tb01797.x

27.

Langkamp

D. L.

Lehman

Lemeshow

(2010). Techniques for handling missing data in secondary analyses of large surveys. Academic Pediatrics, 10, 205-210. doi:10.1016/j.acap.2010.01.005

28.

Mueller

R. O.

Hancock

G. R.

(2008). Best practices in structural equation modeling. In Osborne

(Ed.), Best practices in quantitative methods (pp. 488-508). Thousand Oaks, CA: Sage. doi:10.4135/9781412995627.d38

29.

Nansel

T. R.

Craig

Overpeck

M. D.

Saluja

Ruan

W. J.

(2004). Cross-national consistency in the relationship between bullying behaviors and psychosocial adjustment. Archives of Pediatrics & Adolescent Medicine, 158, 730-736. doi:10.1001/archpedi.158.8.730

30.

Nansel

T. R.

Overpeck

Pilla

R. S.

Ruan

W. J.

Simons-Morton

Scheidt

(2001). Bullying behaviors among US youth: Prevalence and association with psychosocial adjustment. The Journal of the American Medical Association, 285, 2094-2100. doi:10.1001/jama.285.16.2094

31.

Olweus

(1996). The revised Olweus Bully/Victim Questionnaire. Bergen, Norway: Research Center for Health Promotion, University of Bergen.

32.

Olweus

(2010). Understanding and researching bullying: Critical issues. In Jimerson

S. R.

Swearer

S. M.

Espelage

D. L.

(Eds.), Handbook of bullying in schools: An international perspective (pp. 9-34). New York, NY: Routledge.

33.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903. doi:10.1037/0021-9010.88.5.879

34.

Raiche

(2010). nFactors: An R package for parallel analysis and non graphical solutions to the Cattell scree test (R Package Version 2.3.3). Retrieved from http://CRAN.R-project.org/package=nFactors

35.

R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from https://www.R-project.org/

36.

Renshaw

T. L.

Hammons

K. N.

Roberson

A. J.

(2016). General versus specific methods for classifying U.S. students’ bullying involvement: Investigating classification agreement, prevalence rates, and concurrent validity. School Psychology Review, 45, 400-416.

37.

Renshaw

T. L.

Roberson

A. J.

Hammons

K. N.

(2016). The functionality of four bullying involvement classification schemas: Prevalence rates and associations with mental health and school outcomes. School Mental Health, 8, 332-343. doi:10.1007/s12310-015-9171-y

38.

Revelle

(2016). psych: Procedures for personality and psychological research (Version 1.6.9). Evanston, IL: Northwestern University. Retrieved from https://CRAN.R-project.org/package=psych

39.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. Retrieved from http://www.jstatsoft.org/v48/i02/

40.

semTools Contributors. (2016). semTools: Useful tools for structural equation modeling (R Package Version 0.4-14). Retrieved from https://CRAN.R-project.org/package=semTools

41.

Swearer

S. M.

Espelage

D. L.

Vaillancourt

Hymel

(2010). What can be done about school bullying? Linking research to educational practice. Educational Researcher, 39, 38-47. doi:10.3102/0013189X09357622

42.

Vandenberg

R. J.

Lance

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.

43.

Veenstra

Lindenberg

Oldehinkel

A. J.

De Winter

A. F.

Verhulst

F. C.

Ormel

(2005). Bullying and victimization in elementary schools: A comparison of bullies, victims, bully/victims, and uninvolved preadolescents. Developmental Psychology, 41, 672-682. doi:10.1037/0012-1649.41.4.672

44.

Vessey

Strout

T. D.

DiFazio

R. L.

Walker

(2014). Measuring the youth bullying experience: A systematic review of the psychometric properties of available instruments. Journal of School Health, 84, 819-843. doi:10.1111/josh.12210

45.

Vivolo-Kantor

A. M.

Martell

B. N.

Holland

K. M.

Westby

(2014). A systematic review and content analysis of bullying and cyber-bullying measurement strategies. Aggression and Violent Behavior, 19, 423-434. doi:10.1016/j.avb.2014.06.008

46.

World Health Organization. (2014). Health behavior in school-aged children: World Health Organization collaborative cross-national survey. Available from http://www.who.int/en