Factor Structure of the Compulsive Internet Use Scale

Abstract

An important step in Internet addiction research is to develop standardized instruments for assessing Internet addiction–related symptoms. The Compulsive Internet Use Scale (CIUS) is a promising brief questionnaire. The aim of this study was to examine the factor structure of a German version of the CIUS with confirmatory factor analysis in a general population sample. In addition, the best fitting structure was tested for factorial invariance across sex, age, education level, and weekly Internet use. We used a weighted general population sample (N=8,132) of 14–64 years olds spending at least 1 hour online for private purposes per typical working or weekend day. Findings include that a one-factor model was found to fit well. It was invariant across sex, age, education level, and weekly Internet use. The findings support the validity of the CIUS as a short screening instrument.

Introduction

Increasing acceptance of Internet addiction as a relevant health problem has created the need to provide valid instruments to assess Internet addiction in different settings. Particularly in general population settings and for purposes of prevalence estimation, brief measures are needed. Several instruments for assessing Internet addiction have been proposed.^1–3 Most of them are based on DSM-IV criteria for substance dependence, as well as on criteria for pathological gambling.⁴ However, the majority of these instruments were validated using small samples consisting of students. Furthermore, some are too time-consuming for nonclinical settings with more than 30 items. One of the shorter instruments to assess Internet addiction is the Compulsive Internet Use Scale (CIUS),⁵ which includes only 14 items.

Compulsive Internet Use Scale

The CIUS was designed as an instrument that assesses core elements of Internet addiction. These core elements were derived, first, from the DSM-IV pathological gambling and substance dependence criteria, second, from criteria for behavior addiction that had been proposed by Griffiths,⁶ and, third, from a qualitative study on Internet users.⁵ Items were developed to cover five typical symptoms of Internet addiction: loss of control, preoccupation, withdrawal symptoms, coping or mood modification, and conflict.⁵

Initial investigation of the CIUS revealed good psychometric properties in nontreatment settings.⁵ Up to now, the factor structure of the CIUS has been studied in several languages (Table 1). In four analyses, a one-factor solution yielded good fit to the data under the assumption of a number of correlated error pairs, which refer to the part of indicator covariation due to sources other than the common factor.

Table 1.

Previous Factor Analysis Studies of the Compulsive Internet Use Scale

	Meerkerk et al. (2009)	Khazaal et al. (2011)	Alavi et al. (2011)	Khazaal et al. (2012)	Peukert et al. (2012)
CIUS version	Dutch	Persian	Persian	French	German
Sample	447 heavy Internet users (>18 years); 16,925 Internet users (11–80 years)	185 volunteers from the community (15–25 years)	400 students	127 students and volunteers from the community (16–45 years)	2,506 students (18–54 years)
Method	CFA	EFA, CFA	EFA	EFA, CFA	CFA
Criteria for numbers of factors	Predefined	MAP, PA	Eigenvalues	MAP, PA	Predefined
Factors proposed	1	1	3	1	1
Items pairs with correlated errors	1,2	4,7	—	1,2	1,2
	6,7	4,11		12,13	6,7
	8,9	5,7			8,9
	10,11	5,11			10,11
	12,13	8,9			12,13
		12,13

CIUS, Compulsive Internet Use Scale; CFA, confirmatory factor analysis; EFA, explorative factor analysis; MAP, minimum average partial; PA, parallel analysis.

Another relevant point for the evaluation of the construct validity of an instrument is factorial invariance of the factor structure across different groups, that is, whether the construct assessed by the instrument has the same meaning for different groups.⁷ The one-factor structure of the CIUS proved to be highly stable across time, age, sex and Internet use.^5,8

Since initial validation studies of the CIUS are based on high-risk populations or convenience samples (participations recruited via media or email) consisting mostly of students and young adults (Table 1), the generalizability of the findings is limited.

In summary, findings suggest that the CIUS may provide an economic and valid assessment of Internet addiction. Limitations of the evidence include a lack of studies in general population samples.

Current study

We sought to analyze the factorial structure of the CIUS in a large German general population sample. We performed confirmatory factor analysis (CFA) in order to examine the factor structure of a German language CIUS version and to test its invariance across sex, age, education, and weekly Internet use.

Method

Recruitment and subjects

The present analysis was based on a sample from the study “Pathological Gambling and Epidemiology” (PAGE).⁹ One recruitment approach of PAGE was a representative telephone survey with 14–64 year olds, consisting of a German landline telephone sample and a mobile phone-only sample to cover individuals that are exclusively reachable by cellular phone but not via landline phone. For the landline telephone survey, 53 sample points were selected using a probability-proportional-to-size procedure¹⁰ and implicit stratification for German federal states, administrative districts, and counties, as well as number of gambling machines. For each region, a fixed number of landline telephone numbers was generated following a random digit dialing procedure. Among 26,736 eligible persons, 4.6% could not be interviewed because the contacted person refused access to the target person, 1.4% were too ill, 2.7% were not reached, 38.9% refused to participate, and 52.4% (n=14,022) participated in the interview.

For the mobile phone-only sample, nationwide cell phone numbers were randomly generated (n=13,273). Eligible were individuals who were available via their mobile phone number but not via landline telephone (n=1,767). Among eligible persons, 42.3% refused to participate, 1.1% had insufficient German language skills, were too ill, or handicapped, and 56.6% (n=1,001) participated in the interview.

The CIUS was administered if the participants reported an average private Internet use of at least 1 hour per working or weekend day. Of the total 15,023 telephone interview participants, 8,132 (54.13%) fulfilled this inclusion criterion. Among these 8,132 participants, mean age was 35.24 years (SD=13.79), 51.1% were male, 4.9% were unemployed, 33.0% were married, 13.1% reported an educational level equivalent to less than 10 years of schooling, 79.5% equivalent to 10 or more years of schooling, and 7.4% were still at school. A total of 12.9% were born abroad (migration experience), and 24.9% had migration experience or/and had parents born abroad (migration background).

Measures

Time spent online

Participants were asked how many hours they spent online in an average working and weekend day respectively. We calculated total time of Internet use per week.

Internet addiction

The CIUS⁵ is a self-report measure to assess the severity of Internet addiction–related symptoms. Respondents have to provide a self-rating on a 5-point scale ranging from 0=“never” to 4=“very often.” The 14 items cover loss of control (items 1, 2, 5, 9), preoccupation (items 4, 6, 7), withdrawal symptoms (item 14), coping or mood modification (items 12, 13), and conflict (items 3, 8, 10, 11). Internal consistency ranged from α=0.78 to 0.91.^5,8,11–13 Furthermore, the CIUS showed sufficient convergent and criterion validity.⁵ The CIUS items were translated into German by translation and back-translation.

Statistical analysis

Mplus v6.12¹⁴ was used for statistical analysis.

Missing values

Of the 8,132 participants, 2.5% had one missing value, and a further 0.7% had between 2 and 10 CIUS items missing. Since Mplus did not provide all needed parameters for models estimated using multiple imputation,¹⁵ missing data were handled using single imputation by chained equations.^16–18 In this procedure, CIUS item values, age, sex, education level, family status, employment, Internet use per working and weekend day, as well as migration experience/background of the participants were considered. Imputation by chained equations imputes missing values by a series of regression models. Each variable with missing data is modeled dependent on all other variables in the imputation model.¹⁶

Weighting procedures

Design weighting was first performed by adjustment for different probabilities of individuals to be drawn. For the landline telephone sample, probabilities were calculated considering stratification and sampling procedure. For the mobile-only sample, only the number of mobile numbers was considered. Second, a post-stratification was performed to compensate for undercoverage of sampling procedure considering the distribution of German general population according to gender, age, years of school, employment, federal states, size of household, and migration background. After separately weighting the landline and mobile-only sample, both weights were merged accounting for their proportions in the general population (14% of mobile-only users among 14–64 year olds), and a joint post-stratification was performed.

Factorial structure

To evaluate the factorial structure of the CIUS, CFAs for a set of models were calculated (Table 2). First, a one-factor model was tested and inspected for correlated error terms. As alternative models, a five-factor model based on the five proposed relevant dimensions of Internet addiction⁵ and a hierarchical model were tested. This was the same model as before with Internet addiction as higher-order factor. Last, the proposed model of Meerkerk et al.⁵ with one factor and five correlated errors was compared to the other models.

Table 2.

Summary of Fit Indices and Factor Loadings for Tested Models

Model	λ	χ² (df )	CFI	RMSEA	SRMR	AIC
One-factor model	0.52–0.65	1,751 (77)^*	0.861	0.052	0.049	271,763
One-factor model with four correlated errors	0.50–0.66	489 (73)^*	0.965	0.026	0.029	268,143
One-factor model with five correlated errors	0.50–0.65	485 (72)^*	0.966	0.027	0.029	268,123
Five-factor model with correlated factors	0.55–0.93	962 (68)^*	0.926	0.040	0.037	269,504
Hierachical model	0.55–0.98	1,059 (73)^*	0.918	0.041	0.040	269,787

Note. All parameters based on weighted data.

p<0.001.

λ, standardized factor loadings; df, degrees of freedom; CFI, comparative fit index; RMSEA, root mean square error of approximation; SRMR, standardized root mean square residual; AIC, Akaike information criterion.

Skewness varied among items between 0.71 and 2.35 (M=1.52), and kurtosis between 2.78 and 8.56 (M=5.07). Since skewness and kurtosis values greater than one indicate a substantial deviation from a normal distribution which may lead to distortion of parameter estimations while using maximum-likelihood,¹⁹ models were estimated with the robust-maximum-likelihood method which is robust against deviations from normal distribution.

To assess the fit of the models, we used the following criteria: a significant chi-square value indicates a significant deviation of the data from the theoretical model. Because a significant chi-square value might be an effect of the large sample size, we also considered model fit indices.²⁰ Therefore, we used the comparative fit index (CFI), the standardized root mean square residual (SRMR), the root mean square error of approximation (RMSEA), and the Akaike information criteria (AIC), which had been recommended by Hooper et al.²⁰ We considered values of CFI≥0.95, SRMR≤0.08 and RMSEA≤0.06 as a good fit.²¹ The AIC describes the badness of the fit, where smaller values of AIC were regarded as better-fitting of the model.²⁰ To compare the models among each other, the Satorra–Bentler scaled chi-square difference test (SB-Δχ² test)²² was used for nested models, and nonnested models were compared using the above mentioned model fit indices. In addition, modifications indices (MI) and standardized expected parameter changes (SEPC) were used to determine which relevant correlated errors were not included in the current model. We focused on MIs above 3.84 indicating a significant decrease in chi-square if the parameter was freely estimated. Since MI is sample dependent, we focused additionally on high SEPC values (above 0.2), which indicate the estimated parameter size if the parameter is freely estimated.²³

Factorial invariance

To test the best fitting factor solution for invariance across age, sex, education, and Internet use, we focused on two aspects of factorial invariance: configurable and measurement invariance.²⁴ Following Dimitrov,⁷ we tested factorial invariance in a step-wise approach. Four age and education groups were defined in addition to sex groups (Table 3). Following Meerkerk et al.,⁵ participants were divided into heavy (spending≥16 hours on the Internet) and nonheavy Internet users (spending<16 hours on the Internet). In a first step, configurable invariance, that is, determining whether the groups have the same pattern of freed und fixed model parameters, was tested by running individual CFAs in each group. According to Raju et al.,²⁵ measurement invariance is displayed if at least the factor loadings are invariant across the groups, indicating that different groups respond in the same way to the CIUS items.²⁴ After configurable invariance is established, testing for equal factor loadings can be conducted by comparing a baseline model with identical patterns of freed und fixed parameters to a model with constrained factor loadings using multiple group CFAs. To evaluate the invariance of factor loadings, the two models were compared with the SB-Δχ² test.²² A nonsignificant SB-Δχ² test indicates the invariance of factor loadings across groups. Because of the dependency on sample size of the SB-Δχ² test, we also used the difference in CFI as an indicator for model comparison. Cheung and Rensvold²⁶ suggested that a value lower than −0.01 would indicate a lack of invariance.

Table 3.

Summary of Factor Loadings and Fit Indices for Sex, Age, Education, and Internet Use Groups

Sample	λ	χ² (df )	CFI	RMSEA	SRMR	SB-Δχ² (Δdf )	ΔCFI
Gender
Males (n=4,155)	0.46–0.65	304 (73)^*	0.965	0.028	0.030	—	—
Females (n=3,977)	0.52–0.67	247 (73)^*	0.967	0.025	0.030	—	—
Multigroup analysis
Unconstrained model	—	546 (146)^*	0.966	0.026	0.030	—	—
Constrained model	—	585 (159)^*	0.964	0.026	0.035	13 (13)	−0.002
Age
14–17 years (n=774)	0.39–0.61	139 (73)^*	0.941	0.034	0.044	—	—
18–24 years (n=1,543)	0.46–0.64	195 (73)^*	0.949	0.033	0.037	—	—
25–44 years (n=3,455)	0.47–0.64	265 (73)^*	0.965	0.028	0.032	—	—
45–64 years (n=2,360)	0.47–0.73	223 (73)^*	0.948	0.030	0.037	—	—
Multigroup analysis
Unconstrained model	—	851 (292)^*	0.955	0.031	0.036	—	—
Constrained model	—	947 (331)^*	0.950	0.030	0.045	33 (39)	−0.005
Education
<10 years (n=1,066)	0.47–0.69	156 (73)^*	0.963	0.033	0.037	—	—
10 years (n=2,482)	0.48–0.65	207 (73)^*	0.972	0.027	0.029	—	—
>10 years (n=3985)	0.45–0.67	396 (73)^*	0.961	0.032	0.033	—	—
Pupil (n=599)	0.42–0.58	128 (73)^*	0.950	0.036	0.041	—	—
Multigroup analysis
Unconstrained model	—	910 (292)^*	0.963	0.032	0.033	—	—
Constrained model	—	1,017 (331)^*	0.959	0.032	0.041	52 (39)	−0.004
Internet use per week
<16 hours (n=5,881)	0.44–0.63	327 (73)^*	0.964	0.024	0.029	—	—
≥16 hours (n=2,251)	0.49–0.64	232 (73)^*	0.954	0.031	0.035	—	—
Multigroup analysis
Unconstrained model	—	570 (146)^*	0.961	0.027	0.031	—	—
Constrained model	—	638 (159)^*	0.956	0.027	0.038	2 (13)	−0.005

Note. All parameters except group sizes based on weighted data. In constrained models, all factor loadings were constrained to be equal. λ, standardized factor loadings; df, degrees of freedom; CFI, comparative fit index; RMSEA, root mean square error of approximation; SRMR, standardized root mean square residual; AIC, Akaike information criterion; SB-Δχ², Satorra–Bentler scaled difference in χ². ^*p<0.001.

Results

On average, the participants spent 12.87 (SD=12.41) hours per week online for private purposes (range 2–168 hours). For the CIUS, the data revealed a mean sum score of 8.84 (SD=7.46) and a range of 0–52.

Factor structure of the CIUS

The one-factor solution showed an insufficient model fit with respect to the CFI (Table 2). Inspection of MIs and SEPC values for this model suggested four possible correlated error pairs (MI=169–577; SEPC=0.26–0.49). The four correlated error pairs correspond to four of five correlated error pairs that have been postulated by the test author.⁵ The fifth postulated pair, item 10 and 11, reached merely an MI of 20 and a SEPC of 0.09. Hence, a new one-factor model was specified, including the correlation of error variances of items 1 and 2, 6 and 7, 8 and 9, as well as 12 and 13. This model modification resulted in a significant drop of the chi-square value (SB-Δχ²(4)=253, p<0.001), and fit indices were now in the range for a good fit. Factor loadings were substantial (Table 2), and inspection of MIs and SEPC values suggested no further correlated error pairs. Considering all five postulated correlated error pairs of Meerkerk et al.⁵ in the one-factor solution also resulted in a good fit to the data (Table 2). However, CFI and AIC improved only slightly, and according to the SB-Δχ² test, no significant model improvement compared to the one-factor model with four correlated error pairs was reached (SB-Δχ²(1)=1, p=0.317). In addition, the four correlated error pairs suggested by the MI and SEPC values reached a significance of α<0.001, whereas the fifth added error pair did not (p=0.023). In a second step, we tested two alternative models for the CIUS. The five-factor model showed an insufficient model fit and the five postulated factors showed high intercorrelations between r=0.50 and r=0.96. Consequently, the hierarchical model showed also an insufficient fit to the data (Table 2).

Factorial invariance

We proceeded with the invariance test of the one-factor model with four correlated error pairs. By running individual CFAs in each sex, age, education, and Internet use group, this model revealed sufficient fit to the data and substantial factor loadings for each group (Table 3). Configurable invariance of the one-factor model over sex, age, education, and Internet use was established.

We used multigroup CFAs for comparing the unconstrained model (solely same pattern of freed und fixed model parameters) to the model with constrained factor loadings between the groups. The SB-Δχ² test revealed no significant differences between these two models for sex (p=0.448), age (p=0.739), education (p=0.080), and Internet use groups (p=0.999). The difference in CFI for unconstrained and constrained models were also less than −0.01. Factor loadings can be seen as equal across sex, age, education level, and Internet use (Table 3).

Discussion

The current study sought to assess the factor structure of a German version of the CIUS in a population-based sample after evidence before had been limited by several sample restrictions.

In the current study, CFA revealed that a one-factor model with four correlated error pairs fitted the data best. This confirms previous studies that used more restricted samples.^5,8,11,12 More importantly, we found correlated error terms that correspond to four of five postulated error pairs of the test author.⁵ Adding the fifth postulated correlated error pair did not lead to a significant improvement of the model with our data. In previous factor analysis studies of the CIUS, one correlated error pair (item 12 and 13) was found in all analyses that included CFA.^5,8,11,12 In line with that, this item pair was the one with the biggest MI (577) and SEPC value (0.49) in our model.

The four correlated error pairs found in the current study suggest overlapping item content. This represents an opportunity for improvement of validity of the CIUS by constructing a shorter version of the questionnaire. In the same way, a French workgroup²⁷ proposed a short version of the CIUS for adolescents by deleting one item of each item pair with correlated errors suggested by Meerkerk et al.⁵ The remaining nine items had a good fit for a one-factor solution without correlated error pairs.²⁷ However, our results suggest deleting four items, which would resulted in a 10-item version of the CIUS. Further research is needed to show the validity of such a short version.

Although CFA demonstrated evidence that the one-factor solution with four correlated errors had a good fit to the data, we were interested in comparing this to two alternative factor solutions. However, the five-factor model as well as the hierarchical model did not show sufficient model fit. According to MacKenzie et al.,²⁸ discriminant validity becomes problematic with intercorrelations greater than r=0.71. Because correlations among the five dimensions are so strong, considering these as independent dimensions is questionable, and rather indicated a one-factor model than a five-factor model. Contrary to our results, there was one study¹³ that found no one-factor solution for the CIUS. In this study, the use of eigenvalue as the only criteria for deriving the number of factors in explorative factor analysis may have led to an overestimation of factors.²⁹ Other criteria such as Horn's parallel analysis³⁰ and Minimum Average Partial³¹ have been recommended as adequate criteria for factor extraction.²⁹

Another aspect of the present analysis was testing invariance of the model across sex, age, education, and weekly Internet use. In line with previous research,^5,8 the one-factor structure proved to be highly stable across sex, age, education, and weekly Internet use. This means that the constructs are manifested in the same way in each group, and ratings can be compared across groups.²⁴

Limitations of the present study include the inclusion of correlated error terms in model estimation. In a strict sense, one-dimensionality is only given when no correlated errors were present. However, the assumption of uncorrelatedness of errors in classic measurement theory may not hold in every case, for example when common assessment methods, reversed or similarly worded test items, social desirability, or overlapping contents of items are present.³² The inclusion of the correlated errors is justified because, first, all four error pairs were interpretable regarding content. Second, previous studies^5,8,11,12 in part revealed the same correlated error terms. Thus, we included correlated error terms not solely based on statistics, but based on interpretability and replicability, complying with recommendations outlined by Brown.³³ Furthermore, since we used a German language version of the CIUS, this may limit the generalizability to other language versions. Finally, on a conceptual level, one could argue that Internet addiction refers to the addictive use of various applications such as online games, social media, or online pornography. These may share common components, such as salience, mood modification, tolerance, withdrawal, conflict, and relapse. However, the behavioral manifestations of these components may vary according to the specific nature of the application. Therefore, it may be preferable to study the factor structure of the CIUS for each application separately.

In conclusion, this study represents the first general population-based examination of the CIUS. Our data support the assumption of a one-factorial structure with four correlated errors for the CIUS and its invariance across sex, age, education, and weekly Internet use. The findings support the CIUS as a valid assessment instrument of Internet addiction. Practical implications include that the CIUS is appropriate for the use in general population samples, and that sum or factor scores can be calculated. Nevertheless, present data suggest a potential improvement of validity by constructing a shorter version of the CIUS.

Footnotes

Acknowledgments

This study was funded by the German Federal States (Geschz II 6 – 21v06.03-01-09/002). Initial analysis of this paper was reported in 2012 as an Abstract in Sucht 58 Suppl. 1, p. 83. German title: Faktorielle Struktur der Compulsive Internet Use Scale (CIUS) in einer Allgemeinbevölkerungsstichprobe.

Author Disclosure Statement

No competing financial interests exist.

References

Shaw

, Black

. Internet addiction: definition, assessment, epidemiology and clinical management. CNS Drugs, 2008; 22:353–365.

Beard

. Internet addiction: a review of current assessment techniques and potential assessment questions. CyberPsychology & Behavior, 2005; 8:7–14.

Chou

, Condron

, Belland

. A review of the research on Internet addiction. Educational Psychology Review, 2005; 17:363–388.

American Psychiatric Association. (2000) Diagnostic and statistical manual of mental disorders. Washington DC: American Psychiatric Association.

Meerkerk

, Van Den Eijnden

RJJM

, Vermulst

, et al. The Compulsive Internet Use Scale (CIUS): some psychometric properties. CyberPsychology & Behavior, 2009; 12:1–6.

Griffiths

. Internet addiction: fact or fiction?. Psychologist, 1999; 12:246–250.

Dimitrov

. Testing for factorial invariance in the context of construct validation. Measurement & Evaluation in Counseling & Development, 2010; 43:121–149.

Peukert

, Steffen

, ElKasmi

, et al. Faktorielle Struktur der deutschen Version der Compulsive Internet Use Scale (CIUS) nach konfirmatorischer Faktorenanalyse [Factorial structure of the German version of the Compulsive Internet Use Scale (CIUS) after confirmatory factor analysis]. Zeitschrift für Klinische Psychologie und Psychotherapie, 2012; 41:101–108.

Meyer

, Bischof

, Westram

, et al. The project “Pathological Gambling and Epidemiology” (PAGE): study design and results of fieldwork. International Journal of Methods in Psychiatric Research, 2013; (in press).

10.

Levy

, Lemeshow

. (2008) Sampling of populations. Methods and applications. Hoboken, New Jersey: Wiley.

11.

Khazaal

, Chatton

, Atwi

, et al. Arabic validation of the Compulsive Internet Use Scale (CIUS). Substance Abuse Treatment, Prevention & Policy, 2011; 6:32.

12.

Khazaal

, Chatton

, Horn

, et al. (2012) French validation of the Compulsive Internet Use Scale (CIUS). www.springerlink.com/content/602021835723070g/fulltext.pdf (accessed Dec. 14, 2012).

13.

Alavi

, Jannatifard

, Eslami

, et al. Validity, reliability and factor analysis of Compulsive Internet Use Scale in students internet users of Isfahan's universities [Abstract]. 2011. http://him.mui.ac.ir/index.php/him/article/view/204 (accessed Nov. 1, 2012).

14.

Muthén

, Muthén

. (1998–2000) Mplus user's guide. 6th ed. Los Angeles, CA: Muthén & Muthén.

15.

Schafer

, Graham

. Missing data: our view of the state of the art. Psychological Methods, 2002; 7:147–177.

16.

Azur

, Stuart

, Frangakis

, et al. Multiple imputation by chained equations: what is it and how does it work?. International Journal of Methods in Psychiatric Research, 2011; 20:40–49.

17.

Raghunathan

, Lepkowski

, Hoewyk

, et al. A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 2001; 27:85–95.

18.

van Buuren

. Multiple imputation of discrete and continuous data by fully conditional specification. Statistical Methods in Medical Research, 2007; 16:219–242.

19.

Muthén

, Kaplan

. A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical & Statistical Psychology of Addictive Behaviors, 1985; 38:171–189.

20.

Hooper

, Coughlan

, Mullen

. Structural equation modelling: guidelines for determining model fit. Electronic Journal of Business Research Methods, 2008; 6:53–60.

21.

, Bentler

. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling—A Multidisciplinary Journal, 1999; 6:1–55.

22.

Bryant

, Satorra

. Principles and practice of scaled difference chi-square testing. Structural Equation Modeling, 2012; 19:372–398.

23.

Whittaker

. Using the modification index and standardized expected parameter change for model modification. Journal of Experimental Education, 2012; 80:26–44.

24.

Milfont

, Fischer

. Testing measurement invariance across groups: applications in cross-cultural research. International Journal of Psychological Research, 2010; 3:111–121.

25.

Raju

, Laffitte

, Byrne

. Measurement equivalence: a comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 2002; 87:517–529.

26.

Cheung

, Rensvold

. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 2002; 9:233–255.

27.

Cartierre

, Coulon

, Demerval

. Validation d'une version courte en langue française pour adolescents de la Compulsive Internet Use Scale [Validation of a short French version for adolescents of the Compulsive Internet Use Scale]. Neuropsychiatrie de l'enfance et de l'adolescence, 2011; 59:415–419.

28.

MacKenzie

, Podsakoff

, Jarvis

. The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 2005; 90:710–730.

29.

Frazier

, Youngstrom

. Historical increase in the number of factors measured by commercial tests of cognitive ability: are we overfactoring?. Intelligence, 2007; 35:169–182.

30.

Horn

. A rationale and test for the number of factors in factor analysis. Psychometrika, 1965; 30:179–185.

31.

Velicer

. Determining the number of components from the matrix of partial correlations. Psychometrika, 1976; 41:321–327.

32.

Podsakoff

, MacKenzie

, Lee

, et al. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 2003; 88:879–903.

33.

Brown

. (2006) Confirmatory factor analysis for applied research. New York: Guilford Press.