Abstract
Personality-based integrity tests are used in selection procedures to reduce the chance of hiring employees who are likely to engage in counterproductive work behaviour. This study reports the internal psychometric properties of a personality-based measure developed for this purpose. Data collected from 1353 working adults were used to investigate the internal consistency reliability and to examine construct validity with confirmatory factor analysis and Rasch analysis. Results showed that the reliability estimates for all the scales of the assessment were satisfactory. For the confirmatory factor analysis, inspection of the incremental (comparative fit index and Tucker–Lewis index) and absolute (root mean square error of approximation) goodness-of-fit values found strong support for the construct validity of all the scales. Infit statistics from Rasch analysis provided further support for construct validity, with items from all the scales fitting the Rasch model. The confirmatory and Rasch analysis demonstrated that unidimensional, coherent, and meaningful latent constructs are being measured on the Work-related Risk and Integrity Scale. Overall, results found excellent support for the internal psychometric properties of the instrument in a culturally diverse context.
Keywords
Integrity tests are psychometric measures developed for the prediction of counterproductive work behaviour (Berry, Sackett, & Wiemann, 2007). The Work-related Risk and Integrity scale (WRISc; van Zyl & de Bruin, 2016) is a personality-based scale developed to aid organisations in managing the risks associated with counterproductive workplace behaviour (CWB), or so-called workplace deviance. In addition, the instrument is used to identify the presence of potentially dark dispositions relevant to workplace deviance. Specifically, it seeks to index the Dark Triad of personality (Paulhus & Williams, 2002) that includes narcissism, Machiavellianism, and psychopathy (sub-clinical). The purpose of this study is to investigate the psychometric properties of the WRISc in a diverse population, in particular its reliability and internal construct validity.
Integrity testing
A typical but comprehensive definition of an integrity test is provided by the Association of Personnel Test Publishers (1991), who defines it as a psychological inventory designed to predict the likelihood that an applicant will exhibit counterproductive or delinquent behaviour. Two types of integrity tests are typically distinguished, labelled ‘overt’ and ‘personality based’ tests (Sackett, Burris, & Callahan, 1989). Overt tests ask direct questions about a person’s history and attitudes towards dishonest behaviour, illegal acts, and theft (Sackett et al., 1989). In contrast, personality-oriented measures are similar to regular personality assessments. They have a broader scope than overt instruments and are less transparent with respect to what is measured. Personality-oriented measures typically include items measuring ‘normal’ personality attributes which are known to be predictive of counterproductive work behaviour (Berry et al., 2007). The WRISc falls into this category of integrity tests.
Personality is a well-known predictor of many important organisational outcomes, including counterproductive work behaviour. Both broad personality dimensions such as Neuroticism in the Five Factor Model (Berry et al., 2007; Salgado, 2002) and narrow facets of personality, such as the Hostility facet scale on the larger Neuroticism dimension of the Five Factor Model (Hastings & O’Neill, 2009; O’Neill & Hastings, 2011), have been found to predict counterproductive work behaviour well. While some researchers prefer broad personality dimensions for the prediction of CWB (Ones & Viswesvaran, 1996), others argue that CWB is a narrow component of job performance and as such should be predicted using narrower constructs (Wu & LeBreton, 2011).
The development of the WRISc
The WRISc was developed to predict deviant behaviours in organisations using narrow traits of personality. The traits operationalised in the WRISc had to meet two conditions for inclusion in the assessment. The first was that each trait should be theoretically or empirically predictive of CWB or both. The second condition was that each trait should also be a theoretically important element of a larger multidimensional construct, specifically, one of the three dispositions comprising the Dark Triad of personality (Paulhus & Williams, 2002). For a comprehensive theoretical rationale of this approach to dark personality measurement, see van Zyl (2016). In short, important conceptual shifts in the field of dark personality research have in recent years enabled a new way to measure aberrant personalities such as sub-clinical psychopathy (Miller & Lynam, 2015; Miller, Lynam, Widiger, & Leukefeld, 2001; Widiger & Lynam, 1998). This, in turn, allows for new ways to index aberrant dispositions like the Dark Triad using ‘normal’ personality measures, which Wu and LeBreton (2011) have argued is missing from existing approaches to predict CWB (van Zyl, 2016).
The personality constructs identified were operationalised by means of formal definitions that guided the process of item writing. Several rounds of research were conducted during the development of the WRISc. Each round provided new information that was used to refine the instrument. Confirmatory factor analysis (CFA) and Rasch analysis were conducted simultaneously in each round, and the results compared, to identify the best items from two psychometric perspectives. The first research version comprised 250 items, the second version was reduced to 186 items, followed by 148 items in the third version.
The fourth and final version of the WRISc contains 81 items measuring 12 unidimensional personality traits including: Aggression, Effortful Control, Negative Affect, Locus of Control, Callous Affect, Impulsivity, Manipulation, Pessimism, Risk-Taking, Rule-Defiance, Egotism, and Cynicism. As mentioned, in addition to individual predictors of workplace deviance, these scales are also used in combination to index dark personality dispositions such as narcissism, Machiavellianism, and sub-clinical psychopathy. These three aversive dispositions were coined the Dark Triad of personality because they have in common a tendency towards interpersonal malevolence (Jones & Paulhus, 2011; Paulhus & Williams, 2002). These multidimensional indices are not scored as composites of the primary scales and as such fall beyond the scope of this article. Interested readers are referred to van Zyl (2016) and the WRISc manual (van Zyl & de Bruin, 2016) for more information on the underlying theory.
The aim of this study was to examine the psychometric properties of the narrow traits comprising the primary scales of the WRISc. Since scores on these unidimensional constructs form the basis of all further interpretations on the instrument, the focus of this study was on the reliability and internal construct validity of each scale. Specifically, Cronbach’s alpha reliability was evaluated and construct validity was investigated using CFA and Rasch analysis.
Method
Participants
Participants were 1353 working adults with ages ranging from 18 to 71 years (M = 35.6; standard deviation [SD] = 11.7 years), who provided informed consent before participation were allowed. Women comprised 51.4% of the total participants, with 5.8% not indicating their gender. The ethnic composition was as follows: Black (39.5%), White (28.2%), mixed race (10.2%), Indian (8.4%), Asian (0.8%), unspecified (13%), with 8.2% of respondents not endorsing any of the ethnic options. Home language representation included English (33.9%), Afrikaans (18.8%), and Zulu (11.4%), with a further 19.8% from the remaining indigenous language groups in South Africa.
Instruments
WRISc
The WRISc measures 12 primary constructs with 81 items. Responses are captured using a 5-point Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree. A sample item is: ‘I keep working on a task until it is perfect’.
Procedure
Honours level students were recruited to collect data with working adults as part of their course in research methodology at a large university in Johannesburg. Any working adult could participate in the study, provided they were currently employed. The students were also instructed to be as representative as possible regarding the age, gender, and ethnicity of the participants they recruited for data collection.
Ethical considerations
The research received ethics approval from the Higher Degrees Committee (HDC) of the Faculty of Humanities at the University of Johannesburg. Participants were provided with information pertaining to the purpose of the study, assured of anonymity, and that data will only be used for research purposes. It was stressed that participation in the study is voluntary and that there was no obligation to participate.
Data analysis
Reliability analysis
For Cronbach’s alpha, coefficients of .80 and higher were considered very good and .70–.79 were considered acceptable (Kline, 2011). The reliabilities were calculated using polychoric correlations rather than the more conventional Pearson correlations, since Pearson correlations underestimate the true relationship between ordered categorical variables (Gadermann, Guhn, & Zumbo, 2012). Zumbo, Gadermann, and Zeisser (2007) recommend using a polychoric matrix to estimate reliability with ordered categorical or Likert-type items.
CFA
CFA was conducted on the R statistical platform using the ‘lavaan’ package (Rosseel, 2012). Weighted least squares mean and variance corrected (WLSMV) parameter estimation was used (Muthén, Du Toit, & Spisic, 1997). Beauducel and Herzberg (2006) have shown that this estimator functions better than maximum likelihood when analysing ordered categorical data with five response categories. A few univariate outliers were identified, although these were either very high or very low scores on specific personality items. These cases were retained as there is no theoretical reason to consider such responses invalid. Missing data posed no substantive concern in this study given the large sample size and fact that the highest proportion of missing data for any one variable was 1.3%. Missing data were nonetheless handled using multiple imputation with the ‘mice’ package in R (van Buuren & Groothuis-Oudshoorn, 2011). Following Libbrecht, De Beuckelaer, and Lievens (2012), fit was considered reasonable to good when the comparative fit index (CFI) and Tucker–Lewis index (TLI) values were ⩾.90 (more liberal) or .95 (more conservative), and the root mean square error of approximation (RMSEA) values were ⩽.08 (more liberal) or .05 (more conservative) (Davidov, Datler, Schmidt, & Schwartz, 2011; Hu & Bentler, 1999).
Rasch analysis
Winsteps software (Linacre, 2015; version 3.92) was used to conduct Rasch rating scale analyses. In contrast to other forms of statistical analysis, Rasch modelling philosophically requires data to fit the model rather than the model reflecting a good approximation of the observed data (Bond & Fox, 2007). In this respect, the Rasch model represents a model of objective measurement. In the Rasch framework, objective measurement can only take place if the requirements of the model are met. Misfitting items were identified by inspection of the infit mean square values of each item. Bond and Fox (2007) recommend values >.60 and <.1.40 as acceptable for Likert-type items. Items with infit mean square values <.60 overfit the model, suggesting that it might not be contributing any new information, while values >1.40 indicate underfit, which means that the item might be measuring something unexpected (Bond & Fox, 2007).
Results
Descriptive statistics, reliability estimates and bivariate correlations are presented in Table 1. Cronbach’s alpha internal consistency reliabilities for the scales are presented on the diagonal in boldface. The reliabilities were satisfactory with almost all scales having Cronbach’s alpha coefficients >.80 with the exception of Cynicism (α = .74). Raw correlations between the variables are presented below the diagonal of Table 1 and disattenuated correlations above the diagonal.
Descriptive statistics, Cronbach’s alpha reliability estimates, and bivariate correlations among the scales of the WRISc.
WRISc: Work-related Risk and Integrity Scale; SD: standard deviation.
Raw correlations are below the diagonal and disattenuated correlations above the diagonal. Cronbach alpha coefficients are displayed on the diagonal.
CFA
The goodness-of-fit indices for the results of the confirmatory factor analyses are reported in Table 2. Inspection of the incremental (CFI and TLI) and absolute (RMSEA) fit indices revealed satisfactory fit for each of the models. For models where either the RMSEA estimate or the upper limit of its 95% confidence interval exceeded .10, modification indices were inspected. One round of re-specification followed for Locus for Control, Egotism, and Rule-Defiance. The correlated error terms between two items on both the Locus of Control and Rule-Defiance scales were allowed to co-vary. The same was done for two item pairs on Egotism. The original models are indicated as ‘a’ and the re-specified models as ‘b’ in the table. In all these item pairs, there was some content overlap making it theoretically sensible to include the model re-specifications. Inspection of the correlated residuals found none exceeding .10 on any of the scales, including the re-specified models, suggesting that each model explained the observed sample correlations well.
Summary of CFA goodness-of-fit values for each personality scale.
TLI: Tucker–Lewis index; CFA: confirmatory factor analysis; CFI: comparative fit index; RMSEA: root mean square error of approximation; a: original model; b: re-specified model.
The standardised and unstandardised item factor loadings are presented in Table 3. All loadings were statistically significant for all items on their respective scales. In general, the factor loadings were acceptable with most loadings >.50.
Standardised and unstandardised factor loadings for the items of the WRISc.
WRISc: Work-related Risk and Integrity Scale; SE: standard error.
Rasch analysis
The results displayed in Table 4 show that the average infit mean square values for the scales were all close to the expected value of one, demonstrating acceptable fit overall. At scale level, there were no items that overfit the model. Three items were identified for underfitting the model, two on Aggression and one on the Risk-Taking scale. Underfit suggests that these items may be measuring something unexpected (Bond & Fox, 2007). On the Aggression scale, these two items indeed represented more physical forms of aggression when compared to the other items on the scale. However, these items reflect a serious potential for aggression and accordingly were retained by design as this is consistent with the purpose of the assessment. The Risk-Taking item only bordered the cut-off for underfit and was not considered problematic.
Rasch fit statistics for the scales of the WRISc.
IMNSQ: infit mean square; SD: standard deviation.
Discussion
The purpose of this study was to examine the psychometric properties of the WRISc, in particular its reliability and internal construct validity. The reliability estimates were satisfactory for all scales with Cronbach’s alpha coefficients ranging between .74 (Cynicism) and .88 (Aggression and Risk-Taking).
Construct validity was examined from two psychometric perspectives using CFA and Rasch analysis. CFA was conducted on each of the personality variables of the instrument to test if a coherent and unidimensional construct is being measured in each case. We first examined the goodness-of-fit indicators, two incremental fit (CFI and TLI), and one absolute fit index (RMSEA) for each model tested. Reasonable fit was observed for all scales when all three indices are considered together. The incremental fit indices were particularly satisfactory for all scales; however, the RMSEA point estimates for Locus of Control, Egotism, and Rule-Defiance were weaker than expected. This was not considered problematic as the RMSEA index is well known to penalise small models with few degrees of freedom (Kenny, Kaniskan, & McCoach, 2014), as was the case in this study. On the other hand, this aspect of the RMSEA index can to some degree be moderated by large samples (Kenny et al., 2014), which also applies to this study. To err on the side of caution, we inspected the 95% confidence interval of the RMSEA parameter estimates and decided to further investigate models where the upper value of the confidence interval exceeded .10, which would be indicative of poor fit (Kline, 2011). Thus, for Locus of Control, Egotism, and Rule-Defiance, inspection of the modification indices revealed items with a degree of content overlap, which were re-specified accordingly. Satisfactory fit was observed for the re-specified models with substantial improvements on all three fit indicators. Overall, the results from the CFA strongly supported the construct validity of all the scales of the WRISc.
Results from the Rasch analysis were similarly encouraging. We examined the degree to which the items of each scale fit the Rasch model. Substantial deviations from this ideal typically indicate that some of the items are redundant or that it is measuring something unexpected (Bond & Fox, 2007). In general, the average mean square fit values clustered around the expected value of one (Bond & Fox, 2007), showing a close fit to the Rasch model across the scales. For the Aggression scale this was larger than expected, although it makes sense on closer inspection when examining item fit. A total of three items were flagged for potential underfit, two on the Aggression scale and one on Risk-Taking. However, in each case, the items simply referred to rather extreme forms of the behaviour, which would reflect high levels of the underlying trait. It is therefore expected that Rasch modelling might flag such items. However, a major purpose behind the development of the scales is to predict CWB, and the theoretical expectation is that high levels of Aggression and Risk-Taking will be predictive of these forms of behaviour. As such, the items were retained by design.
Of course, our study had some limitations. While the sample in this research was quite representative of the ethnic composition of the South African population, especially with regard to Black and White respondents, future studies may seek to have even better representation of Coloured (mixed race) and Indian participants (other major demographic subgroups of the South African population).
The self-report nature of the scales of the instrument could perhaps be criticised for the potential of socially desirable responding. However, in an unpublished study, we examined invariance across low and high stake conditions and found no evidence to suggest that people meaningfully modify their responses in situations where it might be beneficial to do so.
While this study showed that the WRISc works well with a sample that is ethnically diverse, future studies are required to examine in detail the degree to which it is equivalent across specific gender and ethnic population groups using differential item function (DIF) analysis or measurement invariance analysis. Further research should also investigate the criterion, convergent, and discriminant validity of the scales.
Conclusion
This study examined the psychometric properties of the WRISc. Overall, reliability, confirmatory factor, and Rasch analysis suggest that coherent latent constructs are reliably being measured, which is indicative of a robust instrument as it was developed and tested to function effectively in a multicultural population. Given the positive results found in this context, the tacit assumption is that the test should function well in more homogeneous populations, although this would have to be tested empirically. Overall, from a measurement perspective, the results of this study hold good promise for the WRISc as a psychometric instrument with which to manage CWB in organisations.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
