Don’t Forget the Items

Contributions of Item-Level Meta-Analytic Construct Validation

Scale-level construct validation efforts are unquestionably important tools, but they cannot diagnose the adequacy of each item on a scale because the specific items are excluded from such evaluations. However, the items must be evaluated to determine whether and how a scale should continue to be used. Our study documents that problematic items may have low factor loadings on the theorized measurement model but that these loading issues are solved when fitting a new factor structure that differs from what was initially proposed. Thus, consideration of the items can provide direct implications and directions for the future use and analysis of the scale.

Meta-analysis is a critical tool for understanding empirical relationships without the obfuscating influences of sampling error and unreliability (Schmidt & Hunter, 2015). This points to the importance of reevaluating factor analytic evidence once a sufficient number of studies accumulate to correct sampling error variance, examine the factor structure, and evaluate whether study-level moderators influence factor loadings and model fit. Indeed, when relationships are examined in a cumulative fashion via meta-analysis, multiple primary studies representing various samples and settings are combined, allowing both the scale and its items to be understood in a more conclusive manner.

Justification for item-level meta-analytic construct validation also comes from reviewing the initial development of the Williams and Anderson (1991) scale as well as evidence from studies that subsequently used the scale. First, the original scale was validated via exploratory factor analysis (EFA) of ratings from 127 employees and their peers and supervisors. ² No issues were noted with the reverse-worded items on the scale, but one item (“conserves and protects organizational property” [OCBO6]) was identified as potentially problematic. Given the aforementioned issues we outlined regarding single-study construct validation efforts, it is not surprising that extant primary studies using the Williams and Anderson scale provide inconsistent results regarding the items. For example, subsequent studies have included the potentially problematic item without incident (for a recent example, see Ferris, Lian, Brown, & Morrison, 2015), whereas others have found that reverse-worded items failed to load on the “correct” factors (e.g., Piccolo & Colquitt, 2006; Vogel & Feldman, 2009), and other authors removed the reverse-worded items from the scale prior to analysis. This illustrates that the initial scale validation study may provide evidence regarding the adequacy of the items that is inconsistent with what subsequent studies find (presumably using the same items and different samples and contexts).

We meta-analyze each of the relationships among the 21 items on the Williams and Anderson (1991) scale—this amounts to 231 bivariate correlations—to derive an inter-item correlation matrix that more closely converges on the population relationships among the items compared to a single primary study (Schmidt & Hunter, 2015; see also Klein, Wesson, Hollenbeck, Wright, & DeShon, 2001; Marcus, Taylor, Hastings, Sturm, & Weigelt, 2016). We then use these data (i.e., meta-analytic inter-item correlation matrix) as input for confirmatory factor analyses (CFAs) to evaluate the fit of measurement models that assign task performance and OCB items to their original labels and factors as denoted in Table 1. We use the results of the substantive validity assessment (described in the following) to reassign any mislabeled items to their new factors and determine whether model fit improves.

OPEN IN VIEWER

Table 1.

Williams and Anderson’s (1991) Task Performance, OCBI, and OCBO Items.

	Item Wording	Item Labels
		Original Label		Revised Label
		Two-Factor	Williams and Anderson’s Three-Factor	Revised Three-Factor	Revised Four-Factor
task1	Adequately completes assigned duties.	Task	Task	Task	Task
task 2	Fulfills responsibilities specified in job description.	Task	Task	Task	Task
task 3	Performs tasks that are expected of him/her.	Task	Task	Task	Task
task 4	Meets formal performance requirements of the job.	Task	Task	Task	Task
task 5	Engages in activities that will directly affect his/her performance evaluation.	Task	Task	Task	Task
task 6	Neglects aspects of the job he/she is obligated to perform. (R)	Task	Task	CWB	CWB
task 7	Fails to perform essential duties. (R)	Task	Task	CWB	CWB
ocbi1	Helps others who have been absent.	OCB	OCBI	OCB	OCBI
ocbi2	Helps others who have heavy workloads.	OCB	OCBI	OCB	OCBI
ocbi3	Assists supervisor with his/her work (when not asked).	OCB	OCBI	OCB	OCBI
ocbi4	Takes time to listen to co-workers’ problems and worries.	OCB	OCBI	OCB	OCBI
ocbi5	Goes out of way to help new employees.	OCB	OCBI	OCB	OCBI
ocbi6	Takes a personal interest in other employees.	OCB	OCBI	OCB	OCBI
ocbi7	Passes along information to co-workers.	OCB	OCBI	OCB	OCBI
ocbo1	Attendance at work is above the norm.	OCB	OCBO	OCBa	OCBOa
ocbo2	Gives advance notice when unable to come to work.	OCB	OCBO	OCBa	OCBOa
ocbo3	Takes undeserved work breaks. (R)	OCB	OCBO	CWB	CWB
ocbo4	Great deal of time spent with personal phone conversations. (R)	OCB	OCBO	CWB	CWB
ocbo5	Complains about insignificant things at work. (R)	OCB	OCBO	CWB	CWB
ocbo6	Conserves and protects organizational property.	OCB	OCBO	OCB	OCBO
ocbo7	Adheres to informal rules devised to maintain order.	OCB	OCBO	OCB	OCBO

Note: Items followed by (R) indicate that a reverse-worded item that was reverse-coded before analysis; Task = task performance; OCB = organizational citizenship behavior; OCBI = organizational citizenship behavior toward individuals; OCBO = organizational citizenship behavior toward organization; CWB = counterproductive work behavior or withdrawal.

^aThese items were judged to be conceptually consistent with both task performance and OCB definitions but were empirically consistent with the OCB factor. We note that items perceived as representing CWB or withdrawal in the substantive validity analysis should not be used as a standalone measure of CWB or withdrawal

Overview of Guidelines for Substantive Validity Analysis

In Figure 1, we illustrate the general steps involved in a substantive validity analysis. We note that prior works (e.g., Anderson & Gerbing, 1991; Hinkin, 1998; Hinkin & Tracey, 1999; Schriesheim, Powers, Scandura, Gardiner, & Lankau, 1993) describe various approaches to evaluating the content of items, so we do not detail that here. Rather, we focus our description on the steps researchers can take to select and articulate the construct definitions and select the sample for the assessment. The general purpose of substantive validity analysis is to evaluate each item on the extent it is judged to represent intended and unintended constructs. The results can be used to (a) demonstrate whether each item is judged as conceptually consistent with its intended construct and (b) determine the factor loading assignments for the revised factor models tested via the meta-analytic data. Importantly, substantive validity analysis evaluates the conceptual match between constructs and their operationalizations (Schriesheim et al., 1993) and reveals if items on a scale more closely correspond with the definition of an alternative construct.

OPEN IN VIEWER

Figure 1.

Illustration of steps for conducting item-level meta-analysis and substantive validity analysis for subsequent confirmatory factor analysis.

Step 1: Identify Relevant Constructs and Definitions

The first task of the substantive validity assessment is to select the constructs and definitions that are to be used for the evaluation. Specifically, researchers should identify the definition of the construct that the scale’s items are intended to represent as well as those the items are not intended to represent. First, to identify the intended constructs, it is important to start with the initial scale development paper. For example, the scale’s authors may specify the construct the items were developed to assess and may also provide definitions of these constructs. However, if a definition is not provided in this scale development paper, then it may be necessary to conduct a literature search to identify the construct’s definition, either from the scale author’s prior work or from the literature on the construct itself (i.e., not necessarily written by the scale’s authors).

To identify unintended construct definitions, first consider whether there is a construct that the items may better match. There may be an ongoing discussion in the literature about potential unintended constructs the items may represent. If not, then consult literature both inside and outside of the relevant domain and make subjective judgments to determine potential unintended constructs. Once unintended constructs have been identified, researchers can follow the same steps described previously to develop construct definitions.

In general, the definition used in the substantive validity analysis should be close to the original definition in the literature. However, if there are concerns about whether a sample will comprehend the definition, one solution would be to have an independent sample rate the extent to which they understand the definition; the sample could also provide qualitative comments to aid its understanding.

Step 2: Identify Relevant Sample for Substantive Validity Analysis

The next important decision is the type of sample to obtain for the substantive validity assessment. Substantive validity assessments require that respondents only evaluate the content of the item and not their own enactment of the behavior, which means that biases germane to a performance rating (e.g., leniency and social desirability) are not likely present. Additionally, respondents only need to have the ability to read items and definitions and make judgments about their match (Schriesheim et al., 1993). Anderson and Gerbing (1991) described that the nature of a substantive validity assessment—matching items to construct definitions—required judges who were “representative of the main study sample and population of interest” (p. 734). The authors also described the matching task as “less effortful for the judges, in keeping with their nonexpert character” (p. 734). As a result, substantive validity assessments are appropriately and effectively completed by non-experts, with samples of approximately 20 individuals being appropriate (though there is no definite standard for the size of the sample; Anderson & Gerbing, 1991).

One argument could be that it is more appropriate to have subject matter experts complete the substantive validity assessment. However, this depends on the purpose of the substantive validity assessment. Experts are likely to have more exposure to the items and/or construct definitions and, subsequently, provide ratings based on assumptions or familiarity rather than the item content itself (Schriesheim, Eisenbach, & Hill, 1991). If the purpose of the substantive validity assessment is to evaluate the judgments from people who will most likely encounter the item, then non-experts should be used for the assessment. Indeed, the value of substantive validity assessments lies in the degree of homogeneity in the perceptions and judgments regarding item content. For example, in the case of the present study, if most respondents perceive an item to represent CWB and not OCB, then this demonstrates that the item is conceptually consistent with the construct definition of CWB.

Step 3: Analysis of Substantive Validity Data

After data collection, the data should be converted to an item-level file such that the data for each item are in a separate row. The next step is to determine the number of times the item was perceived to represent each one of the construct definitions. Using Anderson and Gerbing’s (1991) guidelines, the next step is to compute the different substantive validity indices. The proportion of substantive agreement (p_sa ) is simply the proportion of respondents (out of total respondents) who judged the item to represent the intended construct. Next, the coefficient of substantive validity (C_sv ) should be computed to determine the extent to which each item was judged to match one construct over the other construct definitions. Importantly, this index accounts for the number of times an item was judged to represent an unintended construct in addition to the intended construct. This index is computed for each item using the following formula:

C s v = n c − n o N .

1

In the formula, n_c is the number of times an item was judged to represent its intended construct (e.g., an item on an OCB scale that was also perceived in the substantive validity assessment to match OCB); n_o is the largest number of times an item was judged to represent an unintended construct (e.g., an item on an OCB scale that was judged numerous times to match CWB). This quantity (n_c – n_o ) is divided by the total number of individuals who evaluated the item. Thus, C_sv values can range from 0 to ±1.00 and are interpreted as follows: (a) C_sv values approaching zero signal an ambiguous item that was perceived to represent multiple constructs; (b) large, positive C_sv values indicate an item consistently perceived to represent the intended construct; and (c) large, negative C_sv values indicate an item consistently perceived to represent an unintended construct.

These coefficients should also be tested for statistical significance using binomial tests. In line with Anderson and Gerbing (1991), critical C_sv values should be computed to determine whether an item was perceived to significantly represent a construct over the others. The null hypothesis is that there is a 50% chance that an item would be perceived as representing either n_c or n_o , and binomial tests determine the critical number of times an item would need to be sorted to either n_c or n_o to reject this null hypothesis. The binomial test is based only on n_c and n_o , so multiple binomial tests have to be conducted to account for the range of possible values of n_c + n_o . The critical number of sorts (p < .05) to either n_c or n_o needed to reject the null hypothesis is labeled m and used to calculate the critical C_sv with the following formula:

C s v ¯ = 2 m N − 1 ,

2

As mentioned previously, the value of m depends on the number of respondents who perceived an item to represent the intended construct (n_c ) or one unintended construct (n_o ). This means that each item could potentially have a different critical C_sv . For example, for an item with n_c + n_o = 20, m = 15 (p = .021), which means the critical C_sv = 0.50, p < .05. The critical C_sv is influenced by sample size—as the sample size increases, the critical C_sv can become rather small. Thus, researchers can evaluate both (a) the construct to which each item was most frequently assigned and (b) whether the item’s C_sv was statistically significant.

Overview of Guidelines for Item-Level Meta-Analysis and CFA

The purpose of the item-level meta-analysis is to cumulate item-level data to form a meta-analytic correlation matrix that is used for measurement model testing. We illustrate these steps in Figure 1 and describe them in the following.

Step 2: Item-Level Meta-Analyses and Creation of Meta-Analytic Correlation Matrix

Once data are coded for each inter-item correlation, the next step is to meta-analyze each correlation. Researchers should use bare bones meta-analysis to estimate inter-item population correlations from the observed sample size weighted correlations and the estimated variance of correlations across the samples (Schmidt & Hunter, 2015). If researchers have identified moderators, then it is important to conduct separate meta-analyses of the data comprising the different moderator categories. For example, since we evaluated rating source (self-rated vs. supervisor-rated) as a moderator in this study, we meta-analyzed the self-rated correlations separate from the supervisor-rated correlations.

Each inter-item meta-analytic correlation fills an overall meta-analytic correlation matrix that is used for the subsequent CFA. The next step is to convert the meta-analytic correlation matrix into a covariance matrix using the sample size weighted standard deviations. The final step is to determine the harmonic mean of the covariance matrix to serve as a constant sample size for the correlations included in the analyses (Shadish, 1996; Viswesvaran & Ones, 1995). We note that depending on moderator tests, a separate covariance matrix and harmonic mean need to be computed for each moderator category (e.g., separate covariance matrix and harmonic mean for self-ratings vs. supervisor ratings).

How to Evaluate Homogeneity

A typical meta-analysis also provides evidence regarding the homogeneity of the effect sizes across primary studies (Schmidt & Hunter, 2015). Thus, it is important to note here that these analyses can and should be conducted as part of an item-level meta-analysis. In addition to testing for the existence of moderators specified a priori (which we describe in the following), conducting a file drawer analysis will evaluate the extent to which the meta-analytic effect sizes are vulnerable to availability bias. Specifically, researchers can estimate the necessary number of samples with null results that would bring the observed meta-analytic effect size down to a specified value. As an example of what this means for item-level meta-analysis, if we find that correlations among task performance items range from .50 to .75, we can determine how many samples with null findings it would take to bring each correlation down to .10. The next step would be to use the formula:

x = k ( r k ¯ / r c ¯ − 1 ) ,

3

and for each item enter the k (number of samples), r k ¯ , the obtained mean effect size, and r c ¯ , the desired mean effect size (here, .10) and solve for x. Then, the next step would be to evaluate the number of needed studies for each item to determine the extent to which each item-level correlation is vulnerable to availability bias.

Empirical Example: Williams and Anderson’s (1991) Task Performance and OCB Scale

Part 1: Substantive Validity Assessment

Results

Item-level results are presented in Table 2. In the “perceived construct” columns are the number of respondents who perceived the item as representing one of the listed constructs. On the basis of the most frequently perceived construct and the C_sv index (see right-most column), we determined the “assigned construct,” which will be used for the subsequent CFA.

OPEN IN VIEWER

Table 2.

Results of Item-Level Substantive Validity Assessment.

Item Wording	Original Label	Assigned Construct	Perceived Construct				N	NC	NO	CSV
			Task Performance	OCB	Withdrawal	CWB
Adequately completes assigned duties	Task	Task	79	13	5	2	99	79	13	.67*
Fulfills responsibilities specified in job description	Task	Task	83	11	4	1	99	83	11	.73*
Performs tasks that are expected of him/her	Task	Task	83	14	0	2	99	83	14	.70*
Meets formal performance requirements of the job	Task	Task	80	15	1	3	99	80	15	.66*
Engages in activities that will directly affect his/her performance evaluation	Task	Task	56	19	10	14	99	56	19	.37*
Neglects aspects of the job he/she is obligated to perform	Task	Withdrawal	3	6	68	22	99	3	68	–.66*
Fails to perform essential duties	Task	Withdrawal	6	6	46	41	99	6	46	–.40*
Helps others who have been absent	OCB	OCB	16	77	4	2	99	77	16	.62*
Helps others who have heavy work loads	OCB	OCB	20	73	4	2	99	73	20	.54*
Assists supervisor with his or her work (when not asked)	OCB	OCB	24	70	4	1	99	70	24	.46*
Takes time to listen to coworkers’ problems and worries	OCB	OCB	8	82	4	5	99	82	8	.75*
Goes out of way to help new employees	OCB	OCB	19	76	2	1	98	76	19	.58*
Takes a personal interest in other employees	OCB	OCB	8	82	5	4	99	82	8	.75*
Passes along information to co-workers	OCB	OCB	23	68	2	6	99	68	23	.45*
Attendance at work is above the norm	OCB	OCB & TP	44	45	5	5	99	45	44	.01*
Gives advance notice if unable to come to work	OCB	TP	52	40	5	2	99	40	52	–.12*
Takes undeserved breaks	OCB	Withdrawal	0	3	88	8	99	3	88	–.86*
Great deal of time spent with personal phone conversations	OCB	Withdrawal	0	3	76	20	99	3	76	–.74*
Complains about insignificant things at work	OCB	Withdrawal	1	9	51	37	98	9	51	–.43*
Conserves and protects organizational property	OCB	OCB	40	54	5	0	99	54	40	.14*
Adheres to informal rules devised to maintain order	OCB	OCB	27	60	9	3	99	60	27	.33*

Note: N = number of respondents; Nc = number of judges who perceived the item as intended; No = number of judges who perceived the item as an unintended construct; Csv = coefficient of substantive validity; TP = task performance; CWB = counterproductive work behavior; OCB = organizational citizenship behavior. We note that items perceived as representing CWB or withdrawal in the substantive validity analysis should not be used as a standalone measure of CWB or withdrawal.

*p < .05.

First, we found that respondents perceived five of the seven items used to measure task performance as intended. That is, the content of five of the items used to measure task performance was perceived to represent task performance (average C_sv = .75). However, the two reverse-worded task performance items, “neglects aspects of the job he/she is obligated to perform” (C_sv = –.60) and “fails to perform essential duties” (C_sv = –.35), were most frequently classified as withdrawal, not task performance. Both of these C_sv values were significant, indicating that these items were more often perceived to represent negative work behaviors and not task performance.

Next, of the 14 items used to measure OCB, we found that 9 of these items were perceived to represent the intended construct (average C_sv = .51). As one example, the item “takes time to listen to coworkers’ problems and worries” had a C_sv = .75, indicating the respondents viewed this item as representing OCB. Our findings also revealed that two OCB items were judged to be ambiguous. Specifically, the item “attendance at work is above the norm” had a C_sv (0.01) that was near zero and was not statistically significant. We found that the sample was nearly split in classifying this item as either OCB (45%) or task performance (44%). Additionally, the item “gives advance notice if unable to come to work” had a low C_sv (–.12), which, despite being significant, suggested it was judged only slightly more often as representing task performance. Therefore, to further investigate the effect of these ambiguous items, we test alternative CFA models where these items loaded on an OCB factor and then on a task performance factor (see Part 2, described in the following).

Finally, the substantive validity assessment showed that each of the reverse-worded OCB items, “complains about insignificant things at work,” “takes undeserved breaks,” and “great deal of time spent with personal phone conversations,” were classified as withdrawal (average C_sv = –.67). Each item’s C_sv was significant, indicating it was more often judged to represent a negative work behavior and not OCB.

Our findings demonstrate that the content of many of the items on the Williams and Anderson (1991) scale were judged to be conceptually consistent with the intended construct but that other items were either ambiguous, meaning they were judged to reflect at least two constructs, or mislabeled, meaning that they were judged to represent an unintended construct. Next, we describe the steps to conduct an item-level meta-analysis to evaluate the relationships among the full set of 21 items. We then combine these findings to evaluate several meta-analytic measurement models to determine whether CFA model fit improves when items are reassigned to the construct they are judged to represent. ⁴

Part 2: Item-level Meta-analysis and CFA

Method

Results

Measurement Model Evaluation (Original and Revised)

The inter-item meta-analytic correlations are presented in Table 3. The values in the upper diagonal of Table 3 represent the supervisor-rated meta-analytic correlations, while the values in the lower diagonal are the self-rated meta-analytic correlations. The CFA factor loadings for each model are presented in Table 4, and the fit indices for the self- and supervisor-rated CFA models are presented in Table 5.

OPEN IN VIEWER

Table 3.

Meta-Analytic Correlations Among Task Performance and OCB Items.

		Inter-Item Correlations
Item	k	task1	task2	task3	task4	task5	task6	task7	ocbi1	ocbi2	ocbi3	ocbi4	ocbi5	ocbi6	ocbi7	ocbo1	ocbo2	ocbo3	ocbo4	ocbo5	ocbo6	ocbo7
task1	27		.59	.56	.56	.39	.30	.37	.47	.46	.50	.45	.46	.45	.42	.44	.39	.36	.31	.28	.39	.41
task2	27	.62		.61	.60	.39	.34	.38	.48	.47	.51	.45	.47	.44	.43	.46	.43	.41	.34	.29	.40	.42
task3	26	.60	.61		.59	.38	.35	.33	.49	.47	.52	.47	.48	.47	.42	.46	.40	.40	.34	.32	.40	.42
task4	27	.55	.59	.61		.43	.48	.39	.46	.45	.45	.44	.46	.44	.41	.43	.38	.38	.29	.30	.38	.37
task5	23	.41	.43	.45	.44		.21	.27	.37	.37	.39	.38	.40	.34	.36	.35	.27	.27	.25	.28	.36	.39
task6	27	.08	.09	.08	.07	.06		.49	.31	.34	.39	.30	.33	.33	.27	.35	.33	.49	.42	.42	.25	.29
task7	25	.09	.09	.12	.10	.04	.50		.31	.31	.38	.29	.28	.31	.26	.37	.33	.49	.41	.42	.30	.34
ocbi1	12	.30	.34	.32	.32	.23	.22	.17		.65	.53	.48	.54	.53	.48	.52	.36	.28	.22	.25	.35	.39
ocbi2	14	.30	.35	.33	.32	.25	.19	.17	.57		.56	.50	.55	.53	.47	.49	.36	.31	.22	.27	.35	.37
ocbi3	11	.24	.25	.31	.25	.26	.18	.25	.39	.43		.52	.55	.50	.45	.51	.34	.34	.27	.34	.38	.40
ocbi4	14	.33	.32	.32	.34	.28	.16	.18	.40	.43	.32		.59	.62	.54	.41	.35	.25	.22	.21	.36	.39
ocbi5	13	.29	.33	.31	.32	.26	.18	.18	.45	.48	.34	.43		.59	.57	.43	.37	.30	.23	.23	.40	.40
ocbi6	15	.29	.30	.28	.31	.26	.19	.15	.38	.39	.25	.49	.44		.53	.42	.38	.34	.26	.27	.43	.40
ocbi7	12	.29	.33	.31	.33	.30	.21	.19	.33	.32	.21	.42	.39	.50		.39	.37	.26	.20	.18	.37	.39
ocbo1	13	.30	.32	.34	.29	.24	.19	.20	.34	.34	.30	.24	.30	.30	.29		.44	.38	.31	.30	.36	.38
ocbo2	14	.33	.33	.34	.32	.26	.22	.18	.28	.26	.21	.27	.26	.31	.31	.35		.42	.32	.29	.37	.39
ocbo3	14	.26	.27	.27	.25	.20	.39	.35	.20	.20	.12	.22	.20	.23	.23	.27	.30		.54	.45	.25	.35
ocbo4	12	.24	.22	.25	.23	.17	.33	.30	.16	.15	.14	.11	.13	.11	.16	.18	.19	.36		.42	.23	.25
ocbo5	13	.18	.17	.22	.19	.18	.30	.28	.12	.15	.20	.09	.13	.07	.08	.14	.12	.34	.37		.22	.28
ocbo6	9	.33	.32	.32	.32	.24	.26	.25	.25	.25	.22	.30	.30	.33	.36	.28	.23	.30	.22	.21		.57
ocbo7	13	.33	.32	.34	.33	.29	.24	.17	.28	.28	.20	.30	.29	.32	.30	.29	.30	.27	.18	.20	.50

Note: Numbers in upper diagonal are the meta-analytic correlations among supervisor-rated items, and numbers in lower diagonal are the meta-analytic correlations among self-rated items. All meta-analytic correlations are significant at the p < .01 level; k indicates the number of studies in which the item was used. The harmonic mean for supervisor-rated and self-rated items are 1,686 and 1,889, respectively. task = task performance; ocbi = organizational citizenship behavior toward individuals; ocbo = organizational citizenship behavior toward organization.

OPEN IN VIEWER

Table 4.

Confirmatory Factor Analyses: Factor Loadings for Measurement Models.

Item	Williams and Anderson’s Original Three-Factor Model							Revised Three-Factor Model							Revised Four-Factor Model
	Supervisor Ratings				Self-Ratings			Supervisor Ratings				Self-Ratings			Supervisor Ratings					Self-Ratings
	Task	OCBI	OCBO		Task	OCBI	OCBO	Task	OCB	CWB		Task	OCB	CWB	Task	OCBI	OCBO	CWB		Task	OCBI	OCBO	CWB
task1	.72				.77			.74				.75			.74					.75
task2	.78				.79			.77				.78			.77					.78
task3	.76				.80			.76				.79			.76					.79
task4	.76				.77			.75				.75			.75					.75
task5	.47				.51			.55				.57			.55					.57
task6	.54				.16					.69				.66				.69					.66
task7	.54				.17					.68				.62				.68					.62
ocbi1		.74				.70			.73				.64			.74					.67
ocbi2		.76				.74			.73				.66			.74					.69
ocbi3		.59				.45			.72				.50			.72					.51
ocbi4		.75				.63			.72				.64			.73					.66
ocbi5		.79				.69			.76				.65			.77					.67
ocbi6		.75				.63			.74				.64			.75					.65
ocbi7		.69				.55			.68				.60			.69					.59
ocbo1			.67				.57		.65				.52				.68					.55
ocbo2			.69				.56		.54				.48				.61					.51
ocbo3			.56				.53			.74				.63				.74					.63
ocbo4			.53				.41			.65				.55				.65					.55
ocbo5			.46				.37			.61				.51				.61					.51
ocbo6			.63				.48		.57				.52				.69					.64
ocbo7			.69				.58		.58				.53				.72					.66

Note: task = task performance; ocb = organizational citizenship behavior; ocbi = organizational citizenship behavior toward individuals; ocbo = organizational citizenship behavior toward organization; CWB = counterproductive work behavior.

OPEN IN VIEWER

Table 5.

Meta-Analytic Confirmatory Factor Analysis Results.

Model Specifications	df	χ2	CFI	NNFI	RMSEA	SRMR	AIC
Supervisor rated
One-factor	189	2,993.49	.951	.945	.0938	.0652	21,695.50
Original two-factor (Task and OCB)	188	2,702.98	.956	.951	.0891	.0665	21,406.99
Original three-factor (Task, OCBI, and OCBO)	186	2,106.87	.966	.962	.0783	.0565	20,814.87
Revised two-factor (A) Task and OCB, (B) CWB	188	1,922.28	.970	.966	.0740	.0472	20,626.28
Revised three-factor (Task, OCB, and CWB)	186	1,474.61	.977	.975	.0641	.0445	20,182.62
Revised four-factor (Task, OCBI, OCBO, and CWB)	183	1,162.96	.983	.98	.0564	.0358	19,876.97
Comparison: Original three with revised three							ΔAIC = 632.255
Original three with revised four	Δdf = 3	Δχ2 = 943.904*
Revised three with revised four	Δdf = 3	Δχ2 = 311.649*
Self-rated
One-factor	189	3,943.02	.892	.88	.103	.0822	30,336.48
Original two-factor (Task and OCB)	188	2,999.67	.919	.91	.089	.0862	29,395.13
Original three-factor (Task, OCBI, and OCBO)	186	2,478.72	.934	.925	.081	.0784	28,878.18
Revised two-factor (A) Task and OCB, (B) CWB	188	2,882.60	.922	.913	.087	.067	29,278.06
Revised three-factor (Task, OCB, and CWB)	186	1,625.05	.959	.953	.064	.0533	28,024.52
Revised four-factor (Task, OCBI, OCBO, and CWB)	183	1,260.69	.969	.946	.056	.0446	27,666.15
Comparison: Original three with revised three							ΔAIC = 853.667
Original three with revised four	Δdf = 3	Δχ2 = 1,218.04*
Revised three with revised four	Δdf = 3	Δχ2 = 364.37*

Note: Task = task performance; OCB = organizational citizenship behavior; OCBI = organizational citizenship behavior toward individuals; OCBO = organizational citizenship behavior toward the organization; df = degrees of freedom; CFI = Comparative Fit Index; NNFI = Non-Normed Fit Index; RMSEA = root mean square error of approximation; SRMR = standardized mean square residual; AIC = Akaike Information Criterion.

For self-ratings, we first tested measurement models depicting the original labeling and structure of items. A single-factor model (χ² [df = 189] = 3,943.02, CFI = .89, NNFI = .88, RMSEA = .10, SRMR = .08, AIC = 3,0336.48) demonstrated marginal fit to the data, as did a two-factor model (comprising task performance and OCB factors; χ² [df = 188] = 299.67, CFI = .92, NNFI = .91, RMSEA = .09, SRMR = .09, AIC = 29,395.13). Williams and Anderson’s (1991) original three-factor structure (task performance, OCBI, and OCBO) provided a comparatively better fit to the data (χ² [df = 186] = 2,478.72, Δχ² = 520.95 [Δdf = 2, p < .05], CFI = .93, NNFI = .93, RMSEA = .08, SRMR = .08, AIC = 28,878.18).

We next tested a revised three-factor model in which the items that were consistently classified as CWB or withdrawal in the substantive validity analysis were reassigned to a separate negative work behavior factor. Thus, this revised three-factor model comprised task performance, OCB, and negative work behavior factors. For self-ratings, this revised model considerably improved model fit compared to the three-factor measurement model that was based on the original scale structure. Specifically, the chi-square statistic decreased from 2,478.72 to 1,625.05, while the CFI increased from .93 to .96, NNFI increased from .93 to .95, RMSEA decreased from .08 to .06, SRMR decreased from .08 to .05, and AIC decreased from 28,878.18 to 28,024.52. Because these models were not nested, we compared the AIC values (ΔAIC = 853.67), which indicated that the revised three-factor model fit better than the original three-factor model. This model included the ambiguous OCBO items “Attendance at work is above the norm” and “Gives advance notice if unable to come to work” loading on OCB. When these items loaded on the task performance factor, the model showed slightly worse fit (χ² [df = 186] = 1,787.17, CFI = .95, NNFI = .95, RMSEA = .07, SRMR = .06 AIC = 28,186.63 [ΔAIC = 162.11]), which indicated that these items should load on the OCB factor and not the task performance factor.

We also examined each item’s factor loading for the original versus revised models and found that reassigned items had larger factor loadings on the new factor compared to the original factor. For example, as shown in Table 4, the two reassigned task performance items (e.g., “neglects aspects of the job he/she is obligated to perform”) originally loaded .16 and .17 onto the task performance factor, but these loadings increased to .66 and .62, respectively, when they loaded on the new negative behavior factor. Similarly, the reassigned OCBO items (e.g., “takes undeserved work breaks”) had factor loadings of .37, .41, and .53 on the original OCBO factor, but these loadings increased to .51, .55, and .63, respectively, on the new negative behavior factor. Thus, the patterns of item factor loadings also support the revised models.

Next, we tested a revised four-factor model in which OCB was split into separate OCBI and OCBO factors. This revised four-factor model resulted in further improved model fit compared to the revised three-factor model. Specifically, chi-square dropped to 1,260.69, while CFI increased to .97 and NNFI increased to .95, RMSEA dropped to .06, SRMR decreased to .04, and AIC dropped to 27,666.15. This difference in chi-square was significant (Δχ² = 1,218.05 [Δdf = 3, p < .05]), indicating that the four-factor revised model fit significantly better than the three-factor revised model. Thus, for self-ratings, the revised four-factor model in which reverse-worded items were reassigned to a negative work behavior factor provided the best fit to the data compared to any of the model specifications based on the original labels.

As shown in Table 5, we tested these same models using supervisor ratings. For the original factor structure, the three-factor measurement model (i.e., task performance, OCBI, and OCBO) provided a marginal fit to the data (χ² [df = 186] = 2,106.87, CFI = .966, NNFI = .96, RMSEA = .08, SRMR = .06, AIC = 20,814.87). However, similar to self-ratings, the revised four-factor measurement model with reassigned items yielded considerable model fit improvement. Specifically, the chi-square decreased from 2,106.87 to 1,162.96 (Δχ² = 943.90 [Δdf = 3, p < .05]), the CFI increased from .97 to .98, NNFI increased from .96 to .98, RMSEA decreased from .08 to .06, SRMR decreased from .06 to .04, and AIC dropped to 19,876.97. We also evaluated each item’s factor loading (see Table 4) and found that each reassigned item had a factor loading that was larger for the revised factor compared to the original factor. For example, the reassigned task performance items both had loadings of .54 on the original factor, but they increased to .68 and .69 on the negative work behavior factor. Thus, item-level meta-analytic CFA demonstrated that the original factor structure suggested by Williams and Anderson (1991) is not the best fitting measurement model for the items. Rather, a revised four-factor model that accounts for each item’s substantive validity yielded the best-fitting model. ⁵ A summary of the results for the items is presented in Appendix A.

Moderator Analysis: Rating Source as an Example

As noted previously, an important contribution of an item-level meta-analytic CFA is the ability to evaluate whether the factor structure and item factor loadings are affected by moderators. ⁶ Therefore, we evaluated whether our findings were moderated by the use of self-ratings or supervisor ratings to rate the items. Specifically, we evaluated the extent to which these measurement models were invariant across rating sources. As noted previously, these procedures can be used to evaluate the moderating influence of many types of categorical moderator variables (e.g., publication type, year, study context, industry). We followed the recommendations outlined by Vandenberg and Lance (2000) and tested a (a) configural invariance model and (b) more restrictive metric invariance model. We note that the same harmonic means for the CFA (self = 1,889; supervisor = 1,686) were used for invariance testing.

The results of the measurement invariance tests are presented in Table 6. First, we used multigroup CFA to test configural measurement invariance and evaluate whether the 21 Williams and Anderson (1991) items loaded on the same revised four factors for both self- and supervisor ratings. We expected that the chi-square statistic would be significant given the large sample sizes due to the use of meta-analytic harmonic means (χ² [df = 366] = 2,422.29), but the remaining fit indices evaluated provided support for configural measurement invariance of the revised four-factor scale (CFI = .98, NNFI = .97, RMSEA = .06, SRMR = .04, AIC = 2,787.85). This indicates that the revised four-factor structure (with no additional constraints) is supported for both supervisor and self-ratings.

OPEN IN VIEWER

Table 6.

Moderator Analysis: Measurement Invariance Across Rating Sources.^a,b

Model Specifications Across Subgroups	df	χ2	Model Comparison	Δdf	Δχ2	CFI	NNFI	RMSEA	SRMR	AIC
Rating source: Supervisor (n = 1,686, k = 20) versus self (n = 1,889, k = 8)
Revised four-factor
Configural measurement invariance	366	2,422.29				.98	.97	.06	.04	2,787.85
Metric measurement invariance	387	2,606.25	Configural versus metric	21	183.96	.98	.97	.06	.08	2,957.69

Note: df = degrees of freedom; CFI = Comparative Fit Index; NNFI = Non-Normed Fit Index; RMSEA = root mean square error of approximation; SRMR = standardized mean square residual; AIC = Akaike Information Criterion; n is the harmonic mean for each rating source; k indicates the number of studies in which the item was used.

^aConfigural measurement invariance is a model in which no restrictions other than the form of the model are imposed across subgroups.

^bMetric measurement invariance is the same model as partial measurement equivalence with the addition of equality constraints on the factor loadings for each item.

The next step was to test metric invariance to determine whether the actual factor loadings were the same across supervisor and self-ratings. That is, this step further tests whether the relationships between factors and items are the same across rating sources. A multigroup CFA model was tested with additional equality constraints for the factor loadings across groups. The chi-square statistic was again significant as expected (χ² [df = 387] = 2,606.25), but the additional fit indices also provided support of metric invariance (CFI = .98, NNFI = .97, RMSEA = .06, SRMR = .08, AIC = 2,957.69). Indeed, the fit indices did not drop substantially from testing the configural to the metric invariance model.

Discussion

It is premature to conclude the scale development process after only one or a few primary validation studies as the initial evidence that supports and validates the use of scale items may not be as robust when the items are used in varied contexts and with diverse samples. Unfortunately, the items are not usually the focus of subsequent empirical studies using the scale, which makes it difficult for researchers to identify and evaluate whether critical details about a scale’s items—factor loadings, factor structure, and the generalizability of both—actually function as initially promised. The purpose of this study was to illustrate the importance of considering the items when comprehensively (i.e., meta-analytically) evaluating a scale’s performance. Our analysis demonstrated that some of the items on Williams and Anderson’s (1991) task performance and OCB scale were judged via substantive validity analysis to represent constructs other than task performance and OCB and, importantly, that the best fitting factor structure was different from what was initially proposed and validated. Indeed, our findings suggest that the current understanding of the steps constituting scale development be amended to encourage more attention to how the items perform beyond the initial validation effort.

The more commonly employed scale-level meta-analyses are without question important tools for understanding nomological network relationships as well as how such relationships are influenced by scale- and study-level moderators. However, meta-analytic construct validation studies (e.g., Hoffman, Blair, Meriac, & Woehr, 2007; LePine, Erez, & Johnson, 2002) are unable to determine how the actual items influence the underlying factor structure or, more importantly, inform alternative models that fit the data better. Findings from our study indicate that meta-analysts should also strive to incorporate an item-level focus in order to more thoroughly understand whether a construct’s operationalization remains a valid representation of the construct.

Recommendations for and Implications of Item-Level Substantive Validity and Meta-Analytic Approaches

The recommended steps for scale development should be expanded to include meta-analytic examinations of the items once the scale is in use. Indeed, as we illustrate in Figure 1, there are several important reasons to meta-analytically evaluate the empirical evidence about scale items, including concerns about: (a) the discriminant validity of scale factors, (b) the substantive validity of the scale’s items, (c) the empirical fit of the scale’s items (e.g., low or varying factor loadings or coefficient alpha), or (d) the initial scale development effort. This is certainly not an exhaustive list, but should these or other concerns exist and enough data have accumulated for meta-analysis, then we recommend an item-level meta-analysis. This may mean that item-level meta-analytic construct validation studies can be conducted after a scale has been administered in enough independent samples to meta-analyze all of the inter-item correlations, which will likely take several years.

Although the purpose of this demonstration was to illustrate the steps researchers can take to validate scales that are already in use, we note that the approaches we describe (item-level meta-analytic CFA, substantive validity) could also be conducted as part of the initial scale development process. Indeed, if a scale development paper provided empirical support for the underlying measurement model and factor loadings that was based on numerous samples (e.g., presumably obtained from different contexts, industries, rating sources), researchers would have greater confidence that the scale is likely to work as promised. It follows that we also recommend that scale developers consider adapting prospective meta-analytic procedures (e.g., Berlin & Ghersi, 2005) by hypothesizing potential moderating variables that may change the initial understanding of the scale. This could provide researchers with a protocol for the variables to include with the scale items and eventually allow for the use of meta-analysis to evaluate the influence of the moderator.

There are numerous issues that may arise as part of the item-level evaluation. For example, we described how researchers can ascertain the extent to which the item-level sample data are homogenous or reflect the same population. If researchers find evidence of heterogeneity (variance of population effect sizes is greater than zero; Schmidt & Hunter, 2015; or Q-statistic, Hedges & Olkin, 1985), it is important identify possible sources. First, identify whether the heterogeneity can be explained by additional moderator variables. Second, determine whether the heterogeneity is caused by primary second-order sampling error (Schmidt & Hunter, 2015). That is, if the total sample size is small, this can confuse the understanding of whether the item-level sample data represent the same population effect size. In our demonstration, we were only able to obtain a small number of samples for some item pairs. Thus, evidence of heterogeneity for these particular items may not indicate the existence of moderators but rather, the need for additional data. This also signals the importance of conducting a file-drawer analysis to illustrate the vulnerability of obtained effect sizes to additional null findings.

We identified moderator variables on the basis of noted issues with the Williams and Anderson (1991) scale. However, the set of moderators we examined (i.e., rating source) was not exhaustive, and we must note additional general variables that may influence item-level effect sizes in future meta-analyses of other scales. For example, if the scale’s items were initially subjected to a substantive validity analysis or another type of content validity analysis, then researchers could correlate the item-level indices (e.g., substantive validity coefficient) with items’ factor loadings. Researchers should also consider moderator variables pertaining to the match between (a) the initial scale validation study and (b) subsequent primary studies using the scale. For example, researchers should evaluate if characteristics of the validation sample (e.g., demographic characteristics, country, industry) are similar to or different from each subsequent primary study. In addition, researchers should consider the match of the item administration features across studies. Researchers may take license in the number of scale items administered, type of response scale, type of rater, number of response scale anchors, language, and item wording. It is important to evaluate whether these adjustments influence the meta-analytic measurement model.

In our study, for example, we found that some of our obtained samples administered both task performance and OCB items, whereas others focused on only one of the dimensions. We evaluated whether these administrative differences influenced the results by creating separate correlation matrices for samples that measured the items comprising both dimensions and those that measured the items for only one of the dimensions and then tested the same measurement models we described previously. We found that our results were the same for both of these conditions. We report the results of these analyses in Appendix C.

Our previous discussion of primary second-order sampling error highlights a limitation of our demonstration. Though Williams and Anderson (1991) has been cited over 3,000 times, ⁷ we were only able to obtain item-level data from 27 independent samples. One reason for this was the amount of time since the scale’s initial publication in 1991—many authors we contacted were willing to help but no longer had data that were over 20 years old. We suspect this will be a recurring challenge for others seeking to conduct similar analyses. Nonetheless, the challenges associated with obtaining item-level data are outweighed by the important benefits of item-level meta-analytic construct validation studies. To aid future cumulation efforts, we encourage scientists and journals to consider providing more data registries where researchers can provide their item-level data for further analysis. Additionally, it is important that journal editors require scale developers to provide detailed item-level statistics in published studies or ensure that such data are made easily available to those who will use the scale in the future. In any case, the necessary insights that can be gained via item-level meta-analytic CFAs can only be realized if they are based on comprehensive data provided by the researchers.

Item-level meta-analytic CFA also enables researchers to further evaluate alternative explanations for the CFA results. For example, one plausible explanation for the issues pertaining to reverse-worded task performance and OCB items is that these items should load on their originally specified factors but also on a method factor that is uncorrelated with the task performance, OCBI, and OCBO factors. Indeed, demonstrating that this measurement model does not fit better than the revised model that introduces a new trait factor (e.g., CWB) is important. As illustrated in Appendix D, we tested these models with our meta-analytic data and found that the four-factor revised models (with trait factors) fit the data better than the original three-factor model with an additional uncorrelated method factor.

We provide detailed instructions for how the item-level meta-analytic CFA (and substantive validity analysis) can be conducted, which makes it easier for scholars to use these techniques to evaluate the existence and effects of similar item-level issues. As one example, organizational commitment scholars have also grappled with the use of reverse-worded items on affective commitment scales (Magazine, Williams, & Williams, 1996; Merritt, 2012), indicating that item-level meta-analytic CFA would be useful. The use of reverse-worded items represents only one potential item-level issue affecting the fit of measurement models as scales may have myriad other operationalization issues in need of item-level meta-analytic examination.

An important follow-up question from our results is whether the changed dimensional structure also changes the relationships task performance, OCBI, and OCBO have with their respective nomological networks (e.g., job satisfaction, organizational commitment). This was an important question we hoped to address in our study, but we found that the primary studies we obtained did not include overlapping sets of correlates. Additionally, assessing how the reassignment of items influenced nomological relations requires item-level data among the focal constructs and nomological correlates. Future research seeking to conduct item-level meta-analysis should request and collect the item-level data required to construct an item-level meta-analytic correlation matrix comprised of all desired constructs.

Finally, our substantive validity assessment follows recommendations from Anderson and Gerbing (1991) and, accordingly, uses (via crowd-sourcing) a cross-sectional sample of lay employees rather than experts. This strengthened our approach since we were able to obtain perceptions of item content from individuals who are likely to encounter these items in practice. Nevertheless, it is important to consider whether lay persons differ from scholarly experts in perceptions of item content. Though Schriesheim et al. (1991) contended that experts were likely to be biased by their knowledge of the constructs, this ultimately stands as an important empirical question for future research. The substantive validity analysis signals a need to more thoroughly evaluate respondents’ perceptions of item content. This has certainly been described in prior research (e.g., Hinkin, 1998; Hinkin & Tracey, 1999; Schriesheim et al., 1993), but we assert the importance of considering a broader range of competing constructs when evaluating items.

Implications for the Williams and Anderson (1991) Scale

Our empirical demonstration revealed that the original factor structure for the items on the Williams and Anderson (1991) scale was not wholly supported. Specifically, the item-level meta-analytic CFA corroborated the results of the substantive validity analysis—reverse-worded task performance and OCB items belong on a new negative work behavior factor. Importantly, we demonstrated that this revised factor structure held for both self- and supervisor ratings. As a next step, it will be important to evaluate whether the revised factor structure is invariant across other moderator types, such as publication status, industry, job types, and other relevant variables. As we mentioned, we did not obtain adequate data on moderator categories for each of the 231 correlations in our meta-analytic correlation matrix. However, as these data become available, it will be important to further assess invariance. We emphasize that these reverse-worded items should not be used as a standalone scale of withdrawal or CWB as they do not adequately cover the construct domain as defined (e.g., see Robinson & Bennett, 1995). Future research is needed that clarifies if and how these reverse-worded items are to be used moving forward. Nevertheless, our findings demonstrate that these items are not appropriate for use on task performance and OCB scales.

Similarly, as the items “attendance at work is above the norm” and “gives advance notice if unable to come to work” were ambiguous, this indicates that it is important to consider removing these from the Williams and Anderson (1991) scale as well. Indeed, these findings suggest that overlapping items may constitute an important reason why task performance and OCB often display questionable empirical distinctiveness (e.g., Hoffman et al., 2007). As a result, our findings regarding the items reflect important confirmation that the concepts of task performance and OCB are somewhat intertwined. To continue to understand the similarities and distinctions between task performance and OCB without the influence of overlapping items, we recommend that researchers evaluate whether and how results change when the ambiguous items are removed versus retained.

Researchers may encounter situations in which they have administered the Williams and Anderson (1991) scale in a local validity study and found that the best-fitting measurement model differed from this study’s measurement model that was supported via meta-analytic data (see Table 5). Indeed, this study demonstrates that results from a single sample will likely vary considerably from results obtained with meta-analysis. Thus, if this happens, it raises an important dilemma for researchers: Which measurement model should be used with the primary study data? To be clear, this is a delicate issue that requires more future attention by the larger research methods field.

One suggestion is to conduct empirical Bayes meta-analysis (e.g., Brannick, 2001; Newman, Jacobs, & Bartram, 2007) to compare the item-level correlation matrices from the local and meta-analytic data and subsequently evaluate whether the local data (despite fitting a different measurement model) nevertheless fit the established population parameters suggested by the meta-analysis. In Bayesian terms, the inter-item correlation matrix we present in Table 3 would serve as the prior distribution (i.e., estimates the population effect sizes), and the local validity data would comprise the likelihood distribution (i.e., provides a new estimate of the population effect sizes). Combining the prior and likelihood distributions would produce a new distribution—termed the posterior distribution—that yields an updated estimate of the effect size and variability. If this estimate from the posterior distribution is statistically significant, then the likelihood distribution is valid. We note that researchers using empirical Bayesian methods to compare local data to our meta-analytic data would need to use the random effects model (Brannick, 2001) and would also have to evaluate each inter-item correlation in the matrix in order to determine whether the data obtained from the local validity study indeed generalizes. To our knowledge, Bayesian analyses have not yet been applied in this manner to a large meta-analytic correlation matrix. Thus, future research is needed that develops clear guidelines on these techniques so that researchers can more quickly determine how to use their local study data.

Finally, we emphasize that our findings are not intended to condemn the use of Williams and Anderson’s (1991) scale. On the contrary, our findings suggest that authors should use many of the items but organize them in line with the revised four-factor structure for analysis. Indeed, given the common use of the items, researchers likely need to make only minor modifications to analyze the items in a reorganized, substantively valid manner. Furthermore, given that the Williams and Anderson (1991) items are similar to other existing scales (e.g., as shown in Table 7, of the 14 original OCB items from the Williams and Anderson scale, 7 are also measured in either the Smith et al., 1983, or Podsakoff et al., 1990, OCB scales), we expect that our findings also apply to similar OCB scales in the literature, although this remains an empirical question requiring additional research and attention.

OPEN IN VIEWER

Table 7.

Match Between Williams and Anderson’s (1991) Items and the Wording of Published Organizational Citizenship Behavior Items.

	Williams and Anderson’s (1991) Items	Matched Items
1	Adequately completes assigned duties.	Adequately completes responsibilities (Van Dyne & LePine, 1998).
2	Fulfills responsibilities specified in job description.	Fulfills the responsibilities specified in his/her job description (Van Dyne & LePine, 1998).
3	Performs tasks that are expected of him/her.	Performs the tasks that are expected as part of the job (Van Dyne & LePine, 1998).
4	Meets formal performance requirements of the job.	Meets performance expectations (Van Dyne & LePine, 1998).
5	Engages in activities that will directly affect his/her performance evaluation.	—
6	Neglects aspects of the job he/she is obligated to perform. (R)	—
7	Fails to perform essential duties. (R)	—
8	Helps others who have been absent.	Helps others who have been absent (Podsakoff et al., 1990; Smith et al., 1983).
9	Helps others who have heavy workloads.	Helps others who have heavy workloads (Podsakoff, MacKenzie, Moorman, & Fetter, 1990; Smith, Organ, & Near, 1983).
10	Assists supervisor with his/her work (when not asked).	Assists supervisor with his/her work (Smith et al., 1983).
11	Takes time to listen to co-workers’ problems and worries.	—
12	Goes out of way to help new employees.	Orients new people even though it is not required (Smith et al., 1983).Helps orient new employees in this group (Van Dyne & LePine, 1998).Helps orient new people even though it is not required (Podsakoff et al., 1990).
13	Takes a personal interest in other employees.	—
14	Passes along information to co-workers.	—
15	Attendance at work is above the norm.	Attendance at work is above the norm (Podsakoff et al., 1990; Smith et al., 1983).
16	Gives advance notice when unable to come to work.	Gives advance notice if unable to come to work (Smith et al., 1983).
17	Takes undeserved work breaks. (R)	Takes undeserved work breaks. (Smith et al., 1983)Taken an additional or a longer break than is acceptable at your workplace (Bennett & Robinson, 2000).Took an extended coffee or lunch break (Skarlicki and Folger, 1997).Taken a longer break than you were allowed to take (Spector et al., 2006)
18	Great deal of time spent with personal phone conversations. (R)	Great deal of time spent with personal phone conversations (Smith et al., 1983).Spent time on personal matters while at work (Skarlicki and Folger, 1997).
19	Complains about insignificant things at work. (R)	Consumes a lot of time complaining about trivial matters (Podsakoff et al., 1990).
20	Conserves and protects organizational property.	—
21	Adheres to informal rules devised to maintain order.	—

Note: Items followed by (R) indicate that a reverse-worded item that was reverse-coded before analysis.

Conclusion

Though initial scale validation papers are expected to document the empirical support for each item’s inclusion on the scale, the focus on items usually ends once the scale is in use. This study demonstrates the need for greater attention to items as the scale is reexamined, particularly via meta-analysis. We provided step-by-step instructions for conducting item-level meta-analysis and substantive validity analysis and for combining these techniques for meta-analytic CFA. These techniques are critical for evaluating the initial scale development processes, improving construct measurement, and understanding the evolving construct validity evidence for the scale.