Responding to Low Coefficient Alpha: Potential Alternatives to the File Drawer

Background of the Study

A recent review of the “Questions Forum” on Research Gate revealed 35 posts asking for guidance on how to continue research analysis when coefficient alpha is below .70. Coefficient alpha .70 is widely accepted as the benchmark for score reliability (Kaplan & Saccuzzo, 1982; Murphy & Davidshofer, 1988; Nunnally, 1978). As such, Bonett and Wright (2015) stated that they “have both heard numerous reports where manuscripts were rejected simply because the sample value of coefficient alpha was below 0.70” (p. 4). Also, Greco et al. (2018) reviewed 1,296 studies made up of 1,675 independent samples and found that alphas consistently exceeded .70 and by and large were above .80.

Intrigued by the discussion board comments, we considered the possibility that many researchers do not know how to proceed with data when alpha is low. Consequently, studies often end up in the proverbial file drawer (cf. Bosco, 2018). We agree with scholars (e.g., Thompson, 2003) that assessing internal reliability is essential to rigorous research. However, even if a reliability coefficient is low, the data or subsets of the data may still be useful for analysis that enriches empirical, conceptual, and theoretical understandings. For these reasons, we developed a guide to help scholars and scholar–practitioners process data that initially yield low estimates of reliability that can contribute meaningfully to theory building.

Classical Test Theory and Reliability

The concept of reliability was presented in 1904 when Spearman formulated ways to evaluate score reliability that are known today as classical test theory (Thompson, 2003). Classical test theory assumes that observed scores are comprised of true score and measurement error and that total score variance is comprised of observed score and measurement-error variance. Unlike other reliability theories, classical test theory does not consider systematic error in the estimation of measurement error.

In 1910 Spearman and Brown articulated the Spearman–Brown prophecy formula. The formula “provides a rough estimate of how much the reliability of test scores would increase or decrease if the number of observations or items in a measurement instrument were increased or decreased” (Brown, 2018, p. 1558). Commonly known as split-half reliability, a correlation coefficient for each half of the instrument, commonly the odd-numbered and even-numbered items, is calculated. The Spearman–Brown formula is applied to the estimate of the reliability of each half to calculate the full-test reliability (Brown, 2018). Scholars criticized the split-half technique, stating that “the two coefficients are measures of different qualities and should not be identified by the same unqualified appellation reliability” (Cronbach, 1951, p. 298).

In 1931, Kuder and Richardson, critics of the split-half technique, developed two algebraic equations (the Kuder–Richardson Formula 20 and the Kuder–Richardson Formula 21) to extract the coefficient of equivalence, an estimation of reliability, from one set of data (Cronbach, 1951). Thompson (2003) explained that “the seminal formula for an estimate of internal consistency can be traced to the famous algorithm presented as the 20th formula within the article by Kuder and Richardson (1937)” (p. 11). The Kuder–Richardson Formula 20 became the focus of Cronbach’s (1951) Psychometrika article that introduced coefficient alpha. The formulas are conceptually similar. However, the Kuder–Richardson Formula 20 works only for dichotomous item scores, while coefficient alpha can be computed from dichotomous and nondichotomous item scores (Thompson, 2003). Sixty-eight years later, Cronbach’s article has been cited 43,259 times (Google Scholar, February 11, 2020).

Coefficient Alpha

According to Cronbach (1951), coefficient alpha is the “mean of all possible split-half coefficients” (p. 331). From this perspective, Henson (2001) cautioned researchers to realize “that internal consistency coefficients are not direct measures of reliability, but rather are theoretical estimates derived from classical test theory” (p. 177). Cronbach stated that “alpha estimates the proportion of the test variance due to all common factors among items. That is, it reports how much the test score depends upon general and group rather than item specific factors” (p. 320). With that in mind, Gronlund and Linn (1990) explained that “reliability refers to the results obtained with an evaluation instrument and not to the instrument itself. Thus, it is more appropriate to speak of the reliability of ‘test scores’ or the ‘measurement’ than of the ‘test’ or the ‘instrument’” (p. 78). Indeed, the idea that tests are not reliable led Vacha-Haase (1998) to propose a measurement meta-analytic technique, reliability generalization and for Thompson and Vacha-Haase (2000) to recognize that “psychometrics is data metrics” (p. 174). As such, coefficient alpha is influenced by sample characteristics (sample size and heterogeneity or homogeneity), the number of test items, or interrelatedness between items (Multon & Coleman, 2010; Tavakol & Dennick, 2011). For these reasons, it is imperative to measure reliability for every study conducted rather than citing reliability estimates from previous research.

As noted by Henson (2001, pp. 180–182), coefficient alpha can be calculated with either equation 1 or 2, where k equals the number of items in the scale, Σ σ k 2 equals the sum of all the k item variances, σ TOTAL 2 equals the variance of the total scale scores, and Σ COV ( for i < j ) × 2 ) = the doubled sum of the unique covariances :

α = k ( k − 1 ) [ 1 − ( Σ σ k 2 σ TOTAL 2 ) ] ,

(1)

α = k ( k − 1 ) [ 1 − ( Σ σ k 2 ( Σ σ k 2 + Σ COV i j ( for i < j ) × 2 ) ) ] .

(2)

If data are standardized, alpha is simply a function of the average interitem correlation (r) as shown in equation 3 (Helms et al., 2006, p. 640; Kopalle & Lehmann, 1997):

α = k r ( 1 + [ k − 1 ] r ) .

(3)

The higher the value, the greater the reliability. Coefficient alpha of .70 means that 30% of the variance in the data is theoretically measurement error. Acceptable ranges for alpha vary among scholars. Nunnally (1967) stated that, in “the early stages of research on predictor tests or hypothesized measures of a construct, . . . reliabilities of .60 or .50 will suffice” (p. 226). He further recommended .80 for basic research and .90 for applied settings in which the cutoff score is important. In the second edition of his seminal work, Nunnally (1978) raised the benchmark for exploratory research to .70 (pp. 245–246). As noted by Henson (2001) and others, this led many researchers to reference Nunnally (1978) if they attained a coefficient alpha of .70 or higher and Nunnally (1967) if they attained a coefficent alpha of .60 or .50. Kaplan and Saccuzzo (1982) argued that .70 to .80 is an acceptable range for basic research, and for practical research, the level should be .95 (p. 106). With a seemingly more tolerant approach, Murphy and Davidshofer (1988) stated that below .60 is unacceptable, .70 is low, .80–.90 is moderate to high, and .90 is high. Multon and Coleman (2010) explained that the acceptable range should be based partly on the context of what is being measured. Although scientists continue to disagree on the acceptable level of coefficient alpha, the levels suggested by Murphy and Davidshofer (1988) and Nunnally (1978) are the most cited (Google Scholar, Feburary 11, 2020).

Coefficient alpha is ideal for HRD researchers due to its effective reliability assessment of traits such as job satisfaction, organizational justice, and workplace behaviors (Multon & Coleman, 2010). Two additional benefits worth noting are (a) that confidence intervals (CIs) can be constructed around alpha and (b) that coefficient alpha is robust to violations of underlying assumptions (Reinard, 2006).

Fan and Thompson (2001) urged authors to report CIs with score reliabilities along with the estimation method used to determine the values. CIs for coefficient alpha are calculated as indicated in equations 4 and 5, where a ^ = sample coefficient alpha, γ = significance level, F = F distribution for percentiles for γ/2 and 1 − γ/2, n represents the sample size, k = number of items, df1 = n − 1, df2 = (n − 1) (k − 1) (Thompson, 2003, p. 75):

CI lower = 1 − [ ( 1 − α ^ ) F ( 1 − γ / 2 ) , d f 1 , d f 2 ] ,

(4)

CI upper = 1 − [ ( 1 − α ^ ) F γ / 2 , d f 1 , d f 2 ] .

(5)

There are four assumptions to coefficient alpha: (a) the scale adheres to tau equivalence; (b) scale items are on a continuous scale and have the same or reasonably same distribution shapes; (c) the errors of the items do not covary; and (d) the scale is unidimensional (McNeish, 2018, p. 414). These assumptions are also limitations to coefficient alpha. As it relates to the first assumption, McNeish (2018) explained that “tau equivalence is the statistically precise way to state that each item on a scale contributes equally to the total scale score” (p. 415). To meet this assumption, the standardized factor loadings for every item would have to be practically analogous. For the second assumption, most software implementations of coefficient alpha (e.g., R, SAS, and SPSS) calculate item covariances using a Pearson covariance matrix. Pearson covariance matrices assume that the variables are continuous and normal. However, if data are binary, coefficient alpha will underestimate the level of reliability (McNeish, 2018). Third, coefficient alpha assumes that the item errors do not covary. There are a number of situations that may cause the errors to correlate, resulting in an over- or underestimation of alpha (e.g., the order of the items on the scale, transient responses, or unmodeled multidimensionality; McNeish, 2018). Finally, “unidimensionality is the degree to which the items all measure the same underlying construct” (Schmitt, 1996, p. 350). Schmitt (1996) pointed out that “internal consistency refers to the interrelatedness of a set of items, whereas homogeneity refers to the unidimensionality of the set of items” and the “confusion in the literature involves the use of homogeneity and internal consistency as though they were synonymous” (p. 350).

Generalizability Theory and Reliability

With major developments in mathematical statistics, Cronbach no longer considered “the alpha formula as the most appropriate way to examine most data” (Cronbach & Shavelson, 2004, p. 403). Based on his revelation, Cronbach began to sketch the components of variance approach to reliability from which he and his associates developed the generalizability theory (G theory; Cronbach et al., 1963, 1972).

Cronbach observed that

Measurement specialists have often spoken of a test as a sample of behavior, but the formal mathematical distinction between sample of persons and populations of persons, or between a sample of tasks and a population [a universe] of tasks, was rarely made in writings on test theory in 1951 and earlier. (Cronbach & Shavelson, 2004, p. 404).

Quantitative HRD research is not as interested in any one score; rather, it is interested in how that score can be applied to behavior in a broader context. Cronbach et al. (1972) called this the universe score. Shavelson et al. (1989) explained that “a score’s usefulness, then, largely depends on the extent to which it allows us to generalize accurately to behavior in some wider set of situations, a universe of generalization” (p. 922). G theory “recognizes that there may be multiple definitions of true- and error-scores.” Multiple sources of error define the universe of generalization, a fundamental notion in G theory that “a measurement taken on a person records a sample of behavior” (i.e., a score; Shavelson et al., 1989, p. 922). Cronbach et al. (1972) surmised that “the question of reliability thus resolves into a question of accuracy of generalization, or generalizability” (p. 15).

As previously mentioned, coefficient alpha provides an estimate of true score variance. For example, α = .70 suggests that the remaining 30% is measurement error. However, the source of the measurement error variance is unknown. On these grounds, when coefficient alpha is low, a G study can be performed to partition observed variance across effects (e.g., person, item, and the interaction between person and item, along with model specification error). The effects can be used to compute a generalizability coefficient that results in the same value as coefficient alpha (Webb et al., 2006).

Our discussion of G study reflects the parsimonious one-facet design where an observed score for a particular person on a particular item is decomposed into an effect for the grand mean, plus effects for the person, the item, and a residual including the interaction between person and item and unsystematic error (Webb & Shavelson, 2005). Since person is the object of measurement, it is not a source of error and as such, item is the only facet (i.e., one-facet design; Webb & Shavelson, 2005). In a one-facet design, observed variance is decomposed into three components, as shown in Table 1 (cf. Shavelson & Webb, 1981, p. 2).

OPEN IN VIEWER

Table 1.

Observed Score, Mean Variance, and Generalizability Coefficient Variance Components for p × i Design.

Observed score		Mean variance					Generalizability coefficient
Effect	Component	Effect	Score variance component	F	MVC	MVC percentage	Effect	MVC(s)
I	μ i − μ	i	σ i 2	K	σ i 2 f	MVC i Σ MVC
p	μ p − μ	p	σ p 2	1	σ p 2 f	MVC p Σ MVC	Systematic	MVC p
pi, e	X p i − μ p − μ i + μ	pi, e	σ p i , e 2	K	σ p i , e 2 f		Measurement Error	MVC p i , e
Total	X p i	Total			Σ MVC	MVC p i , e Σ MVC	Total	MVC p + MVC p i , e

Note. X = observed score; p = person; i = item; pi = Person × Item; e = error; MVC = mean variance component. Generalizability coefficient = systematic/total.

The item effect (i) is the variance of constant errors related to the item; the person effect (p) is equivalent to the true-score variance of classical test theory; and the remaining effect pi, e represents the interaction between the person and item effects along with unidentified sources of measurement error (Shavelson & Webb, 1981). Usually, G theory presumes that all the ways of the measurement design are random effects (Thompson, 2003, p. 52). Therefore, the score variance components are typically derived using processes that model random effects (e.g., hierarchical linear modeling). Each mean variance component (MVC) is calculated by dividing the score variance component by the frequency of the effect. The MVC of each effect is divided by the total MVC to estimate the component percentage (see Table 1).

The component percentages in G theory can inform the researcher. Take, for example, dichotomously scored items such as those that can be evaluated as right or wrong. A large component percentage for i might suggest that the items varied in difficulty and were a source of unwanted measurement error. A large component percentage for p would suggest that the sample differed systematically in their feelings of the factors measured. A large component percentage for pi, e would suggest that the relative standings of persons varied across items. With that said, remember that the MVC for pi, e contains the interaction between person and item as well as unspecified measurement error variance which cannot be subpartitioned from the pi variance, due to degree of freedom restrictions (Shavelson et al., 1989).

When people are defined as the object of measurement, the person MVC constitutes systematic score variance, and all other MVCs are measurement error variances. As noted by Thompson (2003), “having variance from people being large is a desirable outcome if our premise in a given study is that people differ as individuals and that it is exactly these differences that we wish to quantify or study” (p. 54). In the one-facet design, the measurement error in the generalizability coefficient is the pi, e MVC. The generalizability coefficient is the systematic score variance divided by systematic score variance plus measurement error, as illustrated in Table 1.

Proposed Method for Processing Data With Low Coefficient Alpha

Data with low coefficient alpha should first be evaluated for missing data and outliers. Missing data can occur when a respondent does not provide an answer for one or more items on the scale. Missing data can be treated through a variety of techniques including listwise deletion, pairwise deletion, mean imputation, regression imputation, and maximum likelihood estimation (cf. Edwards & Finch, 2018). However, research suggests that the processing of missing data can have a deleterious impact on coefficient alpha (cf. Enders, 2004; Van Ginkel et al., 2007). For example, Enders (2004) found that the traditional method of listwise deletion yielded biased reliability estimates as compared with methods that incorporated maximum likelihood estimation techniques. Enders suggested that in addition to reporting CIs with reliability coefficients as suggested by Fan and Thompson (2001), missing data and how it was dealt with should also be reported. Edwards and Finch (2018) recommended that the researcher identify the type of missing data (i.e., completely random, random, or not random) before determining how the missing data will be handled.

Outliers can be a source of measurement error that can be detected and addressed by visibly inspecting a plot or histogram for normality or with Tukey’s method, z scores, or median absolute deviation (cf. Gill, 2017). For example, a value that is more than three standard deviations (SDs) from the mean may be regarded as an outlier (Gill, 2017). After missing data and outliers have been ruled out as the cause of low reliability, we suggest evaluating the items and/or the sample as informed by a G theory analysis. If the MVC percentage for i is greater than the MVC for p, evaluate items before evaluating the sample.

Evaluating the Sample

Imagine that the MVC percentages from a G theory analysis yielded a large MVC for pi, e and a small MVC for i. Such findings may suggest that the sample could be the source of the problem. To evaluate the sample, we suggest classifying the sample by total absolute difference (TAD). Coined by Bernardi (1994) as “unbundling the sample” (p. 772), ordering persons by TAD is a process that may help identify people who are contributing to measurement error. Bernardi (1994) stated that “subjects with the highest TAD cause the greatest incremental reduction in reliability” (pp. 772–773). As such, eliminating persons with high TADs may consequently increase coefficient alpha. The TAD is the sum of the absolute difference between all the items for each subject. For example, assuming a scale with three items (X1, X2, X3), equation 5 provides the formula for computing TAD:

TAD = ∑ ( | ( X 1 − X 2 ) | + | ( X 1 − X 3 ) | + | ( X 2 − X 3 ) | ) .

(6)

Once a subset of the data has been identified that yields an acceptable level of coefficient alpha, we recommend bootstrapping the reliability estimates across re-samples of the selected data. This process enables the researcher to determine how stable the reliability estimates are across re-samples. More specifically, the SD of the bootstrapped reliability estimates is a measure of how good the reliability estimate is across samples of people.

Illustrative Example

To illustrate our proposed method, we used a subset of the LibQUAL+^TM data set from Thompson (2004, pp. 163–167). LibQUAL+^TM is a 41-item instrument designed to study the perceptions of service quality at academic libraries (cf. Cook & Thompson, 2001; Thompson et al., 2001, 2002). The example data set contains a random sample of 100 graduate students’ and 100 faculty members’ responses to 12 items. Four items (PER1–PER4) relate to service affect (SA), four items (PER5–PER8) relate to library as a place (LP), and four items (PER9–PER12) relate to information access (IA). The data set is located in Appendix of Thompson (2004).

Our example focuses on the four items (PER9–PER12) that are related to IA. We chose those items as Nimon and Reio (2011) observed issues with the coefficient value for faculty members (i.e., α I A = . 68 ) compared with graduate students (i.e., α I A = . 83 ). Across the four items, there are no missing data or outliers. Because the alpha for the total sample was greater than .70 (i.e., α I A = . 77 ), we recoded responses to PER9 to create unreliable data for illustrative purposes, which will be discussed further when the results are presented.

We developed code in R to implement our proposed method. We chose R as our underlying platform as R is a “cutting-edge, free, open source R Development Core Team (2019) statistical package” that runs on all commonly used operating systems (see R Development) and can be downloaded at https://cran.r-project.org. We developed functions to conduct the G theory analysis (gtheory.r), compute TAD (alphaTAD.r), compute alpha over a range of TAD values (alphaRunning.r), and bootstrap alpha over a range of TAD values (alphaRunningBoot.r). These functions can be downloaded at http://profnimon.com/alpha.zip. The use of these functions that follow the illustrative example is presented in Appendix along with the syntax that creates a plot of the running alpha, similar to Bernardi (1994, Figure 2, p. 773).

To replicate the illustrative example, users will need to install the boot (Canty & Ripley, 2019), car (Fox et al., 2018), lme4 (Bates et al., 2019), psych (Revelle, 2019), and yhat (Nimon et al., 2013) packages as well as download the custom functions to the appropriate working directory and modify the syntax in Appendix to point to their working directory.

Results

Information access scale item responses that had been previously manipulated yielded a coefficient alpha of −.061 [95% CI: −0.28, 0.14]. Examining alpha if item deleted revealed that alpha would be .70 if PER9 was deleted. As seen in Table 2, the effect associated with the person x item interaction that is confounded with unspecified measurement error accounted for the majority of the variance in the total mean variance (67.56) and the generalizability coefficient yielded a value less than .01.

OPEN IN VIEWER

Table 2.

Mean Variance and Generalizability Coefficient Variance Components for Unreliable Data Set.

MVCs					Generalizability coefficient
Effect	Score variance component	f	MVC	MVC percentage	Effect	MVC
I	1.64	4	0.41	32.44
p	<0.01	1	<0.01	<0.01	Systematic	<0.01
pi, e	3.41	4	0.85	67.56	Measurement error	0.85
Total			1.26		Total	0.85

Note. p = person; i = item; pi = Person × Item; e = error; f = effects of four items per person; Mean variance component = score variance component divided by f; Component percentage = MVC divided by sum of MVCs; MVCs = mean variance components. Generalizability coefficient = Systematic/Total < .01.

After returning PER9 back to its original values, coefficient alpha and the generalizability coefficient increased to .77 [95% CI: 0.72, 0.81] and person produced the majority of the variance (75.48%) in the total MVC (see Table 3). As Thompson (2003) noted,

Having variance from people is a desirable outcome if our premise in a given study is that people differ as individuals, and that is exactly these differences that we wish to quantify or study. When we make such an assumption, we are defining people as our “object of measurement.” This assumption, in turn, means that we are defining all the variance due to the main effect of people as systematic or true variance, all other variances as measurement error variances. (p. 54)

OPEN IN VIEWER

Table 3.

Mean Variance and Generalizability Coefficient Variance Components for Corrected Data Set.

MVCs					Generalizability coefficient
Effect	Score variance component	f	MVC	MVC percentage	Effect	MVC
i	0.44	4	0.04	1.75
p	1.55	1	1.55	75.48	Systematic	1.55
pi, e	1.87	4	0.47	22.77	Measurement Error	0.47
Total			2.05		Total	2.01

Note. p = person; i = item; pi = Person × Item; e = error; f = effects of four items per person; Mean variance component = score variance component divided by f. Component percentage = MVC divided by sum of MVCs; MVCs = mean variance components. Generalizability coefficient = Systematic/Total = .77.

Although all the reliability coefficients exceeded .70, we unbundled the sample to examine the robustness of the data and to determine if there were differences by role (i.e., graduate student vs. faculty) in keeping with Bernardi (1994). As seen in Table 4 and Figure 1, reliability increases to .80 [95% CI: 0.7, 0.84], the benchmark offered by Henson (2001) and others, when the 10 responses to the IA scale that yielded TAD values greater than 18 were eliminated.

OPEN IN VIEWER

Table 4.

Alpha and Mean Bootstrapped Alpha by Total Absolute Difference for Illustrative Example.

Sample results							Bootstrap results across 1,000 samples
α	CIlower	CIupper	TAD	N	n Grd	n Fac	M	SE	CIlower	CIupper
1.00	1.00	1.00	0	6	5	1
0.97	0.96	0.98	4	50	32	18	0.97	0.015	0.92	0.99
0.88	0.84	0.91	9	118	65	53	0.87	0.041	0.77	0.93
0.84	0.80	0.87	13	163	83	80	0.83	0.032	0.76	0.88
0.80	0.76	0.84	18	190	95	95	0.79	0.032	0.72	0.85
0.77	0.72	0.81	22	200	100	100	0.76	0.034	0.69	0.82

Note. TAD = total absolute difference; Grd = graduate students; Fac = faculty; CI = 95% confidence interval.

OPEN IN VIEWER

Figure 1.

Total absolute difference (TAD) by alpha.

One can also see that graduate students appeared to yield higher alphas than faculty members, which was confirmed by calculating coefficient alpha for both groups where alpha for graduate students was .83 [95% CI: 0.78, 0.87] as compared with alpha of .68 [95% CI: 0.59, 0.77] for faculty. This difference in coefficient alpha by role is indicative of a measurement invariance issue (cf. Nimon & Reio, 2011).

Note that the bootstrap results are similar to the sample results. Although the mean bootstrapped coefficient alpha for the individuals who had a TAD value less than or equal to 18 (i.e., .79) did not meet the benchmark of .80 that was exhibited in the sample, the bootstrapped reliability estimate for the reduced sample is more replicable than the bootstrapped reliability estimate for the full sample (i.e., .032 < .034).

Discussion

With the limited information α < .70, the researcher is blind to the source of variance, and accordingly, correcting for a low coefficient alpha is merely trial and error. As we have demonstrated, G theory decomposes score variance and compartmentalizes it by source. With this information, the researcher can make an informed decision on how to proceed with the study rather than playing a guessing game.

After correcting for an item that had been erroneously reverse coded, the MVC for pi, e accounted had the highest error component percentage. To correct for the measurement error due to the interaction between person and item, we facilitated a process to eliminate subjects based on the largest TAD, following Bernardi (1994). Note that there is not a cut-off TAD value. It is up to researcher judgment to determine how many response sets to eliminate to achieve a desired alpha.

While we found the method of using TAD to be effective in understanding the relationship between coefficient alpha and sample subsets, a thorough investigation of the literature suggested that no researcher since Bernardi’s publication, 24 years ago, has approached low reliability using this procedure. Perhaps this is because, until now, statistical software did not provide the syntax to make the analysis user-friendly.

There is a misconception among scholars, that when coefficient alpha is below the acceptable range, item deletion is the only solution. However, deleting items may increase coefficient alpha to the detriment of content validity. As a result, the scale may not measure what it is intended to measure (Tavakol & Dennick, 2011). As well, if items are dropped, it may not be possible to compare the reliability coefficients from a reduced item set to reliability coefficients from other studies, as is done in reliability generalization studies (see Vacha-Haase, 1998), unless of course other researchers have eliminated the same set of items (cf. Helms et al., 2006).

Our proposed method demonstrated how to increase the reliability of data with a large measurement error without eliminating items and thus maintaining the integrity of the scale. One could argue that eliminating a small percentage of subjects is worthwhile if doing so maintains the integrity of the scale. Although deleting subjects could result in less representativeness of the population and generalizability of the study results (Kukull & Ganguli, 2012), such losses must be compared with the loss in content validity when a sample-specific set of items are used to measure a construct. In the end, it is up to researcher to make the appropriate tradeoffs including considering publication bias that may result from the file drawer problem if the study is not published because of issues with data reliability.

We would be remiss if we did not discuss the importance of transparency in reporting data. Scholars have a responsibility to maintain the rigor in the HRD literature. For test replication, items, or subjects that are eliminated, must be explained, and the coefficient alpha and effect size before and after the elimination should be reported along with support for the elimination decisions.