On the Benefits of Using Maximal Reliability in Educational and Behavioral Research

Abstract

This note is concerned with the benefits that can result from the use of the maximal reliability and optimal linear combination concepts in educational and psychological research. Within the widely used framework of unidimensional multi-component measuring instruments, it is demonstrated that the linear combination of their components that possesses the highest possible reliability can exhibit a level of consistency considerably exceeding that of their overall sum score that is nearly routinely employed in contemporary empirical research. This optimal linear combination can be particularly useful in circumstances where one or more scale components are associated with relatively large error variances, but their removal from the instrument can lead to a notable loss in validity due to construct underrepresentation. The discussion is illustrated with a numerical example.

Keywords

maximal reliability multi-component measuring instrument optimal linear combination scale reliability unweighted overall sum score unidimensional scale

The phenomena of concern in the educational, behavioral, social, marketing, organizational, and biomedical sciences are exceedingly complex and multi-factorially determined, which necessitates the examination of their multiple aspects, sides, and facets in empirical studies (Crocker & Algina, 2006). As a consequence, multi-component measuring instruments have attracted an enormous amount of attention over the past century by methodologists and substantive scholars in these and cognate disciplines (e.g., Kelley, 1927; McDonald, 1999). Almost all of this interest, especially in empirical studies, has centered on the highly popular overall sum score, as well as on the associated conventional reliability coefficient. In fact, it has been nearly routine practice to work in applied research with the simple, unweighted sum of the components of a given scale, test, subscale, testlet, battery, or in general composite of observed measures (frequently referred to as scale or measuring instrument in the following sections). This marked tendency has developed since the early 20th century and has led in the meantime to a notable disregard in empirical research of a related concept associated with an important optimality property, that of maximal reliability. As a result, for over 80 years now, the notion of maximal reliability and its pertinent linear combination of the instrument components have received less attention than they deserve, while it has been known for a long time that they possess significant benefits for educational and behavioral scientists (e.g., Thomson, 1940; see also Bartholomew, 1996).

This article aims to contribute to redirecting some of the impressive amount of interest in the widely used overall sum score reliability coefficient, toward the optimal linear combination (OLC) with maximal reliability. To this end, we show that the discrepancy between the conventional unweighted scale reliability and maximal reliability coefficients can be considerable already at the population level. This can occur in particular in settings where one or more components of a unidimensional scale possess relatively large residual variances, possibly along with considerable error-free relationships to underlying latent constructs. While these components might be seen as potential candidates for removal in an instrument revision, such a recommendation would be in general misleading because dropping them may lead to serious loss in scale validity due in part to ensuing construct underrepresentation (e.g., Messick, 1995). In such cases, we argue that the use of the optimally weighted linear combination of the scale components in lieu of their overall sum score is worth considering closely. The reason is that this weighted scale produces scores that preserve construct validity and allow comprehensive measurement of the studied latent trait (Crocker & Algina, 2006). We illustrate these beneficial features of maximal reliability and the OLC using a numerical example.

Background, Notation, and Assumptions

In this article, we assume that Y₁, Y₂, . . ., Y_k are k components of a scale under consideration (k≥ 3; the following discussion is also applicable if k = 2, after suitable constraints are added to identify the model in Equation 1 when fitted to data). We consider these variables as fixed beforehand, that is, as the only measures of interest that are not sampled from a potentially large pool of measures to which inferences are subsequently sought. Furthermore, we posit that these k (approximately) continuous components have been administered to a single-class population that is free of clustering effects and is not a mixture of two or more latent classes (subpopulations; e.g., Geiser, 2013; Rabe-Hesketh & Skrondal, 2022).

The remainder of this note is based on the widely used single-factor model assumed to hold for the aforementioned observed variables Y₁, . . ., Y_k. That is, we presume in the remainder the validity of the following empirically testable, common factor model (e.g., Mulaik, 2009),

\underline{Y} = \underline{μ} + \underline{b} η + \underline{ε},

(1)

where Y = (Y₁, . . ., Y_k)′ designates the k× 1 vector of the scale components, $\underline{μ}$ is the k× 1 vector of their means, b = (b₁, . . ., b_k)′ is the associated k× 1 vector of positive factor loadings, and the latent trait η is posited with mean 0 and unit variance. (In this article, underscoring is used to denote vector and priming transposition.) In addition, in Equation 1, $\underline{ε}$ = (ε₁, . . ., ε_k)′ represents the k× 1 vector of corresponding unique factors or residual (error) terms, which are assumed to possess positive variances v, . . ., v_k, respectively, in addition to being uncorrelated among themselves and with the common factor η. We note that the variance of these manifest variable sum, Y₁+ . . . +Y_k, is then positive as well.

Within this unidimensional scale context, we will next consider (a) the overall sum score reliability for the measures Y₁, . . ., Y_k and (b) their linear combination giving rise to the highest possible reliability. Our following discussion will evolve at the population level and will be specialized to sample-based parameter estimation in the illustration section.

Unweighted Scale Reliability and Maximal Reliability Coefficients

Traditional Sum Score Reliability

Over the past, more than a century since the earliest publications on factor analysis (e.g., Spearman, 1904), the simple overall sum score associated with a multi-component measuring instrument has attracted the lion share of the interest in reliability of behavioral and social measurement (e.g., McDonald, 1999). Accordingly, nearly all publications in empirical behavioral and social research were concerned with the reliability coefficient of the unweighted scale score

Y = Y_{1} + Y_{2} + . . . + Y_{k},

(2)

which is at times also referred to as composite or alternatively as unit-weighted, simple, traditional, or overall sum score. The reliability coefficient of Y, denoted as ρ_Y, is readily found to be expressible as follows in terms of the parameters of the underlying model (Equation 1)

ρ_{Y} = {(b_{1} + . . . + b_{k})}^{2} / [{(b_{1} + . . . + b_{k})}^{2} + v_{1} + . . . + v_{k}]

(3)

(e.g., Bollen, 1980) and is often referred to as scale reliability or composite reliability. We note that under the earlier made assumptions for the model (Equation 1) of relevance for this article, the reliability coefficient ρ_Y is well-defined by Equation 3 and always exists.

The OLC and Maximal Reliability

The concept of OLC responds to the query whether there may be one or more linear combinations with non-unit weights for the measures Y₁, . . ., Y_k, which possess reliability that is higher than the composite reliability coefficient in Equation 3. This question was answered affirmatively several decades ago for the unidimensional setting under consideration in this discussion (e.g., Thomson, 1940; see also Green, 1950). Accordingly, the OLC is that of the infinitely many linear combinations Z of the used measures, defined as

Z = Z (w_{1}, . . ., w_{k}) = w_{1} Y_{1} + . . . + w_{k} Y_{k},

(4)

which possesses the highest possible reliability and thus produces the most consistent (with least relative error variance) individual scores based on the used scale. The weights in Equation 4, w₁, . . ., w_k, are typically unknown and referred to as optimal weights (optimal OLC weights). In terms of the population parameters of the model in Equation 1, these optimal numbers w_j are obtained as

w_{j} = b_{j} / v_{j}

(5)

(e.g., Bartholomew, 1996; j = 1, . . ., k). With the weights in Equation 5, the OLC associated with maximal reliability, symbolized as Z^*, results as (see also [4]):

Z^{*} = (b_{1} / v_{1}) X_{1} + . . . + (b_{k} / v_{k}) X_{k} .

(6)

Several interesting observations from Equations 5 and 6 will soon become particularly relevant and consist of the facts that the optimal weights, w_j, are (a) inversely related to the corresponding error variances, v_j, and (b) directly related to the respective factor loadings, b_j (j = 1, . . ., k). At the same time, relative to the overall sum score Y (see Equation 2), the OLC weights also (c) enhance the importance of the scale components with higher loadings and/or smaller residual variances, as well as (d) diminish the importance of the components with larger residual variances and/or smaller loadings (see also Equation 6). That is, instrument components associated with larger error variances receive smaller weights in the OLC and thus have weaker contributions to Z^* than components with smaller residual variances if their loadings are the same (similar) in magnitude. In addition, components with larger factor loadings receive higher weights in Z^* and thus have stronger contributions to the OLC than those with smaller loadings when their error variances are the same (similar).

Equation 6 also entails that the highest possible reliability achievable with a linear combination of the observed measures Y₁, . . ., Y_k, which is referred to as maximal reliability and denoted as ρ^* below, is

ρ^{*} = ({b_{1}}^{2} / v_{1} + . . . + {b_{k}}^{2} / v_{k}) / [1 + ({b_{1}}^{2} / v_{1} + . . . + {b_{k}}^{2} / v_{k})]

(7)

(e.g., Li et al., 1996; see also Bartholomew, 1996). We stress that this maximal reliability coefficient ρ^* is well-defined by Equation 7 and always exists, like the optimal weights in Equation 5 do, under the usual assumptions that have been also made earlier in the note with respect to the underlying model in Equation 1 (see also the discussion immediately following that equation).

The preceding developments imply that

ρ^{*} \geq ρ_{Y}

(8)

always holds, that is, the maximal reliability coefficient is never inferior to the traditional, simple sum score reliability coefficient. In fact, the equality is the case in Equation 8 if and only if all weights are equal to each other, that is, when and only when

b_{1} / v_{1} = . . . = b_{k} / v_{k}

(9)

is true (e.g., Raykov et al., 2016). This condition is readily testable against empirical data within the framework of latent variable modeling (LVM; B. O. Muthén, 2002), as outlined for instance in the penultimately cited source.

Can Maximal Reliability Be Considerably Higher Than Composite Reliability?

To respond to this question, we first define the difference between the population maximal and simple sum score reliability coefficients as

Δ = Δ (ρ_{Y}, ρ^{*}) = ρ^{*} - ρ_{Y},

(10)

and denote by Ω the parameter space associated with the earlier defined model (Equation 1). The space Ω is (a) a proper, 2k-dimensional subspace of the space of all 2k-tuples of real numbers, usually denoted as R^2k (e.g., Apostol, 2006), and (b) consists of all positive numbers for the error variances and generally all real numbers for the factor loadings of that model.

We then readily notice from Equation 10, as well as Equations 3 and 7, that Δ(ρ_Y, ρ^*) is a continuous function of its 2k arguments, viz. the loadings and residual variances of model (Equation 1). Denoting the 2k× 1 vector of these parameters as π, with Equation 9 in mind, we observe that this function Δ(π) attains its minimum in the subspace Ω^* where the equality chain (Equation 9) holds. Hence, due to the continuity of Δ(π), there exist points in Ω but outside Ω^*, where this reliability discrepancy Δ is positive, that is, maximal reliability exceeds composite reliability.

As our next step, for simplicity of the following argument, we introduce the symbol Γ for the squared sum of factor loadings and Θ for the sum of error variances in the model in Equation 1. With this notation, the scale reliability coefficient in Equation (3) is represented as

ρ_{Y} = Γ / (Γ + Θ) .

(11)

Equation 11 implies that an increase in any error variance leads to a decrease in scale reliability, with all else being constant. Similarly, with the notation

Γ^{*} = {b_{1}}^{2} / v_{1} + . . . + {b_{k}}^{2} / v_{k},

(12)

we represent the maximal reliability coefficient in Equation 7 as

ρ^{*} = Γ^{*} / (1 + Γ^{*}) = 1 / (1 + 1 / Γ^{*}),

(13)

whereby we note that Γ^*>0 holds in the setting of relevance to this article (see Equation 1 and its immediately following discussion). Equation 13 entails that an increase in a factor loading, all else kept constant, leads to an increase in maximal reliability; moreover, a decrease in an error variance, all else fixed the same, leads also to an increase in maximal reliability.

These observations from Equations 11 and 13 as well as the earlier discussion in this section show that (a) an increase in the factor loading for a particular scale component along with (b) a suitable increase of its residual variance, while keeping all other model parameters the same, can lead to an increase in the maximal reliability ρ^* and a decrease in the scale reliability ρ_Y. Hence, these activities (a) and (b) can amplify the population discrepancy Δ between the maximal and scale reliability coefficients. To demonstrate this fact using any given set of 2k values for the factor loadings and error variances, the R-function provided in Appendix A can be used to determine the impact of (a) and (b) on the maximal to scale reliability discrepancy Δ when the model in Equation 1 is correct.

Based on the previously made implications from Equations 11 and 13, as well as the earlier observation of diminished contribution to the OLC by components with larger residual variances, we can now make the following consequential observation. Specifically, unidimensional scales (with uncorrelated errors) that contain one or more components characterized by relatively large residual variances, can be associated with maximal reliability coefficients markedly exceeding the pertinent scale (composite) reliability coefficients. In such cases, while these components might be considered as candidates for removal in a contemplated scale revision, we stress that (a) they need not be entirely unreliable components, as they may also be associated with relatively strong error-free relations to the underlying construct, and perhaps more importantly, (b) their removal from the measuring instrument can entail serious loss in its validity due to ensuing construct underrepresentation (e.g., Messick, 1995; see also the following section).

In the next section, we demonstrate the preceding discussion results using a numerical example.

Illustration on Data

We employ here simulated data to show that maximal reliability can be considerably higher than composite reliability already at the population level for a unidimensional measuring instrument. To this end, we consider an instructive setting with a homogeneous scale and generate later in this section data for its k = 6 components with n = 800 cases according to the following model (see also Equation 1 and their subsequent discussion; see also below):

\begin{matrix} Y_{1} = η + ε_{1}, \\ Y_{2} = 2 η + ε_{2}, \\ Y_{3} = η + ε_{3}, \\ Y_{4} = 2 η + ε_{4}, \\ Y_{5} = 2 η + ε_{5}, and \\ Y_{6} = 2 η + ε_{6}, \end{matrix}

(14)

where η is standard normal while ε₁, . . ., ε₆ are independent normal variates with variances 1, 2, 2, 2, 8, and 30, respectively.

First, we observe that because we know all population parameters of the used model (14), we can determine the population maximal reliability and composite reliability coefficients as well as their discrepancy. To this end, based on Equation 14, we utilize the R-function in Appendix A (using the pertinent call “msrel(1,2,1,2,2,2, 1,2,2,2,8,30)”; see also the discussion immediately after (14), and the Note in Appendix A). In this way, we obtain the following population coefficients:

\begin{matrix} ρ^{*} = . 860, \\ ρ_{Y} = . 690, and \\ Δ = . 170, \end{matrix}

(15)

where ρ_Y is as before the reliability of the sum score Y = Y₁+ . . . + Y₆, ρ^* the pertinent maximal reliability coefficient, and Δ is their difference (see Equations 3, 7, and 10). According to the previous discussion in this note, ρ^* is the reliability of the OLC

Z^{*} = Y_{1} + Y_{2} + . 5 Y_{3} + Y_{4} + . 25 Y_{5} + . 067 Y_{6},

(16)

with its optimal weights resulting from Equation 5 as 1/1 = 1, 2/2 = 1, 1/2 = .5, 2/2 = 1, 2/8 = .25, and 2/30 = .067, respectively. Owing to the markedly differing loadings and especially residual variances across the scale components (see Equation 14), we note that the optimal weights also differ considerably among themselves at the population level, unlike all the unit weights in the simple sum score Y (see Equation 2). At least as importantly, we notice the substantially smaller weight in the OLC (16) of the last component Y₆ than the optimal weights for the remaining four scale components Y₁ through Y₄. Similarly, the weight of the penultimate component Y₅ in the OLC, Z^*, is notably smaller than the weights of Y₁ through Y₄. This diminished importance of Y₅ and especially of Y₆ in the OLC Z^* results, as pointed out previously, from the fact that they have considerably higher residual variances than Y₁ through Y₄ (see Equation 14). These relatively large error variances in Y₅ and Y₆, along with their comparable loadings to those of the remaining components, lead to the markedly deflated optimal weights of Y₅ and Y₆ in the OLC relative to the other four components. As a consequence, the impact of the substantial error terms in Y₅ and especially Y₆ upon the maximal reliability coefficient ρ^* is greatly reduced, unlike the much more adverse impact of their residual terms upon the composite reliability coefficient ρ_Y (see Equation 3).

For these reasons, as seen from Equation 15, the population maximal reliability markedly exceeds the sum score reliability, with their difference being .17 in favor of the former reliability coefficient. In addition, the population composite reliability of .69 is considerably lower than what might be viewed as a desirable minimal reliability level of .80 (cf. Crocker & Algina, 2006), while the maximal reliability at large exceeds notably that level. Furthermore, the individual component population reliabilities are .5, .667, .333, .667, .333, and .120, respectively (see Equation 14 and their following discussion for the error variances). Hence, the first five scale components cannot be seen in practical terms as being associated with notably low reliability in this multi-component measuring instrument context. At the same time though, information about the underlying construct η that is contained in the least reliable component, Y₆, is also taken into account in the OLC. It is worth further mentioning here that this component, Y₆, has one of the highest error-free relationships with the underlying construct, η. However, as indicated before, the impact of the marked residual variance in Y₆ upon the optimally weighted scale Z^* in Equation (16) is minimized by the smallest weight of Y₆ (viz. .067) in the OLC that represents a means for producing optimal scale scores in a given sample of population.

In this connection, while one might consider dropping the last component Y₆ in a revision effort, like pointed out previously, this would not be in general recommendable for at least two reasons. One, as seen from Equation 14 and mentioned earlier, Y₆ has the same-strength relationship of its residual-free part, 2η, with the underlying factor η, as three of the remaining components that like Y₆ possess the highest such relationship (viz. Y₂, Y₄, and Y₅). In addition, the remaining two scale components, Y₁ and Y₃, have a twice-weaker relationship of their residual-free part, η, than Y₆ does with the factor η. This indicates that Y₆ in fact contains relevant information about that latent construct, η, of main interest. Two, and at least as importantly, in general, dropping a scale component may lead to loss in validity due to ensuing construct underrepresentation (e.g., Messick, 1995). This will in particular occur when the component taps into or represents an essential aspect of the studied construct, which will no longer be captured by the revised scale (cf. Crocker & Algina, 2006). For these reasons, in such cases, it could be generally recommended to consider using the OLC Z^* as a preferred scale score because it capitalizes in an optimal way on the individual differences in the trait η under investigation. This feature of the OLC is not matched by the traditional sum score, Y, which assigns the same importance to all components and hence overstates the measurement relevance and contribution of the least-reliable components to that score Y. In summarizing the preceding discussion in this section, the considered population-level setting represents a setup where population maximal reliability markedly exceeds composite reliability, and the scale score resulting from the OLC (Z^*) can be preferred to the traditional overall sum score (Y).

To exemplify these population-based developments, as indicated earlier, we simulate next a data set consisting of n = 800 cases according to the previously defined model in Equation 14. (The Mplus command file provided in Appendix B was employed for this purpose and, if used with the same seed, will replicate all results reported below; see also L. K. Muthén & Muthén, 2023, ch. 12.) When fitting the single-factor model (Equation 1) to the resulting data set, we obtain the following tenable goodness-of-fit indices: chi-square = 13.399, degrees of freedom = 9, p-value = .145, and a root mean square error of approximation = .025, with a 90% confidence interval (CI) being [0, .051]. (The Mplus command file in Appendix C can be used for this model fitting.) The parameter estimates, standard errors, 95% bootstrap CIs, and related statistics furnished thereby are presented in Table 1, with the estimates of the optimal weights, composite reliability, maximal reliability, and their difference displayed correspondingly in its last nine rows.

Table 1.

Parameter Estimates, Standard Deviations, t-Values, and Two-Tailed p-Values for the Fitted Single-Factor Model (Used Software Format).

Parameter	Estimate	SE	t-value	p-value	95% CI
F BY
Y1	1.104	0.048	23.037	.000	[1.005, 1.192]
Y2	2.018	0.076	26.580	.000	[1.870, 2.164]
Y3	1.048	0.056	18.806	.000	[.942, 1.158]
Y4	2.179	0.071	30.608	.000	[2.029, 2.309]
Y5	2.082	0.123	16.918	.000	[1.834, 2.311]
Y6	1.807	0.214	8.437	.000	[1.386, 2.233]
Variances
F	1.000	—	—	—	—
Residual variances
Y1	0.966	0.058	16.608	.000	[.853, 1.087]
Y2	2.067	0.145	14.304	.000	[1.790, 2.366]
Y3	1.984	0.110	18.094	.000	[1.788, 2.225]
Y4	1.852	0.154	12.000	.000	[1.567, 2.159]
Y5	8.282	0.438	18.896	.000	[7.454, 9.197]
Y6	29.377	1.628	18.043	.000	[26.412, 32.772]
New/additional parameters
W1	1.142	0.097	11.813	.000	[.968, 1.347]
W2	0.976	0.089	10.990	.000	[.823, 1.177]
W3	0.528	0.043	12.369	.000	[.452, .618]
W4	1.176	0.118	9.987	.000	[.977, 1.434]
W5	0.251	0.021	12.215	.000	[.211, .292]
W6	0.061	0.008	7.522	.000	[.046, .078]
MAXREL	0.875	0.007	120.319	.000	[.857, .886]
SCALE_R	0.702	0.017	42.289	.000	[.666, .732]
DELTA	0.173	0.013	13.169	.000	[.148, .198]

Note. SE = standard error; BY = factor (F = η in Equation 1) is being measured/indicated by following observed measures; MAXREL = maximal reliability coefficient (see Equation 7); SCALE_R = scale reliability (composite reliability) coefficient (see Equation 3); DELTA = difference between maximal and composite reliability coefficients; W1, . . ., W6 = optimal weights for the linear combination with maximal reliability (see Equation 6); “—” = not applicable (due to variance fixing for model identification). (For additional details on software output, see L. K. Muthén & Muthén, 2023; intercept estimates are not shown due to them being irrelevant, nuisance parameters in the fitted-by-software default model with intercepts, while being simulated as 0 during the data-generation process; see Equation 14).

As seen from Table 1, the maximal reliability coefficient is estimated at .875 (.007) with a 95% CI of [.857, .886] (standard error given within first parentheses, followed by CI; see last column of Table 1). In addition, composite reliability is estimated at .702 (.017) [.666, .732], and the estimate of the maximal-to-scale reliability discrepancy, Δ, is .173 (.013) [.148, .198]. We note the considerably smaller standard error and shorter CI of maximal reliability than those of composite reliability (as well as, by implication, of their difference). This is due to the previously indicated optimal use of the analyzed data in estimating the OLC and hence of its reliability, that is, of the maximal reliability coefficient (see also below). We notice in addition that the 95% CIs of all three parameters cover their corresponding population values of .860, .690, and .170, which were determined earlier and given in Equation 15. Similarly, it is observed that these three CIs are asymmetric relative to their pertinent estimates. The reason is that their corresponding parameters are bounded in the population from below and above by 0 and 1, respectively. Moreover, the use of the bootstrap approach yields in general asymmetric CIs for parameters that are not far from their upper or lower bounds (being 1 for the two reliability coefficients; cf. Efron & Tibshiriani, 1993).

From Table 1, we also observe that the OLC of Y₁ through Y₆, which possesses the highest possible reliability of any linear combination, is estimated as (see the lower part of that table containing the optimal weights w₁ through w₆)

Z^{*} = 1.142 Y_{1} + . 976 Y_{2} + . 528 Y_{3} + 1.176 Y_{4} + . 251 Y_{5} + . 061 Y_{6} .

(17)

Comparing this estimated OLC with its population counterpart in Equation 6, we readily notice that for each weight, its respective 95% CI covers its corresponding population value of 1, 1, .5, 1, .25, and .067 (see the last column of Table 1 and Equation 16). As indicated in our earlier, population-based discussion in this section, the weights of the last two scale components Y₅ and Y₆ are the smallest of all six weights, due to the fact that their error variances are highest of all components (while their loadings on the common factor are the same as those of Y₂ and Y₄).

Summarizing our sample-based estimation discussion here, we exemplified on a particular data set the considerable improvement in reliability and estimation quality that can result from being concerned with maximal reliability and its associated OLC, rather than with simple sum score reliability, especially when scale components differ considerably in their error variances (while they may or may not be associated with similar factor loadings). These benefits of the maximal reliability and OLC concepts are not shared with the traditional and currently widely used simple overall sum score reliability, especially in educational and behavioral research. The gains justify a recommendation of considering closely the use of maximal reliability and its pertinent OLC particularly in empirical studies where a researcher may have an informed expectation, prior knowledge, or earlier study results implying varying individual reliabilities and error variances across the components of used measuring instruments.

Discussion and Conclusion

This note was concerned with the possible benefits associated with the concepts of maximal reliability and OLC of behavioral scale components. Unfortunately, little use of them has been made in empirical educational and psychological research over the past 80 years or so. One important reason for this lack of attention especially by substantive scholars may have been the fact that unlike the simple sum score, the OLC depends on generally unknown weights that typically need to be estimated based on sample data from a multi-component measuring instrument under consideration (see also below). In addition, detailed studies of the difference between maximal and overall sum score reliability on the one hand, and its specific relationship to underlying model parameters on the other hand, seem to have been lacking.

The present article aimed to contribute to closing this gap in reliability research. Its main goal was to show that the population discrepancy between maximal and traditional sum score reliability can be considerable. In addition, we argued that possible areas of more intensive and justifiable interest in maximal reliability and the pertinent OLC may well include settings with considerably differing individual component reliabilities and error variances. As indicated in the preceding discussion, the utility of these two concepts may be particularly enhanced in studies where, for validity reasons, dropping scale components with high error variances and potentially limited reliabilities may not be attractive due to ensuing construct underrepresentation.

The discussion in the note has several limitations that need to be explicated. First, all preceding developments were exclusively concerned with unidimensional scales whose components possess uncorrelated errors. The reason is that an explicit expression of the maximal reliability coefficient in terms of the underlying model parameters is currently known for this case (see Equation 7). While extensions to the multi-dimensional scale case have been obtained in recent decades, the resulting expressions for maximal reliability and the OLC are no longer in closed form and involve eigenvalues of appropriate product matrices as well as associated eigenvectors (e.g., Li, 1997). Second, our discussion proceeded in the case of no clustering effects and a single-class population of interest, that is, a population not consisting of multiple latent classes (subpopulations; e.g., Geiser, 2013). Evaluation of maximal reliability under clustering effects may proceed relatively straightforwardly while accounting for them (e.g., Rabe-Hesketh & Skrondal, 2022). Third, dealing with the concepts of maximal reliability and OLC under substantial unobserved heterogeneity is likely to be more involved, however, and may require class-specific approaches. As a consequence, it is conceivable that the benefits of maximal reliability and the OLC may be more pronounced for some but not all the classes (subpopulations), depending on their specifics and the within-class relationship between maximal reliability and sum score reliability. In addition, this topic needs future and detailed research that is herewith similarly encouraged. Fourth, the empirical application of the maximal reliability and OLC notions requires large samples, as it capitalizes on an application of the LVM methodology, a large-sample statistical theory-based approach (e.g., B. O. Muthén, 2002; see below for another motivation for this requirement). We are therefore hopeful that future studies will also examine the possibility of developing guidelines, potentially based on extensive simulation studies that are beyond the confines of this article, which would help researchers in the process of determining sample sizes when the underlying asymptotic theory obtains practical relevance.

In this connection, it is worthwhile pointing out that an important limitation of the maximal reliability coefficient, and especially of the OLC, consists of the observation that they are both sample-dependent in a complicated way. This is a consequence from the fact that maximal reliability can be seen as the maximal possible R-square coefficient when predicting the underlying latent construct from its indicators using a linear combination of them (e.g., Raykov et al., 2015). In particular, the magnitude of this R-square depends on the sample observations. Given that (a) the R-square index is generically associated with an upward bias to begin with (e.g., Agresti & Finlay, 2009), (b) the maximal reliability coefficient is itself a maximized R-square (across all possible selections of component weights), and (c) the optimal weights in the OLC are not constants but functions of model parameters that are themselves also estimated, one readily realizes that the maximal reliability coefficient and OLC result from a multiple-step optimization procedure that makes at each step an essential use of the sample data. As a consequence, both maximal reliability and its pertinent OLC are numerically tied to the empirical study under consideration, are affected by sampling error in a complex manner, and their own values (or weights) will therefore generally change in another sample or study. This sample dependence of the notions of focal interest in the present note will be less pronounced with increasing sample size, however, due to the latter being inversely related to sampling error (e.g., Agresti & Finlay, 2009). Hence, the higher the sample size, the more the empirical researchers using these two concepts will benefit from them relative to using standard overall sum scores (traditional scale scores in Equation 2). This observation adds further relevance to the aforementioned large sample requirement (stated there in order for the underlying asymptotic theory to obtain practical relevance). Therefore, to limit the effect of sampling error-impacted empirical data on the values of the maximal reliability and OLC weights, educational and behavioral researchers are strongly encouraged to use as large samples as possible.

In this context, it needs to be emphasized that (along with maximal reliability) the OLC concept has been developed, and used in this note, only as a means for obtaining optimal scores for the units of analysis (subjects) in a given study. These scores are sample-specific, as pointed out earlier, and will generally change even in an extended study including earlier used units of analysis (see Raykov, 2006, for a method that can be used to evaluate the sample instability and CIs for the resulting individual optimal scale scores). If a researcher is interested instead in estimation of latent and/or observed variable relationships, rather than scoring studied subjects using corresponding scale scores, his or her concerns will be better handled by employing appropriate latent variable models. These models, if found plausible for a given study (analyzed sample data set), will permit obtaining point and interval estimation of the relevant model parameters for these variable relationships and, in this way, respond to those concerns.

In conclusion, the present article aimed to contribute to raising the interest in maximal reliability and the OLC of behavioral scale components among educational researchers, psychologists, and social scientists. Their demonstrated benefits here, in the context of associated limitations pointed out earlier, can be of particular importance in empirical efforts for optimal scoring and enhancing the psychometric properties of multi-component measuring instruments that have been widely used in these and cognate disciplines for a number of decades.

Footnotes

Appendix A

Appendix B

Appendix C

Acknowledgements

The author thanks G. A. Marcoulides, D. M. Dimitrov, and S. Gabler for valuable discussions on reliability evaluation. The author is grateful to the Editor and two anonymous referees for a number of critical and insightful comments on an earlier version, which contributed significantly to the improvement of the article.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Tenko Raykov

Notes

References

Agresti

Finlay

(2009). Statistical methods for the social sciences. Prentice Hall.

Apostol

(2006). Calculus. Wiley.

Bartholomew

D. J.

(1996). The statistical approach to social measurement. Academic Press.

Bollen

K. A.

(1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390.

Crocker

Algina

(2006). Introduction to classical and modern test theory. Harcourt College Publishers.

Efron

Tibshiriani

R. J.

(1993). An introduction to the bootstrap. Chapman & Hall.

Geiser

(2013). Data analysis with Mplus. The Guilford Press.

Green

B. F.

(1950). A note on the calculation of the weights for maximum battery reliability. Psychometrika, 15, 55–61.

Hancock

G. R.

Nevitt

(1999). Bootstrapping and the identification of exogenous latent variables within structural equation models. Structural Equation Modeling, 6, 394–399.

10.

Kelley

T. L.

(1927). Interpretation of educational measurement. World Book Company.

11.

(1997). A unifying expression for the maximal reliability of a linear composite. Psychometrika, 62(2), 245–249.

12.

Rosenthal

Rubin

D. B.

(1996). Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. Psychological Methods, 1(1), 98–107.

13.

McDonald

R. P.

(1999). Test theory. A unified treatment. Lawrence Erlbaum.

14.

Messick

(1995). Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.

15.

Mulaik

S. A.

(2009). Foundations of factor analysis. CRC Press.

16.

Muthén

B. O.

(2002). Beyond SEM: General latent variable modeling. Behaviormetrika, 29, 87–117.

17.

Muthén

L. K.

Muthén

B. O.

(2023). Mplus user’s guide. Muthén & Muthén.

18.

Rabe-Hesketh

Skrondal

(2022). Multilevel and longitudinal modeling with Stata. Stata Press.

19.

Raykov

(2006). Interval estimation of individual optimal scores from multiple-component measuring instruments. Structural Equation Modeling, 13, 252–261.

20.

Raykov

Gabler

Dimitrov

D. M.

(2016). Maximal reliability and composite reliability: A latent variable modeling approach to their difference evaluation. Structural Equation Modeling, 23, 384–391.

21.

Raykov

Rodenberg

C. N.

Narayanan

(2015). Optimal shortening of psychometric scales. Structural Equation Modeling, 22, 227–235.

22.

Spearman

(1904). General intelligence, objectively determined and measured. American Journal of Psychology, 15, 201–292.

23.

Thomson

G. H.

(1940). Weighting for battery reliability and prediction. British Journal of Mathematical and Statistical Psychology, 30, 357–360.