A Note on the Relation Between the Angle of the Reference Composite and Liu,Li,and Liu’s Method 4 for Domain Scores

Abstract

Keywords

Recently, Liu et al. (2019) compared several methods of calculating overall and domain scores based on the bifactor model. Domain scores are subscores based on sets of items intended to measure the same skill or area of knowledge. Each of the methods involves estimating ability for examinee j on the general factor ${\hat{θ}}_{jG}$ and each residual specific factor ${\hat{θ}}_{js}$ . One then forms a composite overall and for each domain s:

{\hat{θ}}_{j, overall} = w_{1 G} {\hat{θ}}_{jG} + \sum_{s = 1}^{S} w_{1 s} {\hat{θ}}_{js},

(1)

{\hat{θ}}_{j, domain_s} = w_{2 G} {\hat{θ}}_{jG} + w_{2 s} {\hat{θ}}_{js},

(2)

where w_1G, w_1s, w_2G, and w_2s are weights. The methods explored by Liu et al. differed in how they calculated the weights. They found that Method 4 (M4) generally had the lowest RMSE for recovering the abilities and the correlations between abilities. The purpose of this note is to show how M4 is related to an older procedure of approximating the reference composite measured by items each loading on multiple factors.

For orthogonal axes, Reckase (2009, p. 117) defined the direction of greatest slope for an item as

α_{ih} = \cos^{- 1} (\frac{a_{ih}}{\sqrt{\sum_{k = 1}^{m} a_{ik}^{2}}}),

(3)

where α_ih is the angle of steepest slope relative to axis h, and a_ih is the discrimination parameter for item i on θ_h. For the bifactor model, h is either the general factor, G, or the specific factor, s. The left panel of Figure 1 (modeled after Reckase, 2009, Figure 5.3) illustrates this concept with an equi-probability contour plot for a_G = 1.8, a_s = 1.0, d = 0, where d is the intercept. Each line shows the combination of θ_G and θ_s that yields probability = 0.1, 0.3, 0.5, 0.7, or 0.9. The probability increases most rapidly in the direction α perpendicular to the contour lines, shown by the arrow. From Equation 3, for this item $α$ = (29°, 61°). If the items within domain s all measured in the same direction, and if ${\hat{θ}}_{domain_s}$ in Equation 2 were defined as the composite in this direction, then w_2G would be proportional to cos(α_G) and w_2s would be proportional to cos(α_s). Although one could scale the weights in a variety of ways, if the bifactor parameters were scaled such that the estimated variance of the general θ and of each specific θ was one, it would be convenient to use the cosines themselves so that the variance of the composite would also be one.

Figure 1.

Measurement angles.

However, even if the true angle of steepest slope were in the same direction for all items within domain s, the estimated angles would vary slightly due to sampling error. Furthermore, it seems realistic that some items might tilt more in the direction of the general factor and less in the direction of the specific factor than other items within the same domain. Wang (cited in Reckase, 2009, p. 126) discussed a reference composite that approximates the unidimensional scale that would be estimated if a unidimensional model were applied. The reference composite is based on the eigenvector corresponding to the first eigenvalue of a′a. The elements of this eigenvector are the direction cosines for the reference composite. In the present context, a is the matrix of discrimination parameters on the general and specific factors for the items within domain s. The direction of the resulting reference composite approximates the unidimensional solution using only the items in domain s. An example of the direction of a composite is illustrated in the right panel of Figure 1. For the plotted items, a _G = (1.0, 1.2, 1.4, 1.6, 1.8) and a _s = (0.4, 0.8, 0.7, 0.8, 1.1). The first eigenvector is (0.876, 0.483). Relative to the general factor, the reference composite is at an angle of cos⁻¹(0.876) = 0.50 radians (29°). Each of the items lies at a slightly different angle to the axes, and the reference composite is essentially an average angle. Note that this angle is not the direction of maximum information for all examinees, which varies slightly depending on the θ profile. Conceptually, it is an average so that all examinees have the same weights for the composite score. Zhang and Stout (1999) suggested that the direction of best measurement “is the direction in which score Y has the maximum discriminating power over all possible directions, on average over the population” (p. 140). They defined this as the global direction of best measurement. The reference composite defined by the first eigenvector can be considered the best measurement in this global sense.

For Liu et al.’s M4, with notation modified slightly to better match Equation 3,

w_{2 G} = \frac{\sum_{i = m}^{m'} a_{Gi}}{\sum_{i = m}^{m'} a_{Gi} + \sum_{i = m}^{m'} a_{si}},

(4)

w_{2 s} = \frac{\sum_{i = m}^{m'} a_{si}}{\sum_{i = m}^{m'} a_{Gi} + \sum_{i = m}^{m'} a_{si}},

(5)

where m is the first item in domain s and m′ is the last.

Equations 4 and 5 are approximately proportional to the direction cosines for the reference composite. Consider a case where a_G is the same for all items within domain s and a_s is also the same for all items within the domain. The eigenvector could then be computed from Equation 3 without actually computing the eigenvalues: $\cos (α_{G}) = a_{G} / \sqrt{a_{G}^{2} + a_{s}^{2}}$ and $\cos (α_{s}) = a_{s} / \sqrt{a_{G}^{2} + a_{s}^{2}}$ . Similarly, using Equation 4 and 5, where M is the number of items in domain s, $w_{2 G} = M a_{G} / M a_{G} + M a_{s} = a_{G} / a_{G} + a_{s}$ and $w_{2 s} = M a_{s} / (M a_{G} + M a_{s}) = a_{s} / (a_{G} + a_{s})$ . The only difference between the weights computed from the direction cosines and the weights computed from Equations 4 and 5 is the denominator. Because the same denominator is applied to both the general and specific factor weights, this is merely a scaling issue. The M4 weights sum to one so it is easy to see the relative contributions of each factor. The direction cosines (or equivalently the eigenvector) instead yield composite scores with a variance of one because the sum of the squared cosines equals one. In most cases, the scores will be linearly transformed to a reporting scale, so this difference does not matter. If desired, the researcher could normalize the M4 weights by dividing each weight by $\sqrt{w_{2 G}^{2} + w_{2 s}^{2}}$ to make the variance of the composite equal one.

For a concrete example, let a_G = 1.6 and a_s = 0.8 for all items in domain s. Regardless of the number of items in domain s, the first eigenvector = (0.894, 0.447), a 2:1 ratio, corresponding to α_G = 26.6° and α_s = 63.4°. Using Equations 4 and 5, w_2G = 2/3 and w_2s = 1/3, also a 2:1 ratio. The weights are the same aside from the scaling. This equality holds when a_G/a_s is constant over items.

Now consider a case where the angle of steepest slope varies considerably across items, such as a _G = (1.0, 1.2, 1.4, 1.6, 1.8, 1, 1.2, 1.4, 1.6, 1.8) and a _s = (0.4, 1.4, 0.3, 1.7, 0.6, 1.0, 0.8, 0.9, 0.5, 1.2). The first eigenvector of a′a = (0.836, 0.549), a ratio of 1.522. This corresponds to α_G = 33° and α_s = 57°. This angle is quite close to the mean of the angles of steepest slope for the individual items, α = (31°, 59°). Using Equations 4 and 5, w_2G = 0.614 and w_2s = 0.386, ratio = 1.591. The ratios are not identical when the angles vary across items, but they are fairly close. The correlation between the composites would be .9998.

Shifting to the overall composite in Equation 1, for Liu et al.’s M4,

w_{1 G} = \frac{\sum_{i = 1}^{I} a_{Gi}}{\sum_{i = 1}^{I} a_{Gi} + \sum_{s = 1}^{S} \sum_{i = m}^{m'} a_{si}},

(6)

w_{1 s} = \frac{\sum_{i = m}^{m'} a_{si}}{\sum_{i = 1}^{I} a_{Gi} + \sum_{s = 1}^{S} \sum_{i = m}^{m'} a_{si}},

(7)

where I is the total number of items, S is the number of domains, and other terms are as defined for Equations 4 and 5. Similarly, when computing the eigenvalues/eigenvector for a′a, one uses the items from all S domains. The matrix a has S+ 1 columns and I rows. The eigenvector corresponding to the first eigenvalue has S+ 1 values; the first value is the weight for the general factor and the remaining values are the weights for the specific factors. There are S+ 1 axes, and each weight is the cosine of the angle between the reference composite and a given axis.

The supplemental material displays the a-parameters for a test with four specific factors, orthogonal to each other as well as to the general factor (cosines only apply to orthogonal axes). The first eigenvector of a′a = (0.970, 0.115, 0.089, 0.174, 0.087). Rescaling the values to sum to 1 for easier comparison with Method 4, the vector would be (0.676, 0.080, 0.062, 0.121, 0.061). Using Equations 6 and 7, w = (0.653, 0.083, 0.102, 0.111, 0.050). The correlation between the composites would be .9979.

In summary, one reason that Liu et al.’s Method 4 recovered the domain and overall scores well was because it yields results very close to using the reference composite.

Supplemental Material

Supplemental_Material – Supplemental material for A Note on the Relation Between the Angle of the Reference Composite and Liu, Li, and Liu’s Method 4 for Domain Scores

Supplemental material, Supplemental_Material for A Note on the Relation Between the Angle of the Reference Composite and Liu, Li, and Liu’s Method 4 for Domain Scores by Christine E. DeMars in Applied Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Christine E. DeMars .

Supplemental Material

Supplementary material is available for this article online.

References

Liu

(2019). Reporting valid and reliable overall scores and domain scores using bi-factor model. Applied Psychological Measurement, 43, 562–576.

Reckase

M. D.

(2009). Multidimensional item response theory. Springer-Verlag.

Zhang

Stout

(1999). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129–152.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB