Latent Structure and Item Functioning of Self-Referent Encoding Task Word Stimuli in Preadolescent Youth

Person Fit

Person-fit statistics reflect the extent to which models fit individuals and can detect outliers and aberrant response behavior, such as misunderstanding items or inattentive responding. Visual inspection of person-fit plots and review of person-fit infit and outfit statistics showed that the percentage of children with infit and outfit z-scores falling outside the range of −1.96 and +1.96 was less than 1% for the model of the positive item set and less than 2% for the model of the negative item set. Thus, the vast majority of individual response patterns fit the models well, and we retained all participants in subsequent analyses.

Item fit

While overall fit statistics are useful summary statistics, reviewing fit statistics for individual items is important for pinpointing specific areas of misfit. We reviewed Pearson’s χ² (S-χ²) statistics (Orlando & Thissen, 2000) as well as infit and outfit statistics. All items showed good fit in 2-PL models, with the exception of the positive item friendly and negative item ashamed, for which fit statistics could not be computed (Table 3).

OPEN IN VIEWER

Table 3

Parameter Estimates for 2-PL Models.

Positive item set						Negative item set
Item	S-χ2 (df)	p	RMSEA	b	a	Item	S-χ2 (df)	p	RMSEA	b	a
Benchmark for good fit	Non-significance	>.05	.00–.05	–	–	Benchmark for good fit	Non-significance	>.05	.00–.05	–	–
fun	1.85 (1)	.174	.04	−2.60	2.81	ashamed a	–	–	–	2.38	26.40
terrific	7.02 (5)	.219	.03	−1.39	2.20	ugly	2.98 (4)	.561	.00	3.26	1.37
exciting	4.52 (5)	.478	.00	−1.85	2.05	lonely	4.81 (3)	.186	.04	2.57	2.12
proud	5.51 (6)	.481	.00	−2.07	2.08	boring	2.89 (3)	.410	.00	3.04	1.63
talented	8.49 (6)	.204	.03	−1.98	2.00	angry	2.47 (3)	.481	.00	2.17	2.38
smart	3.59 (6)	.733	.00	−2.19	2.49	stupid	1.10 (3)	.778	.00	2.87	1.98
strong	6.79 (5)	.237	.03	−1.35	1.78	selfish	3.64 (3)	.304	.02	2.37	2.06
brave	5.99 (5)	.307	.02	−1.67	1.50	lazy	4.89 (2)	.087	.05	.60	1.39
popular	1.94 (3)	.586	.00	−.38	1.51	sad	.30 (4)	.990	.00	2.71	1.41
friendly a	–	–	–	−3.65	1.85	clumsy	1.43 (2)	.490	.00	.20	1.45
clever	1.55 (7)	.980	.00	−2.42	1.27	foolish	4.74 (3)	.192	.03	1.63	1.10
lucky	9.50 (7)	.219	.03	−1.92	1.00

Note. S-χ2 = Pearson’s chi-square; RMSEA = root mean square error of approximation; b = difficulty parameter; a = discrimination parameter.^aItem fit statistics could not be calculated for friendly and ashamed.

*

p < .05. **p < .01. ***p < .001.

We further reviewed infit and outfit statistics for each item. These chi-square statistics measure the degree of fit of individual items compared to the average of other items in a scale, with values under 0.5 indicating underfit and values over 1.5 indicating overfit (Linacre, 2002). Values outside the range of 0.5 to 1.5 are less productive for measurement, although only values above 1.5 are degrading to the model and threaten its validity (Linacre, 2002). Infit down-weights outliers and is more sensitive to unexpected responses on items that are roughly targeted to a person’s position on the latent trait. Outfit is sensitive to outliers, including unexpected responses for a person given their position on the latent trait.

Infit was excellent for the 2-PL model for the positive item set, with all items falling within the ideal range of 0.5 and 1.5 (Figure 1A). No items showed outfit above 1.5, although three items (smart, proud, and fun) fell below 0.5 (Figure 1A). Low outfit values do not threaten the validity of the model, but rather suggest model overfit, such that responses to these items have low variance (e.g., are endorsed by virtually all respondents) and are therefore less useful for measuring children’s positive self-concept. This was unsurprising given that endorsement rates exceeded 93% for these three items.

OPEN IN VIEWER

Figure 1.

Item Infit and Outfit Statistics.

We found a similar pattern of infit and outfit for the 2-PL model for the negative item set, although a greater number of negative items showed low outfit. All but one item (ashamed) fell within the ideal range for infit (Figure 1B). No items showed outfit above 1.5, though six items (stupid, selfish, lonely, boring, ashamed, and angry) fell below 0.5, suggesting that these items had such low variance (i.e., endorsement rates less than 5%) as to be too “predictable” and therefore less useful for measuring children’s negative self-concept (Figure 1B). Together, these findings suggest that the negative item ashamed shows particularly low measurement utility. Considered alongside the poor fit statistics for this item (Table 3), this prompted us to exclude the item ashamed from our subsequent figures, as doing so preserved interpretation of the other negative items.

Item-person fit

We present item-person (or Wright) maps for the models of the positive item set (Figure 2A) and negative item set (Figure 2E). These maps depict how well items cover the range of the latent trait by comparing the distribution of the latent trait against item difficulty (b parameter) using the same scale. Items should ideally span the full range of the latent trait exhibited by the sample. The latent trait distribution for the model of the positive item set was negatively skewed, such that only a small proportion of children possessed low positive self-concepts (Figure 2A). Because the threshold for endorsement of most positive words occurred along the left tail of the person distribution, positive words tended to map the portion of the sample exhibiting particularly low positive self-concept and appeared less well suited to mapping the full range of self-concept present in the sample (Figure 2A).

OPEN IN VIEWER

Figure 2.

Summary Statistics for 2-PL Models of Positive and Negative Items.

The reverse was true for the model of the negative item set. Latent trait distributions for this model were positively skewed, such that only a small proportion of children exhibited very negative self-concepts (Figure 2E). In contrast to the model of the positive item set, the threshold for endorsement of most negative words occurred along the right tail of the person distribution, indicating that negative items best mapped the portion of the sample exhibiting very negative self-concept (Figure 2E).

Scale Characteristic Curves

The scale characteristic curve (SCC) depicts the relationship between theta (i.e., children’s latent positive or negative self-concept) and the number-endorsed score. It shows the extent to which the total number of positive (or negative) items endorsed is a sufficient estimation of children’s positive (or negative) self-concept. SCCs are plotted for the 2-PL model for the positive item set in Figure 2C. The number of items endorsed increased as theta values became less negative before plateauing at theta values over 1.0. This suggests that the number-endorsed score is a sufficient estimation of children’s positive self-concepts when positive self-concepts are in the below-average to average range.

SCCs are plotted for the 2-PL model for the negative item set in Figure 2G. The SCC for this model showed the opposite pattern to the model of the positive item set. The number of items endorsed increased as theta values became more positive before plateauing at theta values below −1.0, suggesting that the number-endorsed score estimates children’s negative self-concepts well when these are in the average to above-average range.

Conditional Reliability

Conditional reliability is the reliability of a scale at different levels of theta. This stands in contrast to classical test theory, which assumes that reliability is constant across the latent trait. Using a cut-off of .75, conditional reliability was best in the −3.0 ≤θ≤−0.5 range for the 2-PL model for the positive item set (Figure 2D). In contrast, conditional reliability was best in the 1.5 ≤θ≤ 3.5 range for the 2-PL model for the negative item set (Figure 2H). Because conditional reliability is mathematically related to the test information and conditional standard error functions, reliability was accordingly poor at higher values of theta in the model of the positive item set and lower values of theta in the model of the negative item set.

Difficulty Parameters

We report the difficulty parameters (b) for positive and negative SRET items in Table 3. Difficulty parameters reflect the position along the latent trait distribution where an individual is equally likely to endorse or not endorse an item. In other words, this value is the threshold at which the probability of endorsing an SRET word flips between probable and improbable. For example, the 2-PL difficulty parameter for the positive word strong (−1.35) indicates that a child scoring lower than 1.35 standard deviations below the mean positive self-concept is unlikely to endorse strong as self-descriptive. Conversely, the 2-PL difficulty parameter for the negative word lazy (.60) indicates that a child scoring lower than .60 standard deviations above the mean negative self-concept is unlikely to endorse lazy. Difficulty parameters can be seen in item characteristic curves (ICCs) as the 50% endorsement rate and in item information curves (IICs) as the location of peak information.

Difficulty parameters for all positive items were negative, such that most children tended to endorse positive items (Figure 3B). This pattern was observed in the extreme for items friendly, fun, and clever; these words went unendorsed only by children with the lowest positive self-concepts. Accordingly, the majority of positive words provided the most information at very low values of theta (i.e., two standard deviations below the mean; Figure 3D). Thus, these words appear well suited to assessing children with average or lower positive self-concepts, but less useful for measurement of children with above-average or higher positive self-concepts. The exception to this pattern was the item popular, for which the inflection point (−.38) and information curve fell closer to theta zero. Compared to other positive items, popular better mapped the average range of positive self-concept—an important range given that this is where most children are likely to fall (Figure 3). However, a majority of children nevertheless endorsed this most “difficult” word (popular).

OPEN IN VIEWER

Figure 3.

Item Characteristic and Information Curves for 2-PL Model of Positive Item Set.

ICCs and IICs for the 2-PL model for the negative item set are shown in Figure 4. In contrast to positive items, most negative items went unendorsed by children, as indicated by very positive b values (Figure 4B). For example, even for children falling two standard deviations above the mean negative self-concept, the probability of endorsing ugly, stupid, or selfish was less than 10%. The item clumsy had the highest probability of endorsement (barring ashamed, which was excluded due to its very high a value [26.40], which obscured meaningful differences between other items when plotted). The item stupid had the highest probability of endorsement at theta values up to 2.0, though ugly had the highest probability for theta values greater than 2.0. In summary, most negative items appear best equipped for assessing children with highly negative self-concepts.

OPEN IN VIEWER

Figure 4.

Item Characteristic and Information Curves for 2-PL Model of Negative Item Set.

Discrimination Parameters

Discrimination parameters (a) reflect how well an item distinguishes between individuals with different levels of the latent trait. They also reflect the magnitude to which items are related to the latent trait. Larger a values indicate better measurement of the latent trait and better ability to differentiate between individuals across the latent trait. Discrimination parameters influence the slopes of ICCs and the amplitudes of IICs. In 2-PL models, where the discrimination parameter is allowed to vary across items, higher a values produce steeper ICCs and taller IICs.

Items varied widely in the extent to which they provided information about children’s positive self-concepts. Notably, when taking into account the ability of items to differentiate between individuals, items lucky and clever showed lower measurement utility (Figure 3D). Specifically, these items showed the smallest IICs as well as gentler ICC slopes compared to other items covering the same ranges of theta, such as fun and smart (Figure 3). Thus, lucky and clever provided the least information about children’s positive self-concepts and were less efficient than other items at differentiating between individuals across the latent trait. In contrast, items fun and smart provided the most information, although at very negative values of theta (i.e., below −2.0; Figure 3D); thus, these items were efficient at differentiating between individuals falling at different positions on the latent trait, but only for children with very low positive self-concepts (Figure 3). Items proud, talented, and exciting showed highly similar ICCs and IICs, suggesting possible redundancy between these items (Figure 3).

Negative items tended to provide the most information at positive values of theta (Figure 4D). Thus, most negative words appeared excellent for differentiating between children with very negative self-concepts (i.e., two standard deviations above the mean), but less useful for assessing children whose negative self-concepts fell within the average range or below. Exceptions to this pattern were clumsy, lazy, and foolish, for which IICs spanned both negative and positive values of theta (Figure 4D), suggesting that these words are better able to differentiate between children across a broader range of negative self-concept severity.

Test Information Curve

The test information curve (TIC) is a summary statistic of how well items of a scale collectively provide statistical information about the latent trait. These plots are enhanced by corresponding conditional standard error curves, which depict the precision with which scores can be estimated across different values of theta. The TIC for the 2-PL model of the positive SRET scale is shown in Figure 2B. This figure indicates that the positive SRET scale performs well when estimating negative scores, but has less precision when estimating theta scores greater than zero. Specifically, the range of theta scores for which the positive scale showed good estimation was −4.0 ≤θ≤ 0.5 (Figure 2B). Thus, the positive scale shows good estimation for positive self-concepts that fall below average, but performs poorly when they are above average.

The TIC for the 2-PL model of the negative SRET scale is shown in Figure 2F. In contrast to the positive scale, the negative scale adequately estimates positive values of theta, but falters when estimating theta scores below zero. Specifically, the range of theta scores for which the negative scale showed good estimation was −0.5 ≤θ≤ 4.0 (Figure 2F). Thus, in contrast to the positive scale, the negative scale shows good estimation for high negative self-concepts and poor estimation for low negative self-concepts.