Examining differential item functioning trends for English language learners in a reading test: A meta-analytical approach

Abstract

In this study, differential item functioning (DIF) trends were examined for English language learners (ELLs) versus non-ELL students in third and tenth grades on a large-scale reading assessment. To facilitate the analyses, a meta-analytic DIF technique was employed. The results revealed that items requiring knowledge of words and phrases in context favored non-ELLs in grade 3, whereas items requiring evaluation skills favored ELLs in grade 10. However, inconsistent patterns were found across gender and ethnicity. Educational implications are discussed.

Keywords

Differential item functioning English language learners meta-analysis reading test

Reading is a foundational, necessary skill for academic achievement in schools and for success in civic life and careers in modern society. Thus building foundational skill in reading, specifically reading comprehension, is of high importance to all students, including those learning to read English as a second language. Despite its recognized importance, we have a limited understanding about how students’ performance on different types of reading-comprehension questions varies as a function of their language-learning status. In the present study, we extend previous studies on performance patterns on different types of reading-comprehension questions for English-only students (those who speak English as a first and primary language) and English language learners (students who have English as a second language; ELLs) using meta-analytic differential-item-functioning (DIF) analyses.

Reading comprehension assessment, particularly in standardized high-stakes assessments, is challenging because items must be appropriate for readers from diverse racial, ethnic, cultural, and linguistic backgrounds (Abedi, Bailey, Butler, Castellon-Wellington, Leon, & Mirocha, 2005). Previous studies have shown that ELLs typically achieve skills in word reading equivalent to those of English-only students, provided that they have been exposed to systematic and explicit instruction in decoding (Lesaux & Siegel, 2003). Despite this comparable achievement in word reading, however, ELLs usually lag behind non-ELLs in text-level reading skills (e.g., Al Otaiba, Petscher, Pappamihiel, Williams, Dyrlund, & Connor, 2009; Kim, Y. -S, 2012; Lesaux, Geva, Koda, Siegel, & Shanahan, 2008) including reading comprehension.

Even on assessments that are not high stakes, ELLs consistently underperform compared to their English-speaking peers. In the national sample of the 2003 National Assessment of Education Progress (NAEP), fourth-grade ELLs scored 33 points below non-ELLs in reading, and in Florida the gap was 22 points. By 2011 these gaps were 36 points nationally and 32 points within the state of Florida, showing virtually no improvement over time in the underperformance of ELLs. This gap is larger at higher grades, though comparisons must be made cautiously because the twelfth-grade and fourth-grade scales are not linked directly (based on reports created using the NAEP Data Explorer at http://nces.ed.gov/nationsreportcard/naepdata/dataset.aspx; National Center for Education Statistics, 2003, 2011).

Several factors might contribute to this gap in reading comprehension, such as cultural background (e.g., Briere, 1968), background knowledge (e.g., Erickson & Molloy, 1983; Hale, 1988), and native language (e.g., Alderman & Holland, 1981; Farhady, 1982; Oltman, Stricker, & Barrows, 1988; Politzer & McGroarty, 1985; Spurling & Ilyin, 1985; Swinton & Powers, 1980). Most notably, it has been suggested that students’ limited linguistic knowledge in L2 (their second language, i.e., English) is likely to be an important cause of ELLs’ reading-comprehension lag (Kim, Y. -S, 2012; Lesaux et al., 2008; Mancilla-Martinez & Lesaux, 2010; Nakamoto, Lindsey, & Manis, 2008). Previous studies have found that minor changes of wording in tests can affect the test scores of ELLs differently than those of non-ELLs, and authors have argued that unnecessarily complex language on test items should be reduced to improve the performance of ELLs (Abedi & Lord, 2001; Wolf & Leon, 2009).

One way to examine the appropriateness of tests for ELLs is by using differential item functioning (DIF) analysis. DIF is “a difference in item performance between two comparable groups of examinees; that is, groups that are matched with respect to the construct being measured by the test” (Dorans & Holland, 1993, p. 35). DIF is present when the probability of answering an item correctly differs between groups, controlling for ability level. If a score gap between ELLs and non-ELLs results from differential item performance, it could threaten the validity of scores for the ELLs (Holland & Wainer, 1993). In other words, if particular types of questions are differentially difficult for ELLs, different total test scores may result for the two different groups, who truly have the same ability (Scherbaum & Goldstein, 2008). It is difficult to assess ELLs’ true reading ability from their reading test performance if their scores are based on items that have DIF.

These previous studies examined students’ performance differences on each item, using an item-based DIF approach. Although useful and informative, item-based DIF approaches are limited in that they do not allow us to examine the possible DIF patterns or trends across items that differ in their content or other characteristics. One way to investigate DIF trends for bundles or subsets of test items is the meta-analytic approach. The goal of meta-analytic summaries is to explore variation in effect sizes – here the DIF indices – that may be related to content differences or other features. If DIF values within the same test (e.g., across different types of questions) differ by more than just sampling error, variation in the DIF values can potentially be explained by the use of a moderator that characterizes item or sample features. In meta-analysis, moderators are important characteristics that may explain variation in effects, specifically the DIF values. Meta-analytic DIF summaries model variation in the Mantel-Haenszel (MH) DIF estimates and the reciprocal of the variance of each MH DIF value is used for weighting. Further technical details are found in Appendix A.

Furthermore, within item sets moderators such as student characteristics may account for some of the variation in DIF values (Koo, 2012). Again, these moderators are not usually examined using typical item-based DIF analyses. We cannot assume that DIF patterns or trends are consistent across all important demographic groups (e.g., gender or ethnicity). Gender and ethnicity are usually factors that are directly tested as sources of DIF rather than as moderators. However, in this study, gender and ethnicity were used as moderators to explain variation in DIF values for ELLs versus non-ELLs. Even though no ELL DIF is detected on a test, it can be useful to examine whether DIF magnitudes are consistent across gender or ethnic background. For instance, ELL DIF might be detected only for females, but not for males, or for Asians, but not for Hispanics.

In the present work, we extend previous studies by examining DIF patterns in different types of reading-comprehension questions for children in different grades, and thus in different developmental phases. Even though previous studies have shown that some reading-comprehension questions classified by item content or features favored ELLs or non-ELLs (Kim & Jang, 2009), to our knowledge no studies have been conducted to identify ELL DIF patterns on reading tests for different age groups. Patterns of DIF might vary as a function of children’s developmental level of reading skills. Young children (e.g., third graders) are still rapidly developing in their language and cognitive skills, which are critical to reading comprehension, whereas older children’s (graders’) language (L1 or L2) and cognitive skills are more advanced. Thus, the extent to which DIF exists in various reading-comprehension questions, and the extent to which moderators play a role in the size of DIF, might vary across grades. In summary, we had two primary goals in the present study: (a) to use meta-analytic techniques to examine DIF patterns related to ELL status on a reading-comprehension test across different types of reading-comprehension questions; and (b) to examine variation in DIF across various types of reading-comprehension questions using the student characteristics of gender and ethnicity as moderators. To address these goals, we used data from a large-scale, high-stakes state reading-comprehension test in grades 3 and 10.

Methods

Data sources

Reading test data from the 2009 Florida Comprehensive Achievement Test (FCAT) for third and tenth graders were examined in this synthesis of Mantel-Haenszel (MH) DIF indices. Non-ELLs served as the reference group and ELLs as the focal group. In the present study ELLs were students who had received ESOL (English for Speakers of Other Language) services in the immediately preceding two years, as noted in the state dataset. Because data on English proficiency were no available, we therefore used receipt of ESOL services as a proxy for proficiency. A total of 173,737 third-grade students were included in the present analyses and 5288 (3%) were considered to be ELLs. In tenth grade, 160,391 students were included in the present analyses and 4293 (2.6%) were considered to be ELLs. Table 1 includes sample sizes for students grouped by gender, ethnicity (White, Black, Hispanic, and Asian), and ELL status. Multi-racial and American-Indian student groups were excluded because of their small sample sizes.

Table 1.

Sample size by subgroups for grade 3 and grade 10.

Grade		Total	Male	Female	Hispanic	Asian	Black	White
3	Non-ELLs	168,449	82,571	85,878	44,285	4501	41,256	78,407
3	ELLs	5288	2775	2513	3760	323	822	383
10	Non-ELLs	157,898	75,641	82,257	37,830	4371	35,773	79,924
10	ELLs	4293	2216	2077	2704	248	1136	205

Item classification

All items in the reading-comprehension assessment are classified into four types of reading-comprehension questions, grouped according to related benchmarks from the Sunshine State Standards. The benchmarks for FCAT are developed by a Reading Content Advisory Committee “composed of 15–20 Reading and/or Language Arts professionals from schools, school districts, and universities” (Florida Department of Education, 2004, p. 15). The Florida State Board of Education reviews the Sunshine State Standards and they are also reviewed by Florida educators, and representatives of Florida College System institutions who have expertise in reading. Items concerning “words and phrases in context” (phrases-in-context hereafter) require students “to use strategies to increase vocabulary through structural clues (prefixes, suffixes, roots), word relationships (antonyms, synonyms), and words with multiple meanings” (Florida Department of Education, 2009, p. 23). “Main idea” items require students “to determine the stated or implied main idea, to recognize an organizational pattern, and to identify the author’s purpose and point of view to construct meaning from text” (Florida Department of Education, 2009, p. 23). “Comparisons and cause/effect” (cause–effect) items require students to recognize cause-and-effect relationships. “Reference and research” (evaluation hereafter) items require students to locate and interpret information for various purposes, to use multiple reference materials, and “to check validity and accuracy of research information” (Florida Department of Education, 2009, p. 23). Appendix B presents excerpts of items as examples of the four item types. Table 2 shows the number of items for each of the question types. Table 3 shows the mean scores (average numbers of correct responses) for the four reading-comprehension question types for each grade by ELL status.

Table 2.

The number of items by reading-comprehension question type for third and tenth graders.

Grade	Phrases-in-context	Main idea	Cause–effect	Evaluation
3	9	22	9	3
10	6	17	12	6

Table 3.

Mean raw scores for four reading-comprehension question types for non-ELLs and ELLs.

	Grade 3				Grade 10
	Phases in context	Main idea	Cause–effect	Evaluation	Phases in context	Main idea	Cause–effect	Evaluation
Non-ELLs	6.61	16.72	6.67	1.98	3.82	11.58	8.23	3.96
ELLs	4.03	10.7	4.56	1.27	2.13	6.67	4.55	2.31

Calculation of effect sizes

For this study, the MH DIF index was used to detect DIF because the MH procedure is one of the most commonly used methodologies to detect DIF (Holland & Thayer, 1988). It serves as the standard among various testing companies such as the Educational Testing Service (ETS). The MH odds ratio, ${\overset{\land}{α}}_{M H}$ , can be transformed into log units; we used the $\ln ({\overset{\land}{α}}_{M H})$ value as our effect size (ES). Negative effect sizes indicate better performance for the reference group, which is non-ELLs in these analyses, controlling for ability level. In contrast positive values show superior item performance by ELLs, conditioned on ability.

A matching criterion is required to analyze DIF. Ideally examinees should be matched in terms of their true reading abilities. However, no perfect reading ability measure exists in reality; thus proxies are used in practice. In this study, the total reading test score across the four item categories was used as a proxy for the true reading ability. The four categories may not measure exactly the same reading ability, but they should share a common underlying dimension, and this is interpreted as overall reading ability. Below we discuss confirmatory factor analyses that investigate whether a unitary structure is reasonable for this test.

The moderators gender and ethnicity were examined to identify possible differences in DIF magnitudes. Ethnicity has four categories – Hispanic, Black, Asian, and White. An ELL DIF effect size was calculated for each of eight different subgroups: Male-Hispanic, Male-Asian, Male-Black, Male-White, Female-Hispanic, Female-Asian, Female-Black, and Female-White. For this study, after calculating ELL DIF effect sizes for each separate subgroup, we investigated whether the effect sizes varied across subgroups. Thus the total number of effect sizes for each type of reading-comprehension question was eight times the total number of items for each question type.

Statistical analysis

Koo (2012) proposed the idea of applying meta-analysis techniques with DIF indices, arguing that each $\ln ({\overset{\land}{α}}_{M H i})$ can be used as an effect size. Mean $\ln (α_{M H})$ values for the series of m items can be calculated using weighted procedures given by Hedges and Olkin (1985). Under the fixed-effects model for meta-analysis, all $\ln ({\overset{\land}{α}}_{M H i})$ values are assumed to estimate a common true effect. In other words, the fixed-effects model assumes that DIF is the same across all items and populations studied. Alternately, under the random-effects (RE) model we conceive of a collection of true DIF values, and our goal is to estimate the mean of the distribution of these population effects. Under the RE approach we account for variance from two sources: the sampling error of each $\ln ({\overset{\land}{α}}_{M H i})$ value, and variation in the true effects across items (Hedges & Vevea, 1998).

Because preliminary analyses suggested that all effect sizes did not measure a single common effect, random-effect models or mixed-effects models were used. To examine differences owing to moderators, weighted mean DIF values were computed for each of the eight different subgroups. Those weighted DIF values were compared to determine whether mean DIF systematically differed between males and females, or across ethnic backgrounds of examinees using meta-analytic analysis of variance (ANOVA; e.g., Borenstein, Hedges, Higgins, & Rothstein, 2009). Meta-analytic ANOVA allows one to examine differences owing to categorical factors such as gender and ethnicity.

In the meta-analytic approach, chi-square (Q) statistics are used to examine whether the moderators explain significant variability in the DIF effect sizes. An overall chi-square test Q examines whether all DIF effects arise from one population, which also provides a test of the fixed-effects model. The overall Q statistic has an asymptotic chi-square distribution with m − 1 degrees of freedom, where m is the number of ESs. Two additional types of Q statistics were also used: the Q statistic for the model (Q_M) and the Q statistic for error (Q_E). Q_M evaluates the variation in DIF indices explained by the moderators (Borenstein et al., 2009). If Q_M is large, it suggests that the moderators explain some or all of the between-items and between-groups variation. Q_E indicates whether the model is correctly specified; that is, whether the moderator explains all the variation in DIF that is not a result of sampling error. If Q_E is small and Q_M is large, the moderators are considered to explain all or nearly all between-items and between-groups differences in DIF. In contrast, if Q_E is big, then the moderators do not explain all DIF variation. In this case, a mixed-effects model that accounts for the residual unexplained variance is needed.

Results

Preliminary analysis: Factor structure of reading-comprehension questions in grades 3 and 10

A confirmatory factor analysis (CFA) was conducted to assess the model fit of one-factor and four-factor structures for the four categories of FCAT reading questions. Several fit indices were used to assess the factor structures: the root-mean-squared error of approximation (RMSEA), comparative fit index (CFI), normed fit index (NFI), and chi-square. The four-factor model was specified a priori, based on the item groupings found in the Sunshine State Standards (i.e., phrases-in-context, main idea, cause–effect, and evaluation). Thus, we used CFA to test the applicability of the four subtest model (four-factor model) to a unitary (one-factor) model.

Table 4 shows results of confirmatory analyses of one-factor and four-factor structures for the reading questions. In grade 3, RMSEA values for the one-factor and four-factor models were <.01 and .02, respectively. RMSEA values of less than .06 reflect a good fit (Hu & Bentler, 1999). Also, the CFIs of .92 and .93 for the one-factor and four-factor models met the criterion (.90 or larger) for acceptable model fit (Hu & Bentler, 1999). In addition, the NFI values of .92 and .93 for the one-factor and four-factor models met the criterion (.90 or larger) for acceptable model fit (Hu & Bentler, 1999). Thus both models are plausible and fit reasonably well for grade 3.

Table 4.

Summary of tests of model fit of four reading question types for grade 3 and grade 10.

		χ²(df)	RMSEA	CFI	NFI
Grade 3	One-factor model	115802.80** (946)	.00	.92	.92
Grade 3	Four-factor model	88839.85** (854)	.02	.93	.93
Grade 10	One-factor model	37815.80** (779)	.00	.96	.95
Grade 10	Four-factor model	37287.34** (773)	.02	.96	.96
Criterion		Non-significant	<.06	.90	.90

Note: **p < .001. df = degree of freedom. RMSEA = Root mean square error of approximation. CFI = comparative fit index. NFI = normed fit index.

To further evaluate model fit of the four-factor model, we examined the chi-square fit test values for grade 3. The chi-square values for the one-factor and four-factor models were 90,120.3 with p < .01 and 88,839.8 with p < .01, respectively. Both were statistically significant, indicating that these models did not show acceptable model fit. However, a chi-square difference test shows that the four-factor model for grade 3 fit significantly better than the one-factor model (χ² difference = 1280.4 with df = 6).

Results were similar for grade 10. Even though the chi-square values for the one-factor and four-factor models (37,815.8 with p < .01 and 37,287.3 with p < .01) were statistically significant, RMSEA (.02), CFI (.96) and NFI (.96) values for both models were identical and met the criteria for good fit. However, the four-factor model was again significantly better than the one-factor model according to the chi-square difference test (χ² difference = 528.46 with df = 6). These results confirmed that both the unitary model and the four-factor model were reasonable for grades 3 and 10. Thus it is not problematic to use the overall test scores as the matching variables for the DIF analyses. However, the different tests show that the four reading categories for grade 3 and grade 10 are acceptable as factors, and they show statistically better fit than the one-factor model for FCAT reading questions in both grades. This suggests that distinctions between item categories in their degrees of DIF may also exist.

Research Question 1: Analysis of overall DIF values for four types of reading-comprehension questions for grades 3 and 10

Table 5 presents the weighted mean DIF values and standard errors under the random-effects model for the four reading question types for grades 3 and 10 and Figure 1 shows the associated confidence intervals of the mean DIF values by question type and grade. In grade 3, two item types showed significant mean DIF values – phrases-in-context and main-idea items. In contrast, the means for cause–effect and evaluation items in grade 3 were essentially zero, indicating that the probability to answer correctly did not differ between grade-three non-ELLs and ELLs of the same ability level. The weighted mean DIF value for phrases-in-context items for grade 3 was −0.19. Its confidence interval did not include zero, suggesting that non-ELLs tended to perform better than ELLs on phrases-in-context items, after controlling for reading ability. The confidence interval for DIF on main-idea items with a mean of .046 also did not include zero. However, although the main-idea DIF effect was significant, the fact that the mean and interval endpoints were all very close to zero suggests a minimal DIF effect.

Table 5.

Random-effects overall DIF estimates and confidence intervals by grade and item type.

Grade	Reading-comprehension question types	k	Weighted DIF mean (standard error)	Confidence interval
				Low	High
3	Phrases-in-context	72	−0.190 (.03)	−0.253	−0.128*
	Main idea	176	0.046 (.01)	0.020	0.073*
	Cause–effect	72	0.006 (.03)	−0.046	0.058
	Evaluation	24	−0.031 (.06)	−0.149	0.086
10	Phrases-in-context	48	−0.017 (.05)	−0.121	0.087
	Main idea	136	−0.043 (.03)	−0.092	0.006
	Cause–effect	96	−0.035 (.04)	−0.117	0.046
	Evaluation	48	0.096 (.04)	0.063	0.132*

Note: *Significant. k = number of effects.

Figure 1.

The random-model confidence intervals for the weighted ELL DIF mean for four types of reading question in grades 3 and 10.

In grade 10, the confidence intervals of mean DIF for all types of reading questions included zero except that for DIF of evaluation items. The positive mean of .096 and positive confidence interval indicate that ELLs in grade 10 tended to perform slightly better on evaluation questions than non-ELLs after controlling for reading ability.

Even though the magnitudes of DIF were not large in our samples, the patterns of DIF may contribute to different total test-score averages for ELLs and non-ELLs if many small differences accumulate systematically (Scherbaum & Goldstein, 2008).

Research Question 2: Categorical analyses using moderators

Meta-analytic ANOVA was conducted to further investigate differences among effect sizes according to two moderators, gender and ethnicity. All statistics including weighted mean DIF values, standard errors, and Q statistics for grade 3 and grade 10 are shown in Tables 6 and 7, respectively.

Table 6.

Categorical analyses across reading-comprehension question types for grade 3.

Question type	Categories	k	Weighted mean DIF (SE)	Q _E
Phrases-in-context	Gender, Q_M (1) = 2.18
	Male	36	−0.16 (.043)^b	197.21*
	Female	36	−0.22 (.044)^b	255.68*
	Ethnicity, Q_M (3) = 14.21*
	Asian	18	−0.17 (.051)^c	16.35
	Black	18	−0.17 (.060)^b	110.96*
	Hispanic	18	−0.14 (.055)^b	257.29*
	White	18	−0.29 (.065)^b	56.56*
Main idea	Gender, Q_M (1) = 0.72
	Male	88	0.04 (.020)^b	168.18*
	Female	88	0.05 (.020)^b	205.80*
	Ethnicity, Q_M (3) = 4.55
	Asian	44	0.03 (.039)^b	68.25*
	Black	44	0.06 (.028)^b	110.58*
	Hispanic	44	0.03 (.023)^b	126.36*
	White	44	0.07 (.032)^b	64.96*
Cause–effect	Gender, Q_M (1) = 0.41
	Male	36	−0.01 (.041)^b	126.59*
	Female	36	0.02 (.040)^b	132.51*
	Ethnicity, Q_M (3) = 4.24
	Asian	18	0.01 (.071)^b	32.32*
	Black	18	−0.04 (.054)^b	77.52*
	Hispanic	18	0.01 (.049)^b	119.81*
	White	18	0.06 (.060)^c	25.62
Evaluation	Gender, Q_M (1) = 0.19
	Male	12	−0.01 (.086)^b	65.68*
	Female	12	−0.05 (.087)^b	99.99*
	Ethnicity, Q_M (3) = 3.36
	Asian	6	0.05 (.090)^c	7.74
	Black	6	−0.10 (.119)^b	50.03*
	Hispanic	6	−0.01 (.111)^b	98.92*
	White	6	−0.02 (.065)^c	5.81

Note: *p < .05. k = number of effects. ^bMixed-effects model. ^cFixed-effects model.

The fixed-effects model was used for Q_E and Q_M.

Table 7.

Categorical analyses across reading-comprehension question types for grade 10.

Question type	Categories	k	Weighted mean DIF (SE)	Q _E
Phrases-in-context	Gender, Q_M (1) = 0.87
	Male	24	−0.01 (.077)^b	203.73*
	Female	24	−0.03 (.078)^b	233.34*
	Ethnicity, Q_M (3) = 3.34
	Asian	12	−0.10 (.120)^b	31.17*
	Black	12	0.00 (.104)^b	138.72*
	Hispanic	12	0.00 (.101)^b	219.93*
	White	12	0.02 (.117)^b	44.78*
Main idea	Gender, Q_M (1) = 4.4*
	Male	68	−0.06 (.035)^b	368.13*
	Female	68	−0.02 (.035)^b	448.35*
	Ethnicity, Q_M (3) = 11.66*
	Asian	34	−0.04 (.059)^b	80.66*
	Black	34	−0.02 (.045)^b	395.34*
	Hispanic	34	−0.01 (.044)^b	282.20*
	White	34	−0.13 (.055)^b	51.02*
Cause–effect	Gender, Q_M (1) = 0.001
	Male	48	−0.02 (.059)^b	490.06*
	Female	48	−0.05 (.059)^b	509.95*
	Ethnicity, Q_M (3) = 1.27
	Asian	24	0.01 (.091)^b	52.72*
	Black	24	−0.07 (.079)^b	460.04*
	Hispanic	24	−0.02 (.078)^b	338.13*
	White	24	−0.06 (.087)^b	147.86*
Evaluation	Gender, Q_M (1) = 0.77
	Male	24	0.05 (.046)^b	101.56*
	Female	24	0.07 (.046)^b	103.26*
	Ethnicity, Q_M (3) = 30.86*
	Asian	12	0.11 (.068)^c	9.67
	Black	12	−0.08 (.058)^b	108.12*
	Hispanic	12	0.14 (.054)^b	41.78*
	White	12	0.12 (.060)^c	15.15

Note: *p < .05. k = number of effects. ^bMixed-effects model. ^cFixed-effects model.

The fixed-effects model was used to compute Q_E and Q_M.

Initial tests of gender differences in DIF for all reading-comprehension question types were not statistically significant in both grades, except for main-idea items in grade 10. Thus gender did not explain differences among DIF values except for grade10 main-idea items (Q_M(1) = 4.4, p < .05). Mean main-idea DIF values for males and females in grade 10 were −0.06 and −0.02, respectively. However, within the two gender groups in grade 10, DIF values varied significantly across items (Q_E values were 368.13 for boys and 448.35 for girls, df = 68). Mixed-effects confidence intervals show that once between-items variation is incorporated into the uncertainty of the analyses, the gender differences are no longer significant (mixed-effects Q_M (1) = 0.71, p = .39). Figure 2 shows these mixed-effects confidence intervals for the mean DIF of main-idea items in grade 10.

Figure 2.

Confidence intervals for the mean ELL DIF of “main idea” items by gender in grade 10.

Four ethnic groups – Asian, Black, Hispanic, and White students– were also examined for differences in their mean DIF values. The test of between-groups differences was statistically significant in grade 3 only for phrases-in-context. The weighted mean DIF values for phrases-in-context items differed by ethnicity under the fixed-effects model (Q_M(3) = 14.21, p < .05), indicating that ethnicity explained some variability in the DIF indices. However, after considered between-item variation, the ethnicity differences in grade 3 were no longer significant (mixed-effects Q_M (3) = 1.76, p = .62).

Figure 3 shows the confidence intervals for the mean DIF of phrases-in-context items in grade 3. None of the confidence intervals included 0 and all were negative. Specifically, the confidence interval for DIF of phrases-in-context items for Whites was much lower compared to those for the other ethnicities, indicating that after controlling for reading ability White ELLs tended to perform much worse than White non-ELLs on phrases-in-context items. The lower bound of the confidence interval for phrases-in-context was −0.43, which is in the moderate DIF range (i.e., category B) according to the ETS criteria for DIF. Zwick and Ercikan (1989) recommended that category B items be replaced with equivalent items.

Figure 3.

Confidence intervals for the mean ELL DIF of “phrases-in-context” items by ethnicity in grade 3.

The results for the ethnicity effects in grade 10 differed from those in grade 3. Ethnicity explained some differences in DIF for main-idea items for tenth graders with Q_M(3) = 11.66, p < .05. After incorporating uncertainty owing to between-item variation, ethnicity differences on main-idea items were not significant (mixed-effects Q_M(3) = 3.42, p = .33). However, as was true for other analyses, significant within-ethnicity variation caused the mixed-model standard errors in Table 7 to be relatively large, so the significant ethnicity effect found under fixed effects should be considered tentative, but worthy of further study. The mixed-effects confidence intervals for the DIF values of main-idea items are presented in Figure 4. The confidence intervals of main-idea-item DIF means for Asians, Blacks, and Hispanics all covered zero, indicating that the performance of Asian, Black, and Hispanic ELLs was similar to that of Asian, Black, and Hispanic non-ELLs on main-idea items after controlling for reading ability. However, the confidence interval of the mean DIF for main-idea items for Whites fell largely in the negative range, indicating White non-ELLs tended to perform better than White ELLs on main-idea items, after controlling for reading ability.

Figure 4.

Confidence intervals for the mean ELL DIF of “main idea” items by ethnicity in grade 10.

Ethnicity explained some differences in DIF for evaluation items for tenth graders under the fixed-effects model (Q_M(3) = 30.86, p < .05) and the mixed-effects models (mixed-effects Q_M(3) = 8.76, p < .05). The confidence intervals for the DIF values of evaluation items are presented in Figure 5. The confidence intervals of evaluation-item DIF values for Asians, Hispanics, and Whites were mostly positive, indicating that ELLs in these three groups tended to perform better than non-ELLs on evaluation items after controlling for reading ability in grade 10. For Hispanics the mean DIF value was positive and also significant, indicating that Hispanic ELLs were significantly more likely than Hispanic non-ELLs of the same ability level to answer evaluation items correctly. In contrast, opposite results were seen for Blacks. The confidence interval of the mean DIF value for evaluation items for Blacks fell largely in the negative range, indicating Black non-ELLs tended to perform somewhat better than Black ELLs on evaluation items, after controlling for reading ability.

Figure 5.

Confidence intervals for the mean ELL DIF of “evaluation” items in grade 10 by ethnicity.

Discussion

In the present study, we examined the extent to which ELL and non-ELL students’ performance on different types of questions in a high-stakes reading-comprehension test varied for students who are in different developmental phases – third and tenth graders. In particular, we investigated whether DIF varied as a function of students’ gender and ethnic backgrounds. First, we found using confirmatory factor analysis that four types of reading-comprehension questions – phrases-in-context, main idea, cause–effect, and evaluation – existed in the FCAT reading comprehension test for third and tenth graders.

Overall, ELL third graders’ performance was lower than that of non-ELLs on the phrases-in-context items, after controlling for ability level. This result is somewhat similar to Kim and Jang’s (2009) finding that ELLs had poorer performance on items involving vocabulary knowledge. Thus, even after ELL students’ overall reading scores were taken into consideration, their performance on words and phrase usage was weaker than that of non-ELL students. The phrases-in-context items included vocabulary knowledge such as morphology, word relations, and polysemy. By definition, the ELLs in the present study were receiving ESOL services as a result of limited English proficiency, for which vocabulary is an important component. Therefore, it is not surprising that ELLs performed poorly on vocabulary-related items. If the goal of reading-comprehension assessment at lower grade levels is largely to assess knowledge of vocabulary and understanding expressions in text, given its critical role in reading and language development (this is true for non-ELLs as well; e.g., Anderson & Freebody, 1983; Stahl, 2003), then the inclusion of vocabulary and expression items (i.e., the words and phrases in context items) is justified. In contrast, if the goal of the reading-comprehension assessment is to measure higher-level inferential and evaluative skills with reading materials, the numbers of such vocabulary and expression items should be limited.

Our finding of significant DIF for phrases-in-context items held for only third graders, and while the effect was strongest on average for White students, all ethnic groups showed negative DIF values. Perhaps the lower grade level ELLs may not have developed sufficient strategies to learn vocabulary compared to the older ELLs. This might lead ELLs at lower grade levels to have difficulties with phrases-in-context items.

In contrast, ELL tenth graders’ performance was significantly better than that of non-ELL students on the evaluation items, after controlling for ability level. A similar result was also found in Kim and Jang’s (2009) study where ELLs tended to perform better than non-ELLs on evaluation items, after controlling for ability level. Although further investigation of the question of why tenth-grade ELLs should perform better on evaluation items than non-ELLs is beyond the scope of the present study, we suspect it may be related to ELLs’ prior educational experiences in their native languages. In other words, most tenth-grade ELLs would have had several years of formal education in their native language. Depending on the nature and quality of the education they received in their native languages, these students’ evaluative skills, which involve identifying and interpreting information from various sources, might have been fairly advanced. Although these students are designated as ELLs as a result of their lack of English language proficiency, they may have been able to transfer key skills needed for evaluation items from their native language to English.

An important way in which the present study extends previous studies is by examining the moderators of gender and ethnic background. We found no gender effects in either grade after accounting for between-items uncertainty in DIF magnitudes. When students’ ethnic backgrounds were considered, all third-grade ELLs tended to perform more poorly than non-ELLs on phrases-in-context items, after controlling for ability level. This held for all ethnic groups but was strongest for White students. The confidence interval of the mean phrases-in-context DIF value for Whites was, in fact, close to the ‘moderate DIF’ category according to ETS criteria. Furthermore, White ELLs in grade 10 tended to perform worse than White non-ELLs on main-idea items, after controlling for ability level. In contrast, tenth-grade ELLs of Asian, Hispanic, and White origin tended to outperform non-ELLs on evaluation items after controlling for ability level. These results suggest that in order to achieve fairness and validity of the test item, writers and test developers should be aware that phrases-in-context and main-idea items are sensitive to DIF for White ELLs at both grade levels. Because most of the White students are Indo-Europeans, these results are likely owing to their native language and ethnic backgrounds. Ethnicity is also a proxy, to some degree, for the native language and cultural/educational differences of ELL students, and thus may tie in to possible item specifics that could lead to DIF in reading tests (e.g., Asian languages have completely different structures than Indo-European languages).

Overall the results of the present study suggest that educators need to provide additional sustained help on phrases-in-context items to ELLs at lower grade levels. Although a need in this area for ELLs might be apparent in the beginning stages of language and reading acquisition, studies have indicated that even after ELLs are no longer designated as “limited English proficiency” learners, they still need continued support to succeed in academic school subjects (Francis, Rivera, Lesaux, & Kieffer, 2006). In terms of results for differential performance by gender, studies in reading have consistently shown that male students lag in reading skills and show disproportionately higher levels of reading disability. Recent studies have shown that this is not likely to reflect differential identification and labeling, but may reflect true performance differences (Badian, 1999; Wagner & Schatschneider, 2009; but see Siegel & Smythe, 2005). In our study, we found that gender differences were not significant after controlling for ability level in both grades, except for main-idea items in grade 10.

Future studies and conclusion

Although the findings of the present study are informative, it has several limitations. First, the results are limited to the reading-comprehension assessment used in the present study and to the racial and gender composition of the student population in Florida. Thus, further investigation with different samples from other places would be necessary in order to generalize our findings. Second, more research is needed to explain the mechanisms behind the moderating factors. Specifically, it is not clear why Asian, Hispanic, and White ELLs outperformed non-ELLs. Understanding the mechanism is important for pinpointing educational implications. However, the findings related to ethnicity should be interpreted with caution because we did not have information about the language and cultural backgrounds of the students from different ethnic backgrounds. It would be helpful to examine the backgrounds of those who were included in each ethnicity category. For instance, Black ELLs might be from the Caribbean or Africa, and thus could have linguistic and cultural experiences that are quite different. Examining ELLs’ linguistic and cultural backgrounds could help us to better understand differential performances on different types of items.

Footnotes

Appendix A: Calculation of Mantel-Haenszel DIF index and effect sizes in the meta-analysis

{\overset{\land}{α}}_{M H} = \frac{\sum A_{j} D_{j} / n_{j}}{\sum B_{j} C_{j} / n_{j}}

where j is the index for level of the total test score and A_j, B_j, C_j, and D_j are the counts given in Table A1. In Table A1, A_j and C_j are the numbers of examinees at score level j who find the correct answer, and B_j and D_j are the numbers of examinees at score level j who do not find the correct answer. Also, m_C_j is the sum of A_j and C_j and m_Ij is the sum of B_j and D_j at score level j. Last, n_Fj, n_Rj and n_j are the total numbers of examinees in the focal group, reference group, and total group at score level j, respectively.

\bar{\ln ({\overset{\land}{α}}_{M H})} = \frac{\sum_{i = 1}^{m} w_{i} \ln ({\overset{\land}{α}}_{M H i})}{\sum_{i = 1}^{m} w_{i}}

where w_i is the reciprocal of the variance of $\ln ({\overset{\land}{α}}_{M H i})$ (Koo, 2012) given below. The weight for each effect size represents the precision of, and relative contribution of, each item to the overall effect size.

{var}_{M H} [\ln ({\overset{\land}{α}}_{M H})] = \frac{\sum n_{j}^{- 2} (A_{j} D_{j} + {\hat{α}}_{M H} B_{j} C_{j}) [A_{j} + D_{j} + \hat{α_{M H}} (B_{j} + C_{j})]}{2 {(Σ A_{j} D_{j} / n_{j})}^{2}}

{var}_{M H} [\bar{\ln ({\overset{\land}{α}}_{M H})}] = \frac{1}{{\sum w}_{i}}

and the standard error under the fixed-effects model is then simply

S E_{M H} [\bar{\ln ({\overset{\land}{α}}_{M H})}] = \sqrt{\frac{1}{{\sum w}_{i}}}

L L_{M H} = \bar{\ln ({\overset{\land}{α}}_{M H})} - 1.96 \times S E_{M H}

U L_{M H} = \bar{\ln ({\overset{\land}{α}}_{M H})} + 1.96 \times S E_{M H}

w_{i}^{*} = \frac{1}{V_{i} + \overset{\land}{σ_{θ}^{2}}}

where V_i is the within-item sampling variance for item i given above and $\overset{\land}{σ_{θ}^{2}}$ is an estimate of the between-items variance.

\overset{\land}{σ_{θ}^{2}} = {[\sum (\ln ({\overset{\land}{α}}_{M H i}) - \bar{\ln ({\overset{\land}{α}}_{M H})})}^{2} / (m - 1)] - \bar{V},

where $\bar{\ln ({\overset{\land}{α}}_{M H})}$ is the fixed-effects average weighted mean DIF, and $\bar{V}$ is the average within-item sampling variance across the m items.

Appendix B: Example items from the 2006 FCAT released reading test for grade 10

Excerpts of items are presented here as examples of the four item types. Text passages that were associated with the items are not presented in some cases.

Read these lines from the poem “Woman with Flower.”

I wouldn’t coax the plant if I were you. Such watchful nurturing may do it harm. Let the soil rest from so much digging

What is the meaning of the word nurturing as it is used in these lines?

References

Abedi

Bailey

A. L.

Butler

Castellon-Wellington

Leon

Mirocha

(2005). The validity of administering large-scale content assessments to English language learners: An investigation from three perspectives. CSE Report 663. Retrieved from http://cse.ucla.edu/products/reports/r663.pdf#page=87.

Abedi

Lord

(2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234.

Alderman

D. L.

Holland

P. W.

(1981). Item performance across native language groups on the Test of English as a Foreign Language (TOEFL Research Report No. 9). Princeton, NJ: Educational Testing Service.

Al Otaiba

S. A.

Petscher

Pappamihiehl

N. E.

Williams

R. S.

Dyrlund

A. K.

Connor

(2009). Modeling oral reading fluency development in Latino students: A longitudinal study across second and third grade. Journal of Educational Psychology, 101, 315–329.

Anderson

R. C.

Freebody

(1983). Reading comprehension and the assessment and acquisition of word knowledge. In Huston

(ed.), Advances in reading research (pp. 231–256). Greenwich, CT: JAI Press.

Badian

N. A.

(1999). Reading disability defined as a discrepancy between listening and reading comprehension: A longitudinal study of stability, gender differences, and prevalence. Journal of Learning Disabilities, 32, 138–148.

Borenstein

Hedges

Higgins

Rothstein

(2009). Introduction to meta-analysis. West Sussex, UK: Wiley.

Briere

E. J.

(1968). A psycholinguistic study of phonological interference. The Hague: Mouton.

Carnine

D. W.

Silbert

Kame’enui

E. J.

Tarver

(2010). Direct instruction reading (5th ed.). Upper Saddle River, NJ: Prentice Hall.

10.

Dorans

N. J.

Holland

P. W.

(1993). DIF detection and description: Mantel-Haenszel and standardization. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 35–66). Hillsdale, NJ: Lawrence Erlbaum.

11.

Erickson

Molloy

(1983). ESP test development for engineering students. In Oller

J. W.

(Ed.), Issues in language testing research (pp. 280–288). Rowley, MA: Newbury House.

12.

Farhady

(1982). Measures of language proficiency from the learner’s perspective. TESOL Quarterly, 16(1), 43–59.

13.

Florida Department of Education. (2004). Assessment & accountability briefing book. Tallahassee, FL: Florida Department of Education. Retrieved from http://fcat.fldoe.org/pdf/fcataabb.pdf.

14.

Florida Department of Education. (2009). Understanding FCAT reports. Tallahassee, FL: Florida Department of Education. Retrieved from http://fcat.fldoe.org/fcatUnderstandReports.asp.

15.

Francis

D. J.

Rivera

Lesaux

Kieffer

(2006). Research-based recommendations for instruction and academic interventions. Practical guidelines for the education of English language learners: Research-based recommendations for instruction and academic interventions. (Under cooperative agreement grant S283B050034 for U.S. Department of Education). Portsmouth, NH: RMC Research Corporation, Center on Instruction. Available online at www.centeroninstruction.org/files/ELL1-Interventions.pdf.

16.

Hale

(1988). Developmental trends in speed of information processing across the lifespan. Unpublished doctoral dissertation, University of Wisconsin-Milwaukee.

17.

Hedges

L. V.

(1992). Meta-analysis. Journal of Educational Statistics, 17, 279–296.

18.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.

19.

Hedges

L. V.

Vevea

J. L.

(1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504.

20.

Herber

H. L.

(1970). Teaching reading in content areas. Englewood Cliffs, NJ: Prentice-Hall.

21.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Hillsdale, NJ: LEA.

22.

Holland

P. W.

Wainer

(1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.

23.

Howard

(1997). Cutting off the world’s roof. Earth, 6(6), 20–21.

24.

Bentler

P. M.

(1999). Cutoff criteria for fit indices in covariate structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.

25.

Kim

Y. H.

Jang

E. E.

(2009). Differential functioning of reading subskills on the OSSLT for L1 and ELL students: A multidimensionality model-based DBF/DIF approach. Language Learning, 59, 825–865.

26.

Kim

Y.-S.

(2012). The relations among L1 (Spanish) literacy skills, L2 (English) language, L2 text reading fluency, and L2 reading comprehension for Spanish-speaking ELL first grade students. Learning and Individual Differences, 22, 690–700

27.

Koo

(2012). Meta-analysis of the Mantel-Haenszel index for the detection of differential item functioning. Unpublished doctoral dissertation, Florida State University.

28.

Lesaux

N. K.

Geva

Koda

Siegel

L. S.

Shanahan

(2008). In August

Shanahan

(Eds.), Developing reading and writing in second language learners, (pp. 27–59). New York: Center for Applied Linguistics and International Reading Association.

29.

Lesaux

N. K.

Siegel

L. S.

(2003). The development of reading in children who speak English as a second language. Developmental Psychology, 39,1005–1019.

30.

Mancilla-Martinez

Lesaux

N. K.

(2010). Predictors of reading comprehension for struggling readers: The case of Spanish-speaking language minority learners. Journal of Educational Psychology, 102, 701–711.

31.

Nakamoto

Lindsey

K. A.

Manis

F. R.

(2008). A cross-linguistic investigation of English language learners’ reading comprehension in English and Spanish. Scientific Studies of Reading,12, 351–371.

32.

National Center for Education Statistics. (2003). Average scale scores for reading by status as English language learner. Retrieved March 21, 2013, from http://nces.ed.gov/nationsreportcard/naepdata.

33.

National Center for Education Statistics. (2011). Average scale scores for reading by status as English language learner. Retrieved March 21, 2013, from http://nces.ed.gov/nationsreportcard/naepdata.

34.

Oltman

P. K.

Stricker

L. J.

Barrows

(1988). Native language, English proficiency and the structure of the Test of English as a Foreign Language. Princeton, NJ: Educational Testing Service.

35.

Politzer

R. L.

McGroarty

(1985). An exploratory study of learning behaviors and their relationship to gains in linguistic and communicative competence. TESOL Quarterly, 19(1), 103–123.

36.

Robins

Breslow

N. E.

Greenland

(1986). Estimators of the Mantel-Haenszel variance consistent in both sparse data and large-strata limiting models. Biometrics, 42, 311–324.

37.

Scherbaum

C. A.

Goldstein

H. W.

(2008). Examining the relationship between race-based differential item functioning and item difficulty. Educational and Psychological Measurement, 64, 537–553.

38.

Siegel

L. S.

Smythe

I. S.

(2005). Reflections on research on reading disability with special attention to gender issues. Journal of Learning Disabilities, 38, 473–477.

39.

Spurling

Ilying

(1985). The impact of learner variables on language test performance. TESOL Quarterly, 19(2), 283–301.

40.

Stahl

S. A.

(2003). Vocabulary and readability: How knowing word meanings affects comprehension. Topics in Language Disorders, 23, 241–247.

41.

Swinton

Powers

D. E.

(1980). Factor analysis of the Test of English as a Foreign Language for several language groups (TOEFL Research Report No. 6). Princeton, NJ: Educational Testing Service.

42.

Wagner

R. K.

Schatschneider

(2009). Classification of students based on developmental trajectories of reading: Robustness of findings across sample and method. Paper presented at the Scientific Studies of Reading Conference, Boston, MA.

43.

Wolf

M. K.

Leon

(2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14, 139–159.

44.

Zwick

Ercikan

(1989). Analysis of differential item functioning in the NAEP history assessment. Journal of Educational Measurement, 26, 55–66.