Opportunity to Learn What Is on the Test and Performance on the Test

Abstract

State accountability systems assume standards based instruction and test content are highly aligned and opportunities to learn the content exist equally for all students. This alignment between content taught and tested is important to understand achievement, yet it is rarely examined. Teachers from Grades 3 to 8 participated along with students without disabilities (n = 116) and students with a disability (n =104) who received all mathematics instruction in their general education. Teachers recorded over 155 days of instructional information for mathematics and administered an interim mathematics test at the end of the year. We found an average of 44% of mathematics content standards were taught and tested, while 22% were not taught but tested. The results indicated students without disabilities did significantly better than students with disabilities on content taught and tested, but not so on content not taught but tested. Limitations and research needed conclude the article.

Keywords

opportunity to learn mathematics content alignment instructional equity

This study examined a fundamental, but rarely tested assumption of standards-based accountability systems: Students, both those with and without disabilities, have an opportunity to learn (OTL) the content standards they are tested on and their achievement is a function of having had such an opportunity. To this end, we had general education teachers in two states document their state’s mathematics content standards they taught; then near the end of the year, we tested all students to determine how well they performed on mathematics items aligned with the state content standards taught. We also examined students’ performances on their state’s content standards tested, but not taught. Students with a disability (SWD), who received all their mathematics instruction in a general education classroom, were the focal group and compared to students without a disability (SWOD). Thus, this study also contributed to our understanding of inclusive and equal instruction and assessment practices.

Test Content, OTL, and their Relation to Student Achievement

It is widely recognized that based on achievement test performances, a significant achievement gap exists between SWDs and SWODs (Stevens, Schulte, Elliott, Nese, & Tindal, 2015). Researchers interested in this achievement gap have examined a number of issues including family income, accommodations, learning time, instructional quality, and OTL (Elliott, Kettler, Beddow, & Kurz, 2018). Research concerning OTL has focused on three dimensions of the enacted curriculum: time, content, and quality (Kurz, 2011). Time is typically divided into allocated, engaged, and actively responded. Content has generally addressed the overlap between the enacted and assessed curriculum (e.g., Comber & Keeves, 1973; Husén, 1967). Quality is often focused on instructional practices and their cognitive demands such as depth of knowledge (DOK).

Research on OTL for SWD has provided mixed findings. For example, Kurz, Elliott, Lemons, Zigmond, and Kloo (2014) found that SWD experienced significantly less time on content standards and less content coverage compared to their overall class. In another study, Elliott, Kurz, Tindal, and Yel (2017) found that SWD and SWOD who received mathematics instruction in the same classrooms had almost equal OTL when OTL was comprehensively defined; yet, significant differences still existed in their mathematics test scores on both interim and summative assessments, with the SWOD group achieving at significantly higher levels than the SWD group.

The present study is a follow-up of the Elliott et al. (2016) investigation with a subset of the teachers and students to take a more in-depth examination of the state content standards taught and content tested relative to the interim mathematics test. Thus, a brief review of previous content-focused OTL research is needed.

Husén developed an item-based OTL measure that required teachers to report their instructional content coverage for each test item. As such, OTL was defined as the match between what is taught and what is tested. Based on his findings, mean correlations between teachers’ content coverage and student achievement in mathematics across 10 countries ranged between .11 and .20 (Kurz, 2011). Husén’s research did not, however, provide any insights specifically regarding SWDs. Comber and Keeves (1973) obtained similar results to Husén with a mean correlation of .12 for their international study of science education. Borg (1979) focused on more immediate teacher recall and controlled for student ability and socioeconomic status in an investigation of OTL. Specifically, he found test-based content coverage accounted for 16% of the variance in student achievement.

The content overlap conceptualization of OTL remained prominent in several other research studies during subsequent years (e.g., Cooley & Leinhardt, 1980; Winfield, 1987, 1993). For a meta-analysis, Scheerens and Bosker (1997) reviewed 19 studies focused on teachers’ content coverage of tested content and reported an average Cohen’s d effect size of .18. While current research on OTL has expanded conceptually well beyond content covered (e.g., Heafner & Fitchett, 2015; Kurz, Elliott, Lemons et al., 2014), studies with a focus on exposure to tested content (e.g., Schmidt et al., 2001; Schmidt, McKnight, Valverde, Houang, & Wiley, 1997) remain relevant because the evidence consistently indicates that greater OTL is related to higher student achievement (Schmidt, Burroughs, Zoido, & Houang, 2015).

SWDs and OTL Content Tested

Educational researchers have been interested in OTL for over a decade, particularly as part of the instructional lives of SWDs (e.g., Elliott, 2015; Elliott et al., 2016; Kurz, Elliott, Kettler, & Yel, 2014; Kurz, Elliott, Wehby, & Smithson, 2010). A number of educational policies have stimulated this interest. Specifically, since the passage of P.L. 105-117 IDEA Amendments in 1997, it has been required that SWDs to be included in all general state and district assessment programs. This inclusive assessment policy was advanced further with the passage of P.L. 107-110 The No Child Left Behind Act in 2001 and extenuated with the passage of P.L. 114-195 ESSA (2015) by its requirement that states establish achievement standards aligned to requirements for college and career readiness (Weigert, 2018). On page 124 of the ESSA bill, in the Challenging Academic Standards and Academic Assessments clause, it specifically states,

Each State, . . . shall provide an assurance that the State has adopted challenging academic content standards and aligned academic achievement standards . . . With respect to academic achievement standards, include the same knowledge, skills, and level of achievement expected of all public school students in the State.

In summary, best practices support and federal law ensures all students have access to grade-level academic content based on grade-level content standards and these content standards are measured by the assessments used to document all students’ academic achievements.

Research data on content-based OTL and the relation with student achievement in the context of special education are limited, although a few researchers have provided some initial evidence for consideration. For example, Kurz et al. (2010) used the Surveys of Enacted Curriculum (SEC) alignment method to examine the relation between OTL (i.e., alignment between the enacted and the intended curriculum was used as an OTL proxy) and student achievement averages for general and special education teachers. The content of instruction delivered by general and special education teachers as measured by the SEC did not indicate significantly different alignment indices between the two groups of students. The correlation between OTL and class averages of student achievement was .64 (p < .05). When general and special education teachers were examined separately, the correlation between alignment and achievement remained significant only for the special education group with .77 (p < .05). These findings, however, are limited due to the study’s small sample size.

A multilevel analysis of the Kurz et al. (2010) data via Hierarchical Linear Modeling (HLM) allowed for variance decomposition of students’ end-of-year achievement using predictors at the student level (i.e., prior achievement) and classroom level (i.e., classroom type, content alignment). The intraclass correlation coefficient (ICC) was $\hat{ρ} = . 34$ (i.e., 34% of variance in students’ end-of-year achievement was between classrooms). The final (main effects) model predicted individual student achievement as a function of overall mean classroom achievement, main effect for classroom type (i.e., general education, special education), main effect for content alignment, prior achievement as a covariate, and random error. All four fixed effects were significant (p < .001), while the random effects were not significant (p > .05). The results of the reanalysis thus supported classroom type and content alignment as significant predictors of individual student achievement even after controlling for prior achievement at the student level. In addition, both classroom type and classroom alignment accounted for virtually all variance in student achievement that was between classrooms.

As noted, the concept of OTL is operationalized as a teacher effect based on instructional variables related to time, content, and quality (Rowan & Correnti, 2009). As such, the concept does not address the interaction and appropriateness between teachers’ OTL provisions and students’ abilities and needs (Kurz, 2011). However, Kurz noted that for SWDs the Individualized Education Program (IEP) delineates the extent to which the general curriculum is part of the student’s intended curriculum and includes a set of specific (intended) educational objectives. In addition, instructional accommodations may further enhance OTL for SWDs. These additional considerations for SWDs (i.e., content of the intended curriculum, instructional accommodations as part of quality-based OTL indices) thus have implications for the respective OTL measurement tools.

Rationale, Research Questions, and Expected Outcomes

As documented, content-based conceptualizations of OTL have focused narrowly on tested content and more broadly on valued content and skills related to particular subject domains. Available data support an empirical association between the content of instruction and student achievement. The quality of the data, however, is limited, which makes it difficult to generalize findings. First, the measures of students’ OTL instructional content vary across studies. As noted by Kurz (2011), researchers have employed two approaches for collecting OTL data on the content of instruction. One approach uses item-based OTL measures, which teachers use to report on the relative content coverage related to each test item (e.g., Husén, 1967; Winfield, 1993). The second approach use taxonomic OTL measures that provide an exhaustive list of subject-specific content topics, which teachers use to report on the relative emphases of enacted content according to different dimensions (e.g., Porter, 2002; Rowan & Correnti, 2009). The quality of achievement measures used across studies, however, is unclear. Specifically, little information is available on the reliability of achievement test scores and the test’s alignment to the intended curriculum. The latter concern is about the extent to which the achievement test in question measured the content that teachers were supposed to teach (i.e., the content prescribed by the standards). That is, alignment between the enacted and intended curriculum cannot be expected to correlate highly with student achievement if the test fails to be aligned with the respective content standards. In addition, the instructional sensitivity of assessments used to detect the influence of OTL on achievement typically remains an unexamined assumption among researchers (D’Agostino, Welsh, & Corson, 2007). Another limitation in the available data on OTL related to the content of instruction is the paucity of research involving SWDs, especially those students receiving all their instruction in general education classrooms.

The present study addressed concerns regarding the measurement of content-focused OTL and paucity of technical information regarding the tests used to document student performance. Specifically, we addressed two questions: (a) What percentage of mathematics content tested was taught during a school year? (b) How well do SWDs and SWODs perform on tests of mathematics content taught and tested versus content not taught but tested?

Based on the research reviewed and our experience measuring OTL in hundreds of classrooms in several states, we expected the alignment between content taught and tested would be good, but less than 75%. Finally, we expected all students to perform better on tests of content they have been taught versus not taught, and SWODs to perform better than SWD on tested content both taught and not yet taught.

Method

Participants

Teachers (N = 63; 47 in AZ, 16 in OR) participated from 18 schools in general education classrooms representing Grades 3 (n = 10), 4 (n =10), 5 (n = 15), 6 (n = 9), 7 (n = 10), and 8 (n = 9). All teachers volunteered and received monetary compensation for time after school and on weekends to be trained on MyiLOGS to a high level of proficiency and reliability.

All students (N = 220; 116 SWOD, 104 SWD) received instruction in the same general education classroom. All 104 students (n = 17 in third, 19 in fourth, 22, in fifth, 13 in sixth, 18 in seventh, 15 in eighth) identified as SWD had an IEP indicating they had either a learning disability (unspecified), behavior disorder, or a speech-language disability. The remaining 116 students (n = 18 in third, 17 in fourth, 29 in fifth, 16 in sixth, 17 in seventh, and 19 in eighth) were selected at random from students in the same classrooms who did not have an IEP and were thus assumed not to have a disability. Generally, two SWDs and two SWODs were selected from each participating teacher’s classroom for detailed instructional reporting by teachers. For purposes of analyses, students in Grades 3, 4, and 5 created our elementary school sample (n = 122) and students in Grades 6, 7, and 8, our middle school sample (n = 98).

Confidentiality concerns expressed by school leaders limited our characterization of disability types by grade or classroom. Inspection of student IEPs was not allowed beyond the identification of disability type; thus, determination of the need for custom content standards to address skill deficits in mathematics was negated.

Measures

MyiLOGS

MyiLOGS (Kurz & Elliott, 2011), an online measure of OTL, was used by teachers to document implementation of intended curricula at the class and student levels. Teachers were expected to daily drag/drop Common Core State Standards (CCSS) for mathematics that were the focus of their lesson onto respective calendar days and indicated the number of instructional minutes dedicated to each standard. MyiLOGS scores have been examined in several studies and found to be reliable (e.g., Kurz, Elliott, Kettler, & Yel, 2014) and moderately correlated (average of 73% agreement) with independent observers (Elliott et al., 2017).

Measure of achievement

We administered easyCBM, online interim assessments Fall, Winter, and Spring during the 2014-2015 school year that provided teachers with brief tests from Grades 1 through 8; these measures were aligned with the CCSS for mathematics in both participating states. Each assessment comprised 48 to 54 multiple-choice items, which typically are completed in 20 to 25 min. Cronbach’s alpha ranged from .78 to .91, indicating moderate to high internal consistency. Similar coefficient alphas were affirmed with this study’s sample. Split-half reliability coefficients ranged from .71 to .89, with a median of .82. Overall, this measure predicts 50% to 65% of the variance in end-of-year mathematics achievement measures.

The easyCBM development team, in coordination with teachers, developed the mathematics items used in this study to be aligned with specific CCSS standards at each grade level. This item content alignment was verified by a second, independent team of three investigators who reviewed the matches between items and CCSS standard. This independent team of reviewers’ consensus resulted in 92% agreement for the original item to CCSS matches. Thus, the test developers’ item alignment with the CCSS was determined to be sound and acceptable for purposes of this study.

Procedures

Each teacher received substantial training in logging their daily instruction online with MyiLOGS for a given mathematics class starting the first week of school and ending 2 weeks after their respective state tests (April in AZ; May in OR). Teachers were cognizant that the reliability of their logging was important given they (a) had to pass a rigorous performance test with the logging software before qualifying to use MyiLOGS and (b) were observed monthly throughout the entire school year by independent observers during an entire mathematics class.

Teachers in AZ and OR reported on their instructional time and content standards coverage an average of 163.8 days and 158 days out of a possible 180-day school year.

Students were requested to complete the easyCBM Mathematics test for their respective grade level online during the week before their state assessment in late April (AZ students) or late May (OR students). The vast majority of these students had taken an easyCBM test two times (Fall and Winter) earlier in the school year, so they were familiar with the process and types of items. All students, however, were still assisted with access to the tests and given opportunities to ask questions about the test directions to ensure they understood the task. They took the test in familiar settings, most of which were in their own classroom on computers they used frequently.

Data Analyses

The primary descriptive statistics were for MyiLOGS instructional content coverage indices of easyCBM items correctly answered on the spring test administration for SWDs and SWODs from each grade. The item responses for each student were then compared to the list of mathematics standards that their teacher had reported teaching for at least 60 min during the previous 150 to 160 class days. The 60-min instructional time criterion was selected a priori to enhance the likelihood that students had a meaningful OTL the expected knowledge or skill taught. As it turned out, very few content standards were taught for fewer than 60 min over the year. From the item to standard comparisons, items were classified as (a) taught and correct, (b) taught but not correct, (c) not taught but correct, and (d) not taught and incorrect. This set of indices provided evidence to address our first prediction regarding the alignment between the percentages of standards taught and tested.

We conducted two sets of inferential analyses. First, we established the reliable and stable differences between the SWD and SWOD groups on achievement across the year via a repeated measure analysis of variance (ANOVA). Then we used ANOVAs to analyze whether the percentage of items correct differed for elementary and middle school SWDs and SWODs on tested content that had been taught and for content not taught.

Results

We report on several types of evidence concerning the extent of mathematical content overlap between content taught and tested, as well as the test performances for groups of SWDs and SWODs. Before examining these featured results on content alignment, we contextualize this work by documenting our students’ mathematics test scores, as measured by easyCBM, across the year at three points in time (Fall, Winter, and Spring). As displayed in Table 1, total mean percent correct scores on easyCBM Fall, Winter, and Spring administrations for SWOD and SWD groups both increase similarly over the course of the year; however, the SWOD group in comparison to SWD group starts the year with significantly more mathematical knowledge. These observations were further supported by a multivariate ANOVA that found the scores between the groups at each time point were statistically significantly different. Specifically, we found a significant main effort for Educational Status (SWOD vs. SWD), F (1,49) = 24.30, p < .000, and for Time (Fall, Winter, Spring), F (1,48) = 27.91, p < .000, but no interaction effort. Thus, the two groups of students differed reliably each of the three times their mathematics achievement was measured with the average effect size (Cohen’s d) difference being .68 and favoring SWOD. Both groups showed positive mathematics growth (Mean Spring test – Mean Fall test / pooled standard deviation) over the course of the year with SWOD showing an effect size of .52 and SWD a larger effect size of .65.

Table 1.

Total Mean Percent Correct Scores on easyCBM Fall, Winter, and Spring Administrations for SWOD and SWD Groups.

Student groups	Fall easyCBMTotal % correct score	Winter easyCBMTotal % correct score	Spring easyCBMTotal % correct score	Fall to SpringChange in % correct
SWODs (n = 116)	58.3 (14.9)	62.3 (13.8)	66.9 (18.0)	8.6
SWDs (n = 104)	47.6 (12.5)	53.1 (13.5)	57.0 (16.1)	9.4
Differences in mean percent scores (SWOD – SWD)	10.7	9.2	9.9

Note. SWOD = students without a disability; SWD = students with a disability.

Evidence Regarding the Mathematics Content Taught and Tested

Table 2 provides detailed data by grade level on the number of content standards teachers reported teaching their entire class for at least 60 min during a school year. Examining the content alignment of the standards taught with the interim test used to provide teachers with data about achievement progress indicated that, on average, 67% of the intended content standards were reported to have been taught. This resulted, however, in 44.1% of the standards being taught and tested, while 22.9% of the standards were taught but not tested. In addition, 21.6% of the intended standards were not taught, but tested and the remaining 11.4% of the intended standards were not taught nor tested. Inspection of the data for Grades 3 to 8 indicated that regardless of grade there are consistent trends for the various conditions. Specifically, (a) more content standards were taught and tested than taught and not tested and (b) more content standards were not taught but tested than standards not taught and not tested.

Table 2.

Overall Alignment Between Mathematics Standards Taught and Testing for Students.

Grades	Number of content standards expected to be taught	Content standard taught and tested (%)	Content standard taught but not tested (%)	Content standard not taught but tested (%)	Content standard not taught and not tested (%)
3	25	52.4	24.8	15.6	7.2
4	28	44.6	30.0	12.5	12.9
5	26	43.1	22.3	22.3	12.3
6	29	45.2	27.6	16.9	10.3
7	24	41.3	15.4	25.4	17.9
8	28	38.1	17.5	36.9	7.5
Average across Grades 3–8	26.7	44.1	22.9	21.6	11.4

Evidence Regarding Students’ Performances on Interim Mathematics Test

In our analysis of students’ test performance, we found substantial differences (i.e., ES ⩾ .58) between the mathematics achievement of SWD and SWOD at the end of the year who receive instruction in dozens of general education classrooms. Table 3 provides details of the performance differences for SWD and SWOD across Grades by Alignment Conditions (Content tested and taught; content tested and not taught). Both groups of students performed slightly, but not significantly, better on test items that covered content taught in class during the present school year. Specifically, SWDs in both grade clusters averaged about a 2% performance correct increase for items aligned with their instruction in comparison to items unaligned with their instruction. The SWODs at the middle school level experienced a similar 2% performance correct increase, while their peers at the elementary level recorded a 7% performance correct increase for aligned versus unaligned item content.

Table 3.

Overall Mean Performances of SWDs and SWODs on Mathematics Test When Content Taught and Not Taught.

Grade and number of teachers	SWD overall correct % on tests		SWOD overall correct % on tests		Average number of minutes of instruction per standards	SWD correct % on test when standard was taught	SWD correct % on test when standard was not taught	SWOD correct % on test when standard was taught	SWOD correct % on test when standard was not taught
Grade and number of teachers	Fall	Spring	Fall	Spring	Average number of minutes of instruction per standards	SWD correct % on test when standard was taught	SWD correct % on test when standard was not taught	SWOD correct % on test when standard was taught	SWOD correct % on test when standard was not taught
3 (10)	53	65	68	81	325.26	63	67	78	83
4 (10)	52	62	72	87	329.25	63	53	86	69
5 (16)	45	51	68	69	365.65	50	45	67	55
6 (9)	46	53	52	62	296.56	54	43	64	70
7 (9)	38	43	42	51	362.75	40	44	49	62
8 (9)	49	56	51	57	309.43	54	59	61	53
Average across Grades 3–8	48	57	58	67	331.48	54.0	51.8	67.5	65.3

Note. SWOD = students without a disability; SWD = students with a disability.

From a statistical perspective, Table 4 documents testing results (percentage of items correct) for our two groups of students characterized by grade cluster (Grades 3-5 vs. Grades 6-8) and disability status (SWD vs. SWOD). The upper half of Table 3 focuses on the desired alignment condition (i.e., content both taught and tested), while the lower half of the table focuses on the undesired alignment condition (i.e., content not taught but tested). A two-way ANOVA for the desired alignment condition indicated significant main effects for both Grade Cluster, F(1,116) = 24.45, p < .0001, and for Disability Status, F(1, 116) = 19.93, p < .0001. For the undesired alignment condition, a second two-way ANOVA yield similar results. Specifically, the main effects for Grade Cluster, F(1, 112) = 10.47, p < .002, and Disability Status, F(1, 112) = 8.88, p < .004, were both statistically significant. Figures 1 and 2 provide a visual illustration of the data in Table 4 and highlight the salient trends noted via the ANOVA results. These figures also highlight the fact that the elementary (Grade Cluster 3-5) test was generally easier for students relative to the middle school (Grade Cluster 6-8) test.

Table 4.

Comparisons of the Mean Performances of on Mathematics Achievement Test for SWDs and SWODs Under When Content Is and Is Not Taught.

Grade cluster	Disability status	n	Mean % correct score	SD
	Content taught and tested
Grades 3–5	SWD	20	62.75	15.75
	SWOD	39	82.21	11.30
Grades 6–8	SWD	40	49.35	18.80
	SWOD	41	61.07	20.27
	Content not taught but tested
Grades 3–5	SWD	16	60.17	22.38
	SWOD	17	75.59	22.40
Grades 6–8	SWD	38	47.92	21.32
	SWOD	39	59.03	21.83

Note. SWOD = students without a disability; SWD = students with a disability.

Figure 1.

Comparison of test performances (mean percentage correct) of SWOD on tests with content taught and not taught.

Figure 2.

Comparison of test performances (mean percentage correct) of SWD on tests with content taught and not taught.

Discussion

This study examined the content alignment between classroom instruction and students’ performances on end-of-year testing, a frequently talked about, but rarely assessed premise of standards-based testing. We operationalized content alignment as overlap in coverage of state content standards in classroom instruction and on an achievement test. In effect, we were interested in teaching to the content standards upon which the test was designed to measure (i.e., “good” teaching to the test). This notion of content coverage has been a central element of OTL for over 50 years, which has become a fundamental aspect of equitable, inclusive, and accessible education for all students (ESSA, 2015; Elliott, Kettler, Beddow, & Kurz, 2018) and test fairness (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014).

Findings and Their Relation to Previous Research

Stimulated by federal assessment policies, such as NCLB and ESSA, and existing OTL research, this study was motivated to advance understanding of the content alignment between mathematics instruction and achievement tests and its relationship to students’ test performance. Content alignment is both a practical instructional matter and a construct representation matter of some importance to the validity of inferences made about resulting test scores. Therefore, a year-long study was conducted with teachers and their students in Grades 3 through 8 to empirically address two questions: (a) What percentage of mathematics content tested was taught during a school year? (b) How well do SWDs and SWODs perform on tests of mathematics content taught and tested versus content not taught but tested?

In the context of federal educational assessment policies relevant for SWDs, and, given previous research on OTL, we expected to find that teachers report teaching a relatively high percentage—60% to 70%—of state content standards, which in turn would also be tested on achievement measures aligned with CCSS. We also expected that both SWDs and SWODs would perform better on test items that aligned, versus not aligned, with their instruction for the past several months. Evidence was found to support these expectations.

State content standards taught

First, we found that our volunteer sample of teachers from two, trained to use MyiLOGS to document their instruction, reported teaching an average of 67% of the CCSS mathematics content standards for at least 60 min of class time during 8 months of instruction prior to an end-of-year achievement test. Some variability from this average was observed across the grades with third-grade teachers covering an average of 77% of the content standards while seventh and eighth grade teachers generally covered 56% of their grade-level content standards.

Of the state content standards taught, about two thirds of them were tested by our achievement measure. Thus, although the item content of the achievement measure had been found to align well with the CCSS, it did not cover the range of content standards, thus the mismatch/misalignment between the content standards taught and tested. Admittedly, creating measures that adequately represent a substantial number of content standards with enough items to be reliable is difficult in the context of time allocated for testing. Thus, it is challenging to attain a high level of content alignment between instruction and the measures used to document achievement. Clearly, more effort is needed to enhance the alignment or overlap between instruction and testing if we are to obtain a clearer and reliable inference about students’ achievement from the instruction they receive. More, not less, teaching to the content standards that tests measure is needed if we want more reliable information about what students have learned. This certainly is part of gaining a better understanding of the notion of achievement gaps between all types of student subgroups besides SWDs. When students start at different places with regard to their knowledge of a subject matter, gaps in achievement are obvious, but the reason for such gaps is not so obvious. Thus the concern about quality instruction and teaching to the content standards that are measured by a major test.

Differences in test performance of SWDs and SWODs

Second, we found, as expected, SWODs performed significantly better than their classmates with disabilities overall at testing Times 1, 2, and 3. Unexpected, however, we found both SWDs and SWODs performed similarly on the achievement test items measuring material they had not been taught in comparison to the material they had been taught. In other words, not directly being taught mathematics content like that measured on our achievement test did not have a significant negative effect on test performance for either SWDs or SWODs.

Limitations and Future Research

Although this study directly examined the influence of content coverage alignment on the performance of SWDs on a mathematics test, it has some limitations that influence its impact and should be addressed in future research. First, like many studies of SWDs in general education classrooms, the size of the sample was relatively small, thus limiting analyses to grade clusters, rather than specific grades, and limiting the generalization of the findings. Second, a related limitation was the lack of OTL information on the instruction of SWDs who received their mathematics instruction outside the general education classroom, presumably on content standards below their grade level assignment. Third, the specific nature of the disability for students with learning disabilities was not recorded due to constraints on demographic and IEP information imposed by the participating schools. It is unknown how many of the students may have had a mathematics disability, and thus required more differentiated instruction that otherwise provided in general education classrooms. Fourth, the assessment used was an interim assessment, rather than the states’ end-of-year achievement test. This was necessary to have a common measure across states and one that easily allowed for the documentation of content alignment. Most state tests are secure and item level scores generally are not available. Thus, the specific item content alignment with standards could not be established and the generalization of our findings to a state mathematics achievement test was thus not possible. Fifth, the OTL measure used in this study did not account for the implementation of instructional accommodations that may have enhanced or hindered student with disabilities’ access to their teachers’ OTL provisions. Sixth, the intended curriculum in this study was narrowly defined based on the CCSS. The extent to which teachers covered IEP objectives—academic, behavioral, or otherwise—remains unknown.

Conclusion

This study focused on OTL content covered, and its influences on performances on an end-of-year achievement test by SWDs and SWODs. Business-as-usual mathematics instruction in over 60 elementary and middle school classrooms in two states for 8 months was recorded by teachers and deemed reliable by periodic independent observers. A widely used interim assessment proven to have items aligned with a CCSS content standard was administered just before state examinations. The results from this basic content alignment study were both confirmatory and surprising.

The confirmatory results were that teachers, on average, teach less than 75% of their state’s intended mathematics content standards and there is a statistically significant achievement performance gap between SWDs and SWODs. Although these student groups start at different place in terms of their content knowledge, they often show very similar rates of achievement over the course of a given year. Previous researchers have documented similar findings (e.g., Kurz et al., 2010; Stevens, Schulte, Elliott, Nese, & Tindal, 2015).

The surprising result concerned students’ test performance on content not taught to them. This is the first time, to our knowledge, that this “not taught but tested” finding has been observed and reported. It clearly deserves replication before interpreting this finding more, but, if substantiated, it may have some interesting implications for the design of achievement tests and practical concerns regarding the need for teachers to teach all the content standards in a given area within a given year. Admittedly, more research needs to be done, but the design utilized in this study where the content of instruction and test items are directly monitored represents a foundation for future researchers to consider when examining OTL, test content, and achievement gaps. Without the OTL the content valued by the fact it is tested is unfair to all learners and in the case of SWDs will only increase achievement gaps, even if it is only a small amount. To maximize the alignment between what is taught and tested, testing leaders are encouraged to provide educators more detailed and accessible testing blueprints documenting the content standards being tested, and educational leaders should continue to support teachers in their efforts to translate those identified content standards into robust and accessible instruction that all students have the OTL.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Coauthors Kurz and Elliott are authors of MyiLOGS, the measure of OTL used in this study. The data from this measure was collected directly online and by having all data analysis conducted by Nedim Yel, rather than the authors of MyiLOGS concerns about potential bias because of a conflict of interest were addressed. No royalty or other financial benefit from the use of this measure was allowed the authors based on a COI agreement with Arizona State University.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.

Borg

W. R.

(1979). Teacher coverage of academic content and pupil achievement. Journal of Educational Psychology, 71, 635–645.

Comber

L. C.

Keeves

J. P.

(1973). Science education in nineteen countries. New York, NY: Halsted Press.

Cooley

W. W.

Leinhardt

(1980). The instructional dimensions study. Educational Evaluation and Policy Analysis, 2(1), 7–25.

D’Agostino

J. V.

Welsh

M. E.

Corson

N. M.

(2007). Instructional sensitivity of a state’s standards-based assessment. Educational Assessment, 12, 1–22.

Elliott

S. N.

(2015). Measuring opportunity to learn and achievement growth: Key research issues with implications for the effective education of all students. Remedial and Special Education, 36, 58–64. doi:10.1177/0741932514551282

Elliott

S. N.

Kettler

R. J.

Beddow

P. A.

Kurz

(2018). Handbook of accessible instruction and testing practices. New York, NY: Springer. doi:10.1007/978-3-319-71126-3_1

Elliott

S. N.

Kurz

Tindal

Yel

(2017). Influence of opportunity to learn indices and education status on students’ mathematics achievement growth. Remedial and Special Education, 38, 145–158. doi:10.1177/0741932516663000

Heafner

T. L.

Fitchett

P. G.

(2015). An opportunity to learn US history: What NAEP data suggest regarding the opportunity gap. High School Journal, 98, 226–249. doi:10.1353/hsj.2015.0006

10.

Husén

(1967). International study of achievement in mathematics: A comparison of twelve countries. New York, NY: John Wiley.

11.

Kurz

(2011). Access to what should be taught and will be tested: Students’ opportunity to learn the intended curriculum. In Elliott

S. N.

Kettler

R. J.

Beddow

P. A.

Kurz

(Eds.), Handbook of accessible achievement tests for all students: Bridging the gaps between research, practice, and policy (pp. 99-129). New York, NY: Springer.

12.

Kurz

Elliott

S. N.

(2011). My instructional Learning Opportunity Guidance System (MyiLOGS). Tempe: Arizona State University.

13.

Kurz

Elliott

S. N.

Kettler

R. J.

Yel

(2014). Assessing students’ opportunity to learn the intended curriculum using an online teacher log: Initial validity evidence. Educational Assessment, 19, 159–184. doi:10.1080/10627197.2014.934606

14.

Kurz

Elliott

S. N.

Lemons

C. J.

Zigmond

Kloo

(2014). Opportunity to Learn: A differentiated opportunity structure for students with disabilities in general education classrooms. Assessment for Effective Intervention, 40(1), 24–39. doi:10.1177/1534508414522685

15.

Kurz

Elliott

S. N.

Wehby

Smithson

(2010). Alignment of the intended, planned, and enacted curriculum in general and special education and its relation to student achievement. The Journal of Special Education, 44, 131–145. doi:10.1177/0022466906341196

16.

Porter

A. C.

(2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3–14.

17.

Public Law 114-95. Every Student Success Act (2015). Retrieved from https://www.congress.gov/bill/114th-congress/senate-bill/1177/text

18.

Rowan

Correnti

(2009). Studying reading instruction with teacher logs: Lessons from the Study of Instructional Improvement. Educational Researcher, 38, 120–131.

19.

Scheerens

Bosker

(1997). The foundations of educational effectiveness. New York, NY: Pergamon.

20.

Schmidt

W. H.

Burroughs

N. A.

Zoido

Houang

R. T.

(2015). The role of schooling in perpetuating educational inequality: An international perspective. Educational Researcher, 44, 371–386. doi:10.3102/0013189X15603982

21.

Schmidt

W. H.

McKnight

C. C.

Houang

R. T.

Wang

H. A.

Wiley

D. E.

Cogan

L. S.

Wolfe

R. G.

(2001). Why schools matter: A cross-national comparison of curriculum and learning. San Francisco, CA: Jossey-Bass.

22.

Schmidt

W. H.

McKnight

C. C.

Valverde

G. A.

Houang

R. T.

Wiley

D. E.

(1997). Many visions, many aims volume 1: A cross-national investigation of curricular intentions in school mathematics. Dordrecht, The Netherlands: Kluwer Academic.

23.

Schulte

A. C.

Stevens

J. J.

Elliott

S. N.

Tindal

Nese

J. F.

(2016). Achievement gaps for students with disabilities: Stable, widening, or narrowing on a state-wide reading comprehension test? Journal of Educational Psychology, 108, 925–942. doi:10.1037/edu0000107

24.

Stevens

Schulte

Elliott

S. N.

Nese

J. F.

Tindal

(2015). Mathematics achievement growth of students with and without disabilities on a statewide achievement test. Journal of School Psychology, 53, 45–62. doi:10.1016/j.jsp.2014.11.001

25.

Weigert

S. C.

(2018). U.S. policies supporting inclusive assessments for students with disabilities: A 60-year history. In Elliott

S. N.

Kettler

R. J.

Beddow

P. A.

Kurz

(Eds.), Handbook of accessible instruction and testing practices: Issues, innovations, and applications (pp. 17–36). New York, NY: Springer.

26.

Winfield

L. F.

(1987). Teachers’ estimates of test content covered in class and first-grade students’ reading achievement. Elementary School Journal, 87, 437–454.

27.

Winfield

L. F.

(1993). Investigating test content and curriculum content overlap to assess opportunity to learn. Journal of Negro Education, 62, 288–310.