An Analysis of the Rise and Fall of the AA-MAS Policy

Abstract

In 2005, to address concerns about students who might fall in the “gap” between the regular assessment and the alternate assessment based on alternate achievement standards (AA-AAS), the U.S. Department of Education announced that states could develop alternate assessments based on modified achievement standards (AA-MAS). This article reports empirical evidence on characteristics of students participating in the AA-MAS, how students were assigned to test types, research on changes in assessment designs over a 4-year period, and changes in rates of participation and proficiency. The results suggest states and districts have struggled with how to appropriately assign students to this test option, and that there is a need to ensure this group of students has access to rigorous standards-based content. It concludes with a discussion of what can be learned from this policy attempt to resolve a very real problem as the United States moves toward the next generation of assessments.

Keywords

alternate assessment standard-based assessment policy AA-MAS

The reauthorization of the Elementary and Secondary Education Act (ESEA) as No Child Left Behind (NCLB) in 2001 affirmed that students with disabilities must be included in state accountability systems (Thurlow, 2002). In general, most students with disabilities participate in the regular state assessment with or without accommodations. Individuals with Disabilities Education Act (IDEA) 1997 required states to develop alternate assessments for students with disabilities who were unable to participate in the regular assessment. Then, in 2003, federal regulations allowed performance on the alternate assessments based on alternate achievement standards (AA-AAS) to count for proficiency for up to 1% of the total student population. All states developed an AA-AAS for the students with the most significant cognitive disabilities. At that time, there were concerns that some students might fall into a “gap” between the regular test and the AA-AAS (see, for example, HB 05-1246 Study Committee, 2005).

NCLB also required states to close the achievement gap between subgroups of students (e.g., ethnic/racial, economically disadvantaged, English language learners, special education). Some school districts were concerned that they might fail to make adequate yearly progress (AYP) in closing the achievement gap and wanted relief. Educators were worried about the large numbers of students with disabilities who were failing to achieve proficiency on the regular test, with or without accommodations, but did not qualify for the AA-AAS, and who were not likely to achieve proficiency on grade-level standards.

To address these concerns, on December 14, 2005, the then Secretary of Education Margaret Spellings issued a decree that set the assessment community and state education agencies scrambling to find new ways to assess a group of students with disabilities who were not making timely progress toward proficiency on grade-level achievement standards. Spellings (2005) stated,

Earlier this year, I announced a new policy designed to help states better ensure the achievement of students with disabilities. That policy allows states to develop modified achievement standards and use assessments aligned with those modified standards for a group of students with disabilities who can make progress toward, but may not reach, grade-level achievement standards in the same time frame as other students.

In a follow-up to Secretary Spellings’ comments, and in an attempt to address the needs of low-performing students with disabilities, in April 2007, federal regulations were released that gave states the option of developing alternate assessments based on modified achievement standards (AA-MAS), which is sometimes referred to as the 2% rule, because for accountability purposes up to 2% of all students may be deemed proficient using the AA-MAS.

But only 4 years after the regulations were enacted, on March 15, 2011, Secretary of Education Arne Duncan indicated that the U.S. accountability system was moving away from the AA-MAS:

I want to say here and now for the record that we are moving away from the 2% rule. We will not issue another policy that allows districts to disguise the educational performance of 2% of students. That’s unacceptable, and that must change. We have to expect the very best from our students and to tell the truth about student performance so that we can give all students the supports and the services they need. (Duncan, 2011)

There were indications even before Secretary Duncan’s pronouncement that federal support of the AA-MAS was waning. In September 2010, the Race to the Top Assessment (RTT-A) program provided funding to consortia of states to develop assessments that are valid, support and inform instruction, and provide accurate information about what students know and can do. The consortia were to develop a regular assessment that was based on college and career ready common core state standards. The RTT-A request for proposals indicated that the new assessments developed by the consortia should meet the needs of all students except for the 1% who are deemed proficient through the use of an AA-AAS (U.S. Department of Education, 2010).¹ This suggests that the assessments developed by the consortia are supposed to meet the needs of students currently being assessed with an AA-MAS, as well as those students with disabilities in the regular assessment. Two RTT-A consortia were awarded grants to develop the new assessments: Smarter Balanced Assessment Consortium (SBAC) and Partnership for the Assessment of Readiness for College and Careers (PARCC). Forty-five states and the District of Columbia are members of one or both consortia. In addition, two General Supervision Enhancement Grant (GSEG) multistate consortia—Dynamic Learning Maps (DLM) and the National Center and State Collaborative (NCSC)—are developing new AA-AAS for students with the most significant cognitive disabilities.

In 2011, the requirements for waiver applications from states seeking flexibility from some of the ESEA requirements stated that the states could not use AA-MAS scores toward their proficiency counts for accountability purposes. As of November 2012, 35 states (plus the District of Columbia) had received flexibility waivers, and additional states were revising their applications. The AA-MAS is quickly moving toward becoming merely a blip in the history of assessments used for accountability purposes, but still it is an experiment that provides us with valuable lessons about assessment of students with disabilities.

What had happened? In just a few years, we witnessed an abrupt about-face in federal policy on accountability for the performance of students with disabilities. The policy/test may be going away, but struggling learners still exist. School personnel will still be challenged mightily to engage in appropriate assessment and instructional intervention for struggling students with or without disabilities, and states will need to consider how to move this group forward toward attaining grade-level standards and proficiency on state assessments. As the United States moves to the next generation of assessments, it is important to develop an understanding of what happened during this foray into the AA-MAS so that policy makers, states, and the consortia can learn from the past about the characteristics of low-performing students, with and without disabilities, and how to better instruct and assess these students.

In this article, we examine what we learned from our research on state and district attempts to use the AA-MAS. We review legal and regulatory policies, variability in the application of those policies, and variability in the performance of students who took the AA-MAS. We also examine variability across states in test-type eligibility criteria, participation in AA-MAS, and variability in student performance on the AA-MAS. We conclude with a discussion of what can be learned from this, perhaps misguided, policy attempt to resolve a very real problem as the United States moves toward the next generation of assessments.

Background Information

The April, 2007, regulations indicated that the AA-MAS were designed for

A small group of students with disabilities whose progress is such that, even after receiving appropriate instruction, including special education and related services designed to address the students’ individual needs, the students’ individualized education program (IEP) teams are reasonably certain that the students will not achieve grade-level proficiency within the year covered by the students’ IEPs (U.S. Department of Education, 2007, p. 17748).

The regulations also indicated that students eligible for an AA-MAS may be from any disability category and must have an Individualized Education Program (IEP). The IEP must be standards-based and include annual goals that align with grade-level academic content standards. Students who take the AA-MAS must have access to the grade-level curriculum.

IEP teams were required to demonstrate that, even though the student was provided with appropriate grade-level instruction, he or she was unlikely to achieve grade-level proficiency within the year covered by the student’s IEP (U.S. Department of Education, 2007).

Over the next 4 years, 17 states developed or implemented an assessment they considered to be an AA-MAS, but the devil’s in the details. During that time period, only four states successfully completed the U.S. Department of Education’s peer review process that determines whether the assessment fulfills the necessary requirements for the state to receive federal funds (Price, Hodgson, Lazarus, & Thurlow, 2011). Based on peer review analyses, it appeared that states found it challenging to identify the population that met the regulatory requirements and to design a less difficult assessment of appropriate difficulty and complexity for low-performing students with disabilities whose instruction (as indicated by the regulations requirements) was based on grade-level content standards (Filbin, 2008).

The Characteristics of the Students

Educators debated and continue to debate the nature of the population of students with disabilities eligible for participation in the AA-MAS. This population of students with disabilities has often been difficult to define but is typically described as persistently low-performing students, students with persistent academic difficulties, or “students in the gap” (HB 05-1246 Study Committee, 2005; U.S. Department of Education, 2007). Researchers in many states have attempted to gain a better understanding of the characteristics of these students (Bechard & Godin, 2007; HB 05-1246 Study Committee, 2005; Hess, McDivitt, & Fincher, 2008). The researchers defined low-performing students somewhat differently across studies, but all found that low-performing students included both students with and without disabilities. These related studies indicated that the lowest-performing students often come from historically underserved populations, including students of ethnic minority, low socioeconomic status, or special education (Lazarus & Quenemoen, 2011).

More recently, Wu et al. (2012) investigated whether the characteristics of the lowest performing students in special education who took the regular assessment differed from the characteristics of the lowest performing non-special education students who took the regular assessment in four states.² Low-performing students were defined by Wu et al. as students who scored at the 10th percentile or below on the statewide assessment in any of 1 of 3 years of data.³ Persistently low-performing students were defined as students who scored at the 10th percentile or below for all 3 of the years. Wu et al. found that the characteristics of persistently low-performing students from special education and non-special education were similar—and more likely to be male, non-White, and from low-income backgrounds than the total group of students (which included both special education and non-special education students) in the same grade level. They also found special education students identified as “low performing” in Year 1 of the data set were more likely to become persistently low performing on state reading and mathematics assessments than their low-performing non-special education peers. These findings were further confirmed in a study by Shaftel and Rutt (2011) that examined data from another state. Shaftel and Rutt found that students who participated in the AA-MAS were “more likely to be of minority ethnicity, to be English language learners, and to experience poverty” (p. 1) than students who participated in the regular assessment.

Cho and Kingston (2011) found that students who took the AA-MAS belonged primarily to two disability categories (learning disabilities [LD] and intellectual disabilities). According to Bechard and Snow (2010), students who took an AA-MAS tended to be concrete learners. Cameto and Nagle (2010) had similar findings in an examination of behavioral and academic characteristics of students who took an AA-MAS in one state. They found that the participants were those who read slowly, had difficulty drawing inferences from grade-level text, and had difficulty answering comprehension questions. In math, they demonstrated difficulty with problems requiring multiple-step solutions and had slow or inaccurate retrieval of basic math facts. According to Cameto and Nagle, these students had difficulty with organization and keeping track of work, needed frequent clarification of instructions, failed to follow directions, had difficulty finishing assignments, and were easily distracted. In short, these are the characteristics attributed to both students with specific LD and low-achieving/at risk learners in most textbooks.

After the 2007 regulations were enacted, some local education agency (LEA) staff and other stakeholders (e.g., parents, representatives of institutions of higher education) had misperceptions about the characteristics of students who could participate in an AA-MAS. In an attempt to better understand who LEA staff and other stakeholders believed would be candidates to participate in an AA-MAS and how it aligned (or did not align) with the federal regulation requirements, one state conducted a series of meetings and found that many participants initially envisioned students who were not qualified to participate according to the regulations (e.g., students with 504 plans; students who have self-esteem, motivation, lack of sleep, hunger, and health or attendance issues, etc.) as possible candidates for this assessment option. Once the meeting attendees understood who the assessment was for (based on the regulations), there was a general realization that the issues were more about instruction than about assessment (Berndt et al., 2011).

A debate ensued around the characteristics of the instruction for students who would participate in the AA-MAS. For example, Zigmond and Kloos (2009) reported that the regulations (§200.1(e)(2)(ii)(A)) stipulate that a student should not be assigned to an AA-MAS if the child’s IEP is not of “high quality” and “specially designed to . . . move [the] child closer to grade-level achievement.” Zigmond and Kloos raised the question of whether that means a struggling student being taught reading or mathematics at an instructional level rather than at “grade-level” would be precluded from taking the AA-MAS. They asked whether that meant that a struggling student with an IEP that was not standards-based would be kept from taking the AA-MAS for at least a year, during which time the IEP team would write a better IEP.

Assignment of Students to Test Types

IEP teams are charged with the task of assigning students to the various types of tests: general, AA-AAS, and AA-MAS. The validity of the state accountability process is dependent on accuracy in assigning students to test type. IEP teams use participation criteria to determine which assessment option is appropriate for a student. Each state developed its own set of guidelines but generally included criteria that aligned with the regulations.

However, in a study that used 2008 and 2009 data from one Midwestern state, Cho and Kingston (2011) found that some students who had been deemed proficient in 1 year through the use of the general assessment ended up in the AA-MAS the next year. The investigators considered the test type taken in 2008 (general, AA-MAS, AA-AAS) and 2009 as well as the extent to which students were proficient across years. Cho and Kingston found that 5.3% of students assigned to the 2009 AA-MAS reading assessment and 6.2% of those assigned to the 2009 math assessment scored proficient or above on the previous year’s (2008) general reading and math assessments. The findings of Cho and Kingston raised major questions about the validity of the process by which teams assigned students to test types.

Research on Changes to Tests to Reduce Difficulty

There have been a number of experimental studies of the effects of changing items or tests to better meet the needs of low-performing students with disabilities who might participate in an AA-MAS. Changes to tests and items have consisted primarily of a variety of attempts to make tests less difficult for students with disabilities, an action that should result in “differential boost” in their performance. Some typical changes included in a package of changes were removal of a response option, simplified language, addition of a graphic, change in layout, embedding questions in passages, and simplified graphics.

Elliott et al. (2010) administered a modified version and a non-modified version of an assessment to 755 eighth-grade students from four states. The modified version resulted in differential boost and in more students with disabilities achieving proficiency. The changes Elliott et al. included were rewriting items to lower language load, changing response options, and various other formatting changes (adding white space, bolding important words, etc.). In an exploratory analysis, shortening the question stem was found to potentially be a highly effective change, whereas adding graphics to reading items was a poor change in terms of affecting student performance. Another study (Kettler, 2011) looked at “modification” packages in reading and math content areas for seventh-grade students. In general, measurement precision was better for eligible students on the “modified” test. A differential boost was seen for reading, but results were more mixed for math. Overall, studies demonstrated that items could be successfully changed to reduce the difficulty and potentially increase access for eligible students (Kettler et al., 2011).

Purpose

As states and consortia transition to the next generation of assessments, the lessons learned from the research that looked at issues related to the AA-MAS can provide a window into how to better instruct and assess low-performing students. The literature provides many clues about the characteristics of low-performing students and what types of test changes might be needed to create a valid AA-MAS. However, little is known about how states’ understanding of who the students are who might be candidates for an AA-MAS has evolved over time. In addition, there is little longitudinal information about how the changes states made to assessments to create an AA-MAS have been changed in response to federal peer review feedback and to the emerging literature base. The purpose of this study was to explore how states defined the students who participated in the AA-MAS and how those definitions changed over time, how assessments were changed for this group, how participation rates and proficiency have changed, and the implications for the next generation of assessments. The following specific research questions were addressed:

Research Question 1: What participation criteria have states and districts used in identifying students who should participate in the AA-MAS and how have these changed from 2007–2008 through 2010–2011?

Research Question 2: What is the nature of changes that have been made in the design of the AA-MAS from 2007–2008 through 2010–2011?

Research Question 3: Were there changes in the participation rate for the AA-MAS from 2007–2008 through 2010–2011?

Research Question 4: Have there been any changes in rates of proficiency on the AA-MAS from 2007–2008 through 2010–2011?

Research Question 5: Who is and is not getting counted in AYP based on their AA-MAS scores? Are there students who score proficient on the AA-MAS and who do not get counted in AYP?

Method

Information on states’ participation guidelines and selected test design characteristics for states’ AA-MAS were gathered from states’ websites. The information was then compiled and analyzed. States were given the opportunity to verify that the compiled information was correct. If a state requested a change, the written documentation of the source of the requested change was required before the change was made. For additional details about how the information was gathered and compiled, see Albus, Lazarus, Thurlow, and Cormier (2009); Hodgson, Lazarus, and Thurlow (2010); Lazarus, Hodgson, Price, and Thurlow (2011); Lazarus, Hodgson, and Thurlow (2010); Lazarus, Rogers, Cormier, and Thurlow (2008); Lazarus, Thurlow, Christensen, and Cormier (2007); and Price et al. (2011).

Data were also compiled and analyzed on the proficiency levels of students who participated in an AA-MAS. The data included (a) the number of students participating in the AA-MAS by state; (b) the percentage of students deemed proficient on states’ AA-MAS; and (c) the percentage counted as proficient and nonproficient for AYP purposes. The source of the proficiency data were the Annual Performance Report (APR) data submitted to the Office of Special Education Programs (OSEP) by states receiving Part B funding under the Individuals With Disabilities Education Act (IDEA).

Results

Changes in Participation Criteria Over the 4 Years

Prior to the federal regulations, several states had assessments that were very similar to an AA-MAS. In 2007, when federal regulations were first published, six states had operational assessments they considered to be an AA-MAS described on their websites. Over time, additional states joined the rank of states with an AA-MAS. As previously noted, by 2011, 17 states had operational assessments that they considered to be an AA-MAS, including the four that had received federal peer approval.

Table 1 shows the percentage of states with an AA-MAS whose participation guidelines contained selected criteria from 2007–2008 to 2010–2011. Across all 4 years, as required in the regulations, all states’ guidelines indicated that the student must have an IEP. However, the percentage of states with an AA-MAS that used other selected criteria has changed substantially over the years. In 2007–2008, two thirds of the states used performance multiple years behind grade-level expectations as a criterion—4 years later no state used that criterion. In 2007–2008, the guidelines of no states included previous performance on multiple measures; by 2010–2011, it was included in the guidelines of 94% of the states. And, in 2007–2008, only 17% of the states used not progressing at rate expected to reach grade-level proficiency within the school year covered by IEP as a criterion—by 2010–2011, all states had this criterion. Other criteria whose inclusion in guidelines increased substantially across years included the following: learning grade-level content (50% to 94%), IEP includes goals based on grade-level content standards (17% to 88%), receives specialized/individualized instruction (17% to 76%), previous performance on state assessment (33% to 71%), receives accommodations during classroom instruction (0% to 53%), and not receiving instruction based on extended or alternate standards or not eligible to take the AA-AAS (0% to 47%).

Table 1.

States’ AA-MAS Participation Criteria Across 4 Years.

	Percentage of states
Participation criteria	2007–2008 (n = 6)	2008–2009 (n = 9)	2009–2010 (n = 14)	2010–2011 (n = 17)
Use of criterion remained stable
Has IEP	100	100	100	100
Not based on disability category label	67	67	57	65
Not due to excessive absences, social, cultural, language, economic, or environmental factors	67	44	50	65
Not based on placement setting	33	33	43	34
Use of criterion decreased
Does not have a significant cognitive disability	67	33	29	57
Not determined administratively	33	22	14	24
Performance multiple years behind grade-level expectations	67	22	21	0
Use of criterion increased
Not progressing at rate expected to reach grade-level proficiency within school year covered by IEP	17	67	79	100
Previous performance on multiple measures	0	78	86	94
Learning grade-level content	50	78	79	94
IEP includes goals based on grade-level content standards	17	56	64	88
Receives specialized/individualized instruction	17	56	50	76
Previous performance on state assessment	33	44	43	71
Receives accommodations during classroom instruction	0	56	43	53
Cannot demonstrate knowledge on regular assessment even with the provision of accommodations	17	67	57	50
Not receiving instruction based on extended or alternate standards or not eligible to take the AA-AAS	0	44	50	47
Receives or has received research-based interventions	0	11	21	29
Receives high-quality instruction	0	0	21	18

Note. AA-MAS = alternate assessment based on modified achievement standards; IEP = individualized education program; AA-AAS = alternate assessment based on alternate achievement standards.

Changes in Assessment Designs Over the 4 Years

Many states that developed an AA-MAS changed an existing grade-level test rather than develop a new one. The most common changes included reducing vocabulary, shortening the test, or shortening reading passages. Across all years in this study, most states with this assessment option used a multiple-choice test; sometimes, however, the AA-MAS in a few states also included a writing prompt or items that required a constructed response. One state had a portfolio assessment.

The AA-MAS that states developed differed from the regular tests in a number of ways. Table 2 shows selected design changes that states made between 2007–2008 and 2010–2011. The use of the following design changes increased the most across years: key text underlined/bolded/bulleted (17% to 82%), additional graphics (17% to 65%), simplified graphics (17% to 53%), and segmenting of passages (17% to 58%). Other design changes whose use increased substantially between the 2007–2008 and 2010–2011 school years were as follows: simplified language (67% to 94%), distractor removed (67% to 82%), fewer items (67% to 82%), and shorter passages (50% to 82%). The use of fewer passages decreased from 50% of the states in 2007–2008 to 35% in 2010–2011.

Table 2.

States’ Assessment Design Changes for the AA-MAS Across 4 School Years.

	Percentage of states
Test characteristics	2007–2008 (n = 6)	2008–2009 (n = 9)	2009–2010 (n = 12)	2010–2011 (n = 17)
Simplified language	67	56	50	94
Distractor removed	67	67	64	82
Fewer items	67	89	57	82
Key text underlined/bolded/bulleted	17	33	50	82
Shorter passages	50	56	57	82
Fewer items/page	50	56	57	71
Larger font size	50	44	50	71
Additional graphics	17	56	36	65
Additional white space	0	11	21	59
Simplified graphics	17	22	29	59
Embedded formulas/conversions	0	11	21	53
One-column format	0	33	36	53
Segmenting of passages	17	33	21	53
Limit steps in multistep problems	0	11	14	47
Fewer passages	50	44	29	35
Graphic organizers	0	0	21	35
Different typeface	0	22	29	29
Add hint/thought boxes	0	22	14	29
Eliminate grid-in items	0	11	14	24
Breaks as needed	33	11	7	18
Calculator	33	11	14	12
Simplified numbers	0	22	14	8

Note. AA-MAS = alternate assessments based on modified achievement standards.

Changes in Rate of Participation and in Proficiency in State AA-MAS

Tables 3 and 4 are summary tables showing the number of students participating in the AA-MAS, and student participation and proficiency rates in reading and mathematics at Grades 5 and 8 across the 12 states that reported AA-MAS data in their 2009–2010 APR reports. The tables show that during the 2009–2010 school year, California had the most students participating in the AA-MAS at Grade 5 across content areas (math: 20,432; reading/English Language Arts [ELA]: 22,959); at Grade 8, California had the most students participating in the reading/ELA AA-MAS (18,308) while Texas had the most students participating in the math AA-MAS (16,591). Rates of participation in the mathematics AA-MAS ranged from a low of 0.9% of all fifth graders in Michigan and 0.9% of the eighth graders in California and Michigan, to a high of 8.3% of eighth graders in Oklahoma. Rates of participation in the reading assessment were similar, ranging from 1.0% of eighth graders in Michigan to 9.0% of fifth graders in Oklahoma.

Table 3.

Mathematics AA-MAS Summary, APR Data, 2009–2010.

	Number of students who took AA-MAS		Percentage of all students who took AA-MAS		Percentage of students with IEPs who took AA-MAS		Percentage proficient on AA-MAS out of total population		Percentage counted as proficient toward AYP		Percentage proficient who counted as nonproficient
State	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8
California	20,432	4,112	4.4	0.9	38.7	8.3	1.7	0.1	1.7	0.1	0.0	0.0
Connecticut	1,375	1,060	3.3	2.5	25.4	19.7	2.3	1.0	2.0	1.0	0.3	0.0
Kansas	1,070	1,038	3.1	3.1	22.4	24.5	2.3	2.2	2.0	2.0	0.3	0.2
Louisiana	1,052	2,276	2.1	4.4	18.4	33.7	0.4	0.7	0.4	0.7	0.0	0.0
Maryland	1,438	1,948	2.4	3.1	18.6	26.6	0.9	0.7	0.9	0.7	0.0	0.0
Michigan	1,099	1,080	0.9	0.9	6.2	6.4	0.6	0.3	0.6	0.3	0.0	0.0
North Carolina	3,442	2,963	3.0	2.7	22.9	23.2	2.6	2.5	2.0	2.0	0.6	0.5
North Dakota	192	216	2.8	2.9	18.6	21.5	2.3	2.3	2.0	2.0	0.3	0.3
Oklahoma	3,910	3,783	8.1	8.3	45.0	48.8	3.7	3.1	2.0	2.0	1.7	1.1
Pennsylvania	2,590	3,063	1.9	2.2	11.1	13.1	1.0	0.9	1.0	0.9	0.0	0.0
Tennessee	1,975	1,700	2.6	2.4	22.2	21.7	0.4	0.1	0.4	0.1	0.0	0.0
Texas	16,100	16,591	4.4	4.8	43.6	45.3	3.9	3.8	2.0	2.0	1.9	1.8

Note. AA-MAS = alternate assessments based on modified achievement standards; APR = annual performance report; IEP = individualized education program; AYP = adequate yearly progress.

Table 4.

Reading/ELA AA-MAS Summary, APR Data, 2009–2010.

	Number of students who took AA-MAS		Percentage of all students who took AA-MAS		Percentage of students with IEPs who took AA-MAS		Percentage proficient on AA-MAS out of total population		Percentage counted as proficient toward AYP		Percentage proficient who counted as nonproficient
State	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8	Grade 5	Grade 8
California	22,959	18,308	4.9	3.8	43.4	36.7	1.6	1.0	1.6	1.0	0.0	0.0
Connecticut	1,796	1,177	4.3	2.7	33.2	21.9	2.8	1.7	2.0	1.7	0.8	0.0
Kansas	1,234	1,036	3.6	3.0	25.8	24.4	3.0	2.7	2.0	2.0	1.0	0.7
Louisiana	1,060	2,267	2.1	4.3	18.5	33.6	0.4	1.0	0.4	1.0	0.0	0.0
Maryland	1,462	1,972	2.4	3.1	18.9	27.0	0.9	1.4	0.9	1.4	0.0	0.0
Michigan	1,398	1,192	1.2	1.0	7.9	7.0	0.6	0.6	0.6	0.6	0.0	0.0
North Carolina	4,098	3,156	3.6	2.9	27.2	24.7	2.8	2.5	2.0	2.0	0.8	0.5
North Dakota	257	239	3.7	3.3	24.8	23.8	3.0	2.8	2.0	2.0	1.0	0.8
Oklahoma	4,305	3,753	9.0	8.2	49.4	48.8	3.7	3.9	2.0	2.0	1.7	1.9
Pennsylvania	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA
Tennessee	1,963	1,676	2.6	2.4	22.1	21.4	1.4	0.4	1.4	0.4	0.0	0.0
Texas	16,425	14,733	4.5	4.2	44.5	40.2	4.3	3.9	2.0	2.0	2.3	1.9

Note. ELA = English language arts; AA-MAS = alternate assessments based on modified achievement standards; APR = annual performance report; IEP = individualized education program; AYP = adequate yearly progress; NA = state did not offer an AA-MAS option for that grade level.

Because the federal regulations only allow a maximum of 2% of all students to be counted as proficient using an AA-MAS for AYP purposes, we considered the percentage of students participating in the AA-MAS who were proficient, and the percentage participating in AA-MAS who demonstrated proficiency but whose scores may not be counted as proficient because they exceeded the 2% allowance.

We found that there were a large number of students who participated in the AA-MAS, who scored proficient on the assessment, and whose scores did not count because they exceed the 2% cap for their state. As shown in Tables 3 and 4, there was considerable variability in percentages of students with IEPs who participated in an AA-MAS. In several states, students who scored proficient were about twice the number of students allowed to be counted as proficient toward AYP. For example, in Texas, 4.3% of students in the state received scores that indicated they are proficient in fifth grade reading/ELA, but only 2.0% counted for AYP purposes.

The percentage of students who scored proficient but did not count toward AYP was affected by (a) the percentage of students with IEPs who participated in the AA-MAS and (b) the cut point that determined whether a student scored at a proficient level. States where a relatively high percentage of students with IEPs participated in the AA-MAS generally also had a higher percentage who scored proficient but did not count toward AYP. However, there were cases where there was relatively high participation in the AA-MAS yet the state did not bump up against the cap. For example, according to Table 4, in California, 4.9% of all Grade 5 students (43.4% of students with IEPs) took the AA-MAS in reading, yet only 1.6% of all students scored proficient using this assessment option. There were also cases of states with relatively low participation rates that went over the cap. For example, as shown in Table 3, in North Dakota, 2.8% of all Grade 5 students (18.6% of students with IEPs) took the AA-MAS, but 2.3% of all students scored at proficient levels with this assessment. This suggests that students who took the AA-MAS were much more likely to receive proficient scores than students who took the regular test.

Little relationship was found between the number of criterion in states’ participation guidelines and the percentage of students participating in the AA-MAS. As shown in Figures 1 and 2, in 2009–2010, for states that reported APR data, states had between 9 and 15 criteria in their participation guidelines. North Dakota had the fewest criteria (9) and Connecticut had the most (15). The lowest percentage of students with IEPs participated in the AA-MAS in Michigan in both reading and mathematics in Grade 5—6.2% in mathematics and 7.9% in reading/ELA. The highest percentage was in Oklahoma—45.0% of students with IEPs participated in the AA-MAS in mathematics and 49.4% in ELA. Across all states, a higher percentage of the students with IEPs participated in the AA-MAS in reading/ELA than in mathematics.

Figure 1.

Number of participation criteria and percentage of students with IEPs participating in AA-MAS—Mathematics, 2009–2010.

Figure 2.

Number of participation criterion and percentage of students with IEPs participating in AA-MAS—Reading/ELA, 2009–2010.

Discussion and Policy Implications

It has been 7 years since Secretary Spellings issued her mandate for AA-MAS and 5 years since we have had federal regulations designed to guide practice. The Secretary’s mandate relieved school districts from having to report large numbers of students as nonproficient and enabled them to use modified assessment procedures to account for instructional progress that did not bring low-performing students up to grade-level proficiency.

It is clear that the AA-MAS mandate came about after pressure from the field. Educators were concerned about the large numbers of students with disabilities who were failing to achieve proficiency on grade-level tests, with or without accommodations, who did not qualify for the AA-AAS, and who were not likely to achieve proficiency on grade-level content standards for a very long time. The mandate was designed to help states count more of their students with disabilities as proficient, though proficient based on a modified set of achievement standards based on grade-level content standards. The mandate was issued without sufficient detail, and state and school personnel were left to make decisions about which students were eligible as well as the kinds of changes that could be made in expectations and the assessment of those expectations. Federal policy played out differently in different states and resulted in some cases in vastly different practices across states.

Participation in the AA-MAS was limited to students with disabilities, in spite of the fact that very large numbers of students without disabilities were persistent low performers and not expected to achieve grade-level content standards within a year. Many low-performing students, both with and without disabilities, are from subgroups that historically have been denied access to high-quality instruction (ethnic/racial minorities, low socioeconomic status; Lazarus & Thurlow, 2012; Perie, Fincher, Payne, & Swaffield, 2012; Shaftel & Rutt, 2011). Quenemoen (2010) asked, “What evidence exists to suggest that students with disabilities who are low-performing differ from minority students or poor students who are low-performing?” and “If the consequences of participating in an AA-MAS are positive, how can we deny the same opportunities to other students with similar achievement profiles other than disability status? If the consequences are negative, how can we justify the use of the options for students with disabilities?” (p. 22).

States’ participation guidelines often are difficult for IEP teams to use for decision-making purposes. According to Zigmond and Kloos (2009), IEP teams were sometimes left to choose among those with disabilities who were the almost proficient and those who were nowhere near proficient. Though we know of no data on the topic, we would assume that some low-performing students without disabilities were identified as having a disability to qualify them to take the AA-MAS, because historically students have been retained, remediated, or referred and classified as having a disability to escape being counted in state accountability systems (Allington & McGill-Franzen, 1992; McGill-Franzen & Allington, 1993). For example, in Texas there was a significant increase in the numbers of secondary age students referred for possible LD classification when the state permitted students with disabilities to achieve their high school diploma by meeting their IEP objectives rather than by passing the high school exit exam.

The 2007 regulations may have encouraged districts to identify additional students as having a disability as there are no restrictions on the numbers of students with disabilities who can participate in the AA-MAS. However, states and districts were permitted to count as proficient a maximum of 2% of their students. This translates to roughly 20% of students with disabilities. We learned that those states that actually developed and used an AA-MAS included from 1% to 9% of all students (6% to 49% of students with disabilities) in this assessment. Given that the maximum percentage of students who may be counted as proficient toward AYP is 2% of all students, large numbers of students are being excluded from state accountability systems. We found that as many as 2% to 4% of students who participate in the AA-MAS are scoring proficient but not having their scores count. This reflects, we think, the overassignment of students with disabilities to the AA-MAS and is reflected by the percentage of students participating.

States’ participation guidelines are supposed to be used by IEP teams to make participation decisions. It might be expected that a lower percentage of the students with disabilities might participate in the AA-MAS in states with stringent guidelines. The number of criteria in a state’s guidelines might potentially be a proxy for rigor of the guidelines. However, little relationship was found between participation rates in the AA-MAS across states and the number of criterion in states’ guidelines.

The findings of this study suggest that some states’ ideas about who the students were who might qualify for this assessment option evolved over time. States’ participation guidelines focused more on the specific requirements in the 2007 regulations—and criteria such as multiple grades below grade level that were widely used in 2007 were dropped by most states by 2010–2011, whereas the use of criteria that used the language in the regulations, such as not progressing at rates expected to reach grade-level proficiency within school year covered by IEP, increased substantially. The guidelines did not necessarily become more or less stringent over time—instead they became better aligned to the requirements in the regulations.

Further research is needed to learn why states and districts made the decisions they made regarding the AA-MAS. Interviews with key individuals at both state and local levels could provide rich data that could help answer questions such as the following: Did states consider how the inclusion (or exclusion) of various participation criteria might affect the percentage of students who were proficient on this assessment? How did IEP teams use the state participation guidelines to make participation decisions? Did student participation in an AA-MAS affect the instruction that they received? Such questions should be asked to the extent that they inform decision-making processes in general or for new assessment systems.

Relevant Lessons for theNext Generation of Assessments

The results of this study illuminated several lessons that were learned from the AA-MAS that are particularly relevant as the next generation of assessments is developed.

Students need access to the content

The issuance of the AA-MAS mandate was an effort to modify assessments to solve a major instructional problem; it was an effort to increase access to grade-level content for larger numbers of students with disabilities. Yet the effort also highlighted the very large numbers of students, including students with disabilities, who are failing to achieve grade-level standards. The issue here is an instructional issue, not an assessment issue, for both low-performing students with an IEP and low-performing students who do not have an IEP. There are preliminary indications that even with the AA-MAS option, some low-performing students are not getting more access to grade-level content (Altman, Cormier, Lazarus, & Thurlow, 2012; Lazarus et al., 2011). Improving the opportunity to learn for low-performing students with disabilities may reduce the need for an AA-MAS.

Seamless participation guidelines are needed

In each RTT-A and GSEG consortia, there are some states that currently have an AA-MAS and many that do not. A lesson learned from the demand for an assessment to fill the gap is that the consortia need to develop participation guidelines that will seamlessly include all students in the new assessment systems. The development of guidelines with no gaps will require cooperation and collaboration across states and consortia.

Use the findings of AA-MAS studies to develop universally designed assessments

Some of the changes that were made to regular tests to create a test considered to be an AA-MAS could be considered elements of universal design. The results suggest some states approached design changes for their AA-MAS from a universal design perspective. Because universally designed large-scale tests promote “maximum readability and comprehensibility,” and “maximum legibility,” some AA-MAS test design changes (e.g., key text underlines/bolded/bulleted, fewer items/per page, larger front size, additional graphics, additional white space, simplified graphics, embedded formulas/conversions, one-column format) may be related to the elements of universally designed tests (Price et al., 2011; Thompson, Johnstone, Anderson, & Miller, 2005; Thompson, Johnstone, & Thurlow, 2002).

Conclusions

Understanding how to best assess low-performing students with disabilities continues to remain a challenge for states. The development of an AA-MAS in 17 states provided the opportunity to learn more about the characteristics of this population of students and how they can best demonstrate their knowledge and skills on accountability assessments. The AA-MAS option has been viewed as hiding or disguising the performance of the students for whom it is intended. The AA-MAS option may be going away, but there still is a population of struggling learners with disabilities who are challenging to instruct and assess. It is vital that we learn from the past as we move toward the next generation of assessments.

Footnotes

Authors’ Note

Opinions expressed herein do not necessarily reflect those of the U.S. Department of Education or Offices within it.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The preparation of this article was supported, in part, by a Cooperative Agreement (#H326G110002) with the Research to Practice Division, Office of Special Education Programs, U.S. Department of Education and the University of Minnesota. It was also funded with partial support from the Multi-State GSEG Toward a Defensible AA-MAS. This project is supported by General Supervision Enhancement Grants (#H373X070021) from the Research to Practice Division, Office of Special Education Programs, U.S. Department of Education.

Notes

References

Albus

Lazarus

S. S.

Thurlow

M. L.

Cormier

(2009). Characteristics of states’ alternate assessments based on modified academic achievement standards in 2008 (Synthesis Report No. 72). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

Allington

R. L.

McGill-Franzen

(1992). Unintended effects of educational reform in New York. Educational Policy, 6, 397–414. doi:10.1177/0895904892006004003

Altman

J. R.

Cormier

D. C.

Lazarus

S. S.

Thurlow

M. L.

(2012). Tennessee special education teacher survey: Training, large-scale testing and TCAP-MAAS administration (Tech. Rep. No. 61). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

Bechard

Godin

(2007). Finding the real assessment gaps: A process for states to identify gaps in their assessment systems. Newton, MA: CAST, The Education Alliance, EDC, INTASC, Measured Progress. Retrieved from http://www.measuredprogress.org/documents/10157/15651/Finding_real_assessment_gaps.pdf

Bechard

Snow

(2010, May). Identifying students in need of modified achievement standards and developing valid assessments. Poster session presented at the meeting of the American Educational Research Association, Denver, CO. Retrieved from http://www.cehd.umn.edu/NCEO/Presentations/Posters/2MTGSEGPoster.pdf

Berndt

Ebben

Kubinski

Sim

Liu

Lazarus

Christian

(2011). Who are the students who may qualify for an alternate assessment based on modified academic achievement standards (AA-MAS)?: Focus group results (Synthesis Report No. 79). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

Cameto

Nagle

(2010, May). Improving the Oklahoma modified alternate assessment program general supervision enhancement grant (GSEG) 2007-2010. Poster session presented at the meeting of the American Educational Research Association, Denver, CO. Retrieved from http://www.cehd.umn.edu/NCEO/Presentations/Posters/4OKMAASAERA.pdf

Cho

H. J.

Kingston

(2011). Capturing implicit policy from NCLB test type assignments of students with disabilities. Exceptional Children, 78, 58–72.

Duncan

(2011). Preparing students with disabilities for success: Secretary Duncan’s remarks to the American association for people with disabilities. Retrieved from http://www.ed.gov/news/speeches/

10.

Elliott

S. N.

Kettler

R. J.

Beddow

P. A.

Kurz

Compton

McGrath

Roach

(2010). Effects of using modified items to test students with persistent academic difficulties. Exceptional Children, 76, 475–495.

11.

Filbin

(2008). Lessons from the initial peer review of alternate assessments based on modified achievement standards. Washington, DC: U.S. Department of Education, Office of Elementary and Secondary Education Student Achievement and School Accountability Program.

12.

HB 05-1246 Study Committee. (2005, December 31). Assessing “students in the gap” in Colorado. Retrieved from http://education.umn.edu/nceo/Teleconferences/tele11/ColoradoStudy.pdf

13.

Hess

McDivitt

Fincher

(2008, June). Who are the 2% students and how do we design test items and assessments that provide greater access for them? Results from a pilot study with Georgia students. Paper presented at the CCSSO National Conference on Student Assessment Orlando, FL. Retrieved from http://www.nciea.org/publications/CCSSO_KHPMMF08.pdf

14.

Hodgson

J. R.

Lazarus

S. S.

Thurlow

M. L.

(2010). Characteristics of states’ alternate assessments based on modified academic achievement standards in 2009-2010 (Synthesis Report No. 80). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

15.

Kettler

R. J.

(2011). Effects of modification packages to improve test and item accessibility: Less is more. In Elliot

Kettler

Beddow

Kurz

(Eds.), A handbook of accessible achievement tests for all students (pp. 231–241). New York, NY: Springer.

16.

Kettler

R. J.

Rodriguez

M. C.

Bolt

Elliott

S. N.

Beddow

Kurz

(2011). Modified multiple-choice items for alternate assessments: Reliability, difficulty, and differential boost. Applied Measurement in Education, 23, 210–234. doi:10.1080/08957347.2011.580620

17.

Lazarus

S. S.

Hodgson

Price

L. M.

Thurlow

M. L.

(2011). States’ participation guidelines for the alternate assessment based on modified achievement standards (AA-MAS) in 2010 (Synthesis Report No. 82). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

18.

Lazarus

S. S.

Hodgson

Thurlow

M. L.

(2010). States’ participation guidelines for alternate assessments based on modified academic achievement standards (AA-MAS) in 2009 (Synthesis Report No. 75). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

19.

Lazarus

S. S.

Quenemoen

(2011). Instructing and assessing students who may be candidates for alternate assessments based on modified achievement standards (AA-MAS). In Russell

Kavanaugh

(Eds.), Assessing students in the margin: Challenges, strategies, and techniques (pp. 215–234). Charlotte, NC: Information Age Publishing.

20.

Lazarus

S. S.

Rogers

Cormier

Thurlow

M. L.

(2008). States’ participation guidelines for alternate assessments based on modified academic achievement standards (AA-MAS) in 2008 (Synthesis Report No. 71). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

21.

Lazarus

S. S.

Thurlow

M. L.

(2012). Lessons learned through diverse approaches to addressing students not reaching proficiency on regular state assessments. In Thurlow

M. L.

Lazarus

S. S.

Bechard

(Eds.), Considerations for transitioning students from operational 2% assessments to the general assessment: Challenges for states (pp. 123–163). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

22.

Lazarus

S. S.

Thurlow

M. L.

Christensen

Cormier

(2007). States’ alternate assessments based on modified achievement standards (AA-MAS) in 2007 (Synthesis Report No. 67). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

23.

McGill-Franzen

Allington

R. L.

(1993). Flunk’em or get them classified: The contamination of primary grade accountability data. Educational Researcher, 22, 19–22. doi:10.3102/0013189X022001019

24.

Perie

Fincher

Payne

Swaffield

(2012). Understanding low-performing students with disabilities and their barriers to success on traditional assessments: A Southern tale. In Thurlow

M. L.

Lazarus

S. S.

Bechard

(Eds.), Considerations for transitioning students from operational 2% assessments to the general assessment: Challenges for states (pp. 59–86). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

25.

Price

L. M.

Hodgson

J. R.

Lazarus

S. S.

Thurlow

M. L.

(2011). Characteristics of states alternate assessments based on modified academic achievement standards in 2010-2011 (Synthesis Report No. 85). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

26.

Quenemoen

R. F.

(2010). Assessing students with disabilities using modified achievement standards. In Perie

(Ed.), Teaching and assessing low-achieving students with disabilities (pp. 3–31). Baltimore, MD: Brookes.

27.

Shaftel

Rutt

B. T.

(2011). Characteristics of students who take an alternate assessment based on modified achievement standards. Journal of Disability Policy Studies, 23, 156–167. doi:10.1177/1044207311424909.

28.

Spellings

(2005). Key policy letters signed by the education secretary or deputy secretary. Retrieved from http://www2.ed.gov/policy/speced/guid/secletter/051214.html

29.

Thompson

S. J.

Johnstone

C. J.

Anderson

M. E.

Miller

N. A.

(2005). Considerations for the development and review of universally designed assessments (Tech. Rep. No. 42). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

30.

Thompson

S. J.

Johnstone

C. J.

Thurlow

M. L.

(2002). Universal design applied to large scale assessments (Synthesis Report No. 44). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

31.

Thurlow

M. L.

(2002). Positive educational results for all students: The promise of standards-based reform. Remedial and Special Education, 23, 195–202. doi:10.1177/07419325020230040201

32.

U.S. Department of Education. (2007, April 9). Final rule 34CFR Parts 200 and 300: Title I Improving the academic achievement of the disadvantaged; Individuals with Disabilities Education Act (IDEA). Federal Register, 72(67). Washington, DC: Author. Retrieved from http://cehd.umn.edu/NCEO/2percentReg/FederalRegApril9TwoPercent.pdf

33.

U.S. Department of Education. (2010). Race to the top assessment program application for new grants: Comprehensive assessment systems. Washington, DC: Author. Retrieved from http://www2.ed.gov/programs/racetothetop-assessment/resources.html

34.

Liu

K. L.

Thurlow

M. L.

Lazarus

S. S.

Altman

Christian

(2012). Characteristics of low performing special education and non-special education students on large-scale assessments (Tech. Rep. No. 60). Minneapolis: University of Minnesota, National Center on Educational Outcomes.

35.

Zigmond

Kloos

(2009). The “two percent students”: Considerations and consequences of eligibility decisions.Peabody Journal of Education, 84, 478–495. doi:10.1080/01619560903240855