Examining Teachers’ Decisions on Test-Type Assignment for Statewide Assessments

Abstract

Recent studies have raised concerns about the vagueness of alternate assessment eligibility guidelines, specifically, that students with mild disabilities (SWMD) have been inappropriately assigned to alternate assessment–alternate achievement standards (AA-AAS). In this study, special education teachers (N = 317) were surveyed about SWMD in vignettes to explore (a) perceptions of the clarity of eligibility guidelines, (b) teachers’ test-type decisions, (c) factors considered when making test-type decisions, and (d) teachers’ priority rankings of these factors. Teachers’ test-type decisions varied by state, subjectivity, and noninstructional factors. Findings raise concerns about the inconsistency and lack of specificity of the current state guidelines, their vulnerability to subjective interpretation of eligibility, and states’ different alternate assessment systems. Directions for future research, implications of the current findings, and limitations are discussed.

Keywords

alternate assessment–alternate achievement standards alternate assessment–modified achievement standards test-type assignment academic achievement

Clear guidelines for identifying the intended population of the alternate assessment–alternate achievement standards (AA-AAS, for students with the most significant cognitive disabilities), and the alternate assessment–modified achievement standards (AA-MAS, for students with disabilities who are consistently low achievers, but for whom the AA-AAS is not appropriate) are a foundation for test validity. Therefore, adequately describing the population has become a fundamental necessity in determining appropriate assessments for students. A statement from the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education clearly establishes the importance of eligibility guidelines for alternate assessments: “The population(s) for which a test is appropriate should be clearly delimited” (1999, p. 17). However, as imperative as it is, this mandate has been challenging to implement. Some special education teachers on Individual Education Program (IEP) teams have experienced challenges in appropriately identifying the intended population for participation in the AA-AAS and AA-MAS based on the current eligibility guidelines (Cho & Kingston, 2012). The purpose of the present study is to explore the effect of current eligibility guidelines—often described by researchers as broad, vague, and inconsistent (Cho & Kingston, 2012; Musson, Thomas, Towles-Reeves, & Kearns, 2010; Thurlow, 2004)—on assessment assignments for students with disabilities by surveying special education teachers across three states, one that has implemented only the AA-AAS and two that have implemented the AA-AAS and AA-MAS.

The reauthorized Individuals With Disabilities Education Act Amendments (IDEA) of 1997 (Pub. L 105-117) required (a) that all students with disabilities who had been exempt from statewide and school district assessment be included in the standard assessment systems with appropriate accommodations and modifications as necessary, and (b) that students with disabilities have access to the general education curriculum, including instruction for the grade in which they are enrolled (§612 [a][17]). This act also introduced alternate assessment for students with disabilities who cannot participate in state- or district-wide assessment programs even with appropriate accommodations. However, the regulation did not specify definitive participation eligibility criteria or give clear guidance about the format, alignment with content standards, scoring, standard setting, or technical adequacy of the alternate assessment.

The No Child Left Behind (NCLB) Act of 2001 addressed some of these limitations through required accountability measures for all states, including (a) counting up to 1% of students as proficient for purposes of Adequate Yearly Progress (AYP), (b) ensuring that the AA-AAS is “aligned with the state’s academic content standards” (34 C.F.R. § 200.1[d][1]-[3]), and (c) limiting the intended population for the AA-AAS to students with the most significant cognitive disabilities. The IDEA Improvement Act of 2004 stated that “students with the most significant cognitive disabilities” refers to those for whom the regular assessment, even with appropriate accommodations, would be an inappropriate measure of student progress within the general education curriculum. According to the U.S. Department of Education (2005), the term “students with the most significant cognitive disabilities” refers to a small number of students who (a) are within one or more of the existing categories of disability under the IDEA (e.g., autism, multiple disabilities, intellectual disability, traumatic brain injury) and (b) whose cognitive impairments may prevent them from attaining grade-level achievement standards, even with the very best instruction (p. 24).

However, this description is far from exclusive and applies to a broad array of students, ranging from those with significant cognitive disabilities to those with mild disabilities, and some with nonintellectual disabilities. Students with these disabilities have shared characteristics, and their cognitive abilities range from typical to profound disability. According to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000), an individual is classified as having mild intellectual disability if his or her IQ level is 50 to approximately 70; moderate if the IQ level is 35 to approximately 50; severe if the IQ level is 20 to approximately 35; and profound if the IQ level is below 20. Cho and Kingston (2012) suggested that these IQ ranges may not be as accurate as hoped, but they would support IEP teams and prevent them from assigning students with mild disabilities to the AA-AAS.

Thurlow (2004) described the relationship of category labels to the most significant cognitive disabilities or mild disabilities or nonintellectual disabilities. Most students with disabilities (75% altogether) have learning disabilities, speech or language impairments, other health impairments, or emotional or behavioral disabilities (Quenemoen, 2009). These students, along with those who have physical, visual, or hearing impairments (another 4%-5%), are all without intellectual impairments, and either the AA-MAS or the general education assessment is appropriate for them (Quenemoen, 2009). Students with an intellectual disability, autism, or multiple disabilities represent the primary disability categories that are candidates for AA-AAS (Cho & Kingston, 2012; Kearns, Towles-Reeves, Kleinert, Kleinert, & Thomas, 2011; Quenemoen, 2009) because they range from mild to significant cognitive disabilities. Unfortunately, the regulation required that decisions not be based on “disability category or other similar qualities” (Musson et al., 2010). It allowed each state to determine specific eligibility guidelines and criteria. This has produced inconsistency across states in their conceptualizations of classification systems for alternate assessment eligibility (Musson et al., 2010; Thurlow, 2004).

Several studies have examined the development and status of state participation guidelines for alternate assessments (Thompson, Erickson, Thurlow, Ysseldyke, & Callender, 1999; Thompson & Thurlow, 2000; Thompson, Thurlow, Johnstone, & Altman, 2005; Thurlow, 2004; Warlick & Olsen, 1999). Key findings reported by Thurlow (2004) included inconsistencies in eligibility criteria across the 50 states, as well as a lack of specificity in existing criteria (e.g., referring to students’ general or instructional characteristics), and the absence of some criteria (e.g., disability category and IQ score). Thurlow (2004) also reported that many states targeted students with significant cognitive disabilities for participation in the alternate assessment; however, about 25% of states included students in the alternate assessment who did not have a significant cognitive disability and who would have participated in the general assessment according to the guidelines of other states. Musson et al. (2010) also reported a lack of specificity in states’ eligibility guidelines, with most states including in their eligibility criteria some general characteristics of students with disabilities, such as whether the student required intensive instruction or generalization, or had significant cognitive impairment, or adaptive skills. Musson et al. (2010) also found that none of the states mentioned IQ level, IQ cutoff scores, or existing disability category, which could help differentiate mild disabilities from the most significant cognitive disabilities.

This lack of specificity in eligibility guidelines is a critical issue in states where the AA-MAS has been implemented (Cho & Kingston, 2011; Filbin, 2008). The AA-MAS is intended for students with disabilities who are not able to achieve proficiency on the general assessment and yet would not be assessed appropriately by the AA-AAS. These students fall into a gap between the two assessments. NCLB (2001) allowed states to develop an alternate assessment based on modified achievement standards and use it for accountability purposes (i.e., 2% caps). The regulations state that “there is a small group of students whose disability has precluded them from achieving grade-level proficiency and whose progress is such that they will not reach grade-level proficiency in the same time frame as other students” (34 C.F.R. Part 200).

In their study of one Midwestern state, Cho and Kingston (2011) reported that 1.6% (n = 96) of elementary school students with mild disabilities who took the 2009 AA-MAS had been assigned to the AA-AAS the year before, and 5.8% (n = 347) had taken and passed the 2008 general assessment the year before. These findings did not seem to comport with the intended use of the AA-AAS and AA-MAS, because students with mild disabilities should not be assigned to the AA-AAS, and the AA-MAS is for students with disabilities who are consistently low achievers and unlikely to master their grade-level academic content, even with the best instruction.

In their follow-up case study, Cho and Kingston (2012) interviewed six special education teachers of students with mild disabilities who were assigned the 2010 AA-AAS in the same Midwestern state. They reported that the factors teachers considered when they assigned these students to the AA-AAS were whether (a) students’ IEP goals in reading and/or math were aligned to the alternate academic standards; (b) students were low performers and performed at less than the 1 percentile level; (c) teachers were concerned about the 1% or 2% AYP caps their school had to contend with; (d) a policy existed prohibiting a read-aloud accommodation on the reading subsections of assessments; (e) students used classroom modifications and adaptations; and (f) students had significant cognitive disabilities. When asked, none of the special education teachers correctly defined the term “significant cognitive disability.” Their understandings of significant cognitive disabilities were based on their students’ need for significant adaptation and modification on the classroom materials, information processing differences, extremely low performance, and cognitive challenges, which are seen in some students with mild disabilities and in all students with the most significant cognitive disabilities. Some of these students with disabilities had grade-level comprehension skills, but were assigned to the AA-AAS because the read-aloud accommodation was not permitted for reading passages in the general assessment. Especially in small school districts, teachers claimed that their schools struggled to meet the 1% and 2% AYP caps for students taking the AA-AAS and AA-MAS.

Overall, this case study suggested the effect that the lack of clear eligibility guidelines can have: IEP teams might tend to make subjective test-type decisions regardless of their belief in or understanding of the guidelines, or the number of annual assessment assignment training sessions they had received. Although these findings provided potentially valuable information for the field, the study involved only six teachers, and their test-type decisions were based on only one state’s eligibility guidelines, so generalizability may be limited. This study provides potential benefits for the field in light of the phasing out of AA-MAS, when it is likely that more higher-risk, lower achievers with mild disabilities will be assigned to the AA-AAS.

The purpose of the present study was to examine the test-type decisions of teachers under the imprecise eligibility guidelines from three different states—one Western, one Northeastern, and one Midwestern—based on three vignettes concerning fictional students. The guidelines of all three states held the following requirements in common for assignment to the AA-AAS: The student (a) must have a current IEP, (b) must be delayed in adaptive areas, (c) requires intensive instruction in multiple settings to transfer and generalize skills, (d) is currently taught under the Alternate Academic Standards, (e) cannot participate in general education assessment, even with appropriate accommodations, and (f) must have a significant cognitive disability. The eligibility guidelines for AA-MAS in the Northeastern and Midwestern states indicate that the IEP goals must be based on the academic content standards of the grade in which the student is enrolled, that the student is not likely to achieve grade-level proficiency within the IEP’s stated timeframe, and that the decision to enroll the student in AA-MAS must be made annually and must be based on multiple objective and valid measures. The Northeastern and Midwestern states have an exclusionary criterion for the AA-AAS: The student’s delays cannot be only in the academic area. The Northeastern state also addressed that under the general education standards, delays of students with learning disabilities cannot be due to behavioral issues. The Midwestern state’s exclusionary criteria for the AA-AAS and the AA-MAS specify that none of the following could be the sole reason for assigning a student to an alternate assessment: excessive or extended absence; disability category; specific disability category label; social, cultural, or economic differences; or the amount of time a student receives special education services.

Method

This study asked 317 teachers across the three states to read three vignettes and then to identify (a) the familiarity and clarity of their state’s eligibility guidelines extended to the training for test-type assignments, (b) which math and reading assessments they would assign the three students to, (c) what factors they would consider when making assignment decisions, and (d) the rank of these factors in order of importance.

Sample

Participants were 317 special education teachers in three states who were leaders of IEP teams or case managers for students with disabilities. Of these participants, 127 were from the Midwestern state, 49 were from the Northeastern state, and 141 were from the Western state. In addition, 66.3% (n = 210) taught in middle school, 15.1% (n = 48) taught in high school, and 18.6% (n = 59) taught in elementary schools. Grouped by classroom environment, 40% (n = 127) of the teachers taught students primarily in resource classrooms, 35% (n = 111) taught in general education and resource classrooms, and 25% (n = 79) taught in self-contained rooms. Nearly 20% (19.6%) of special education teachers (n = 62) taught in rural settings, 42.2% (n = 174) taught in suburban settings, and 38.2% (n = 121) taught in urban settings.

Procedure

The first author developed questionnaires and adapted three vignettes for this survey, which was based on a case study: “Why IEP Teams Assign Low-Performers with Mild Disabilities to the AA-AAS” (Cho & Kingston, 2012). The author adapted from the case study 14 factors that teachers might consider when making test-type decisions. A pilot study was conducted to ensure the efficacy of the vignette survey: Special education teachers and a school psychologist were asked to review the survey and provide feedback regarding its clarity, utility, accuracy, and understandability. Based on the feedback, minor revisions were made to clarify the language. After securing institutional review board approval, the researchers asked the three states’ education departments to participate in a vignette survey. After reviewing the surveys, the state departments gave the researchers permission to survey teachers in their states. In November 2011, researchers emailed the online survey’s web address to the education departments. Each education department then sent an email to its special education teachers who were involved in test-type decisions on IEP teams, including information about the survey and its purpose and requesting that teachers click on the link for the online survey. The link directed users to a description of the survey’s intention, the three vignettes, and a request to complete the survey. Specifically, the survey asked special education teachers to make test-type decisions for each student in the reading and math vignettes. The survey was available for 2 months. One state’s response rate was 40%. The two other states were unable to apprise researchers of the precise number of special education teachers who received invitations to the survey because in these states, requests to survey teachers go to the special education directors, who then choose whether to forward requests to all or some of the state’s special education teachers. With regard to these two states, researchers also could not determine whether nonresponding special education teachers had received a second “reminder” invitation.

Instrument

The survey had two sections. The first section covered demographic information about the teacher (e.g., teaching experience in reading and math, classroom setting, and school setting) and his or her familiarity with and perceived clarity of his or her state’s eligibility guidelines via three open-ended questions:

Are you familiar with the state eligibility guidelines?

Have you had test-type assignment training?

Are the state eligibility guidelines clear in helping you and IEP teams decide assessment test-type assignments?

The second section presented the vignettes describing (a) the characteristics of fictional students, such as their disability category, IQ range, cognition, and emotional and behavioral issues; (b) the quality of instruction the students received, such as classroom support, accommodations, modifications, and classroom materials; (c) the students’ present level in reading and math; and (d) the students’ academic performance in the classroom, on school district assessments, and on the previous year’s statewide assessment. Each vignette was followed by six questions regarding the math and reading test-type decisions the teacher would make for the fictional students. The following three questions would be asked concerning the math assessment decision and the same set of questions would be asked concerning the reading decision:

If you were Samuel’s teacher, which statewide math assessment would you recommend his IEP team assign him to?

What are the factors you would consider in deciding the math assessment for Samuel? IQ score; previous year’s statewide assessment performance/test type; disability category; student’s cognitive, behavioral, emotional characteristics; 1 or 2% AYP caps; use of adaptations/modifications; read-aloud accommodation policy; performance on formative assessment and school district assessments, and so on.

Please rank the factors you selected (1 for highest priority, 2 for second highest, 3 for third highest, etc.).

The second question in each set presented 14 factors that special education teachers might consider in their decision making, and the third question asked teachers to rank the factors they checked in the second question.

Data Analysis

Data were analyzed qualitatively and quantitatively for each state separately because the states have different alternate assessment systems and eligibility guidelines. Open-ended responses concerning the special education teachers’ perceptions of the clarity of their state’s alternate assessment eligibility guidelines were analyzed qualitatively and coded according to themes. A researcher and two research assistants reviewed the responses and coded them independently. Four themes emerged: (a) a clear, positive statement (yes); (b) a clear, negative statement (no); (c) a mixed response (mostly, somewhat, in general); and (d) an unspecified opinion (statement of personal experience or opinion rather than yes, no, or mixed). There were some discrepancies between the primary researcher and the two research assistants regarding the following: Some special education teachers said, in effect, “the guidelines for the state are clear, but there is a problem when each district comes up with additional guidelines to follow,” or “the AA-AAS guidelines are open to interpretation.” Responses that could be interpreted as either a “yes” or a “no” were categorized as “mixed” responses. There was 97.23% agreement among the coders and disagreements were resolved through discussion.

The fourth theme of survey responses was labeled “unspecified opinion” because the responses were indeterminate as to whether the statement reflected the opinion of the special education teachers: “We were told it is not the IEP team’s decision to make, we must go off the child’s IQ,” “We leave testing issues to the judgment of our school psychologists,” or “We are told that testing options are based on individual needs, that even though we have guidelines to follow, sometimes there are cases where a parent refuses to put his or her child on an appropriate assessment and at those times, LEAs [local educational agencies] support the parent.” Once the four themes were established, the researcher and research assistants categorized special education teachers’ responses and descriptive statistics were used to summarize the results.

Data were analyzed using the statistical program SPSS^® version 20. Descriptive statistics (i.e., means, standard deviations, and frequency tabulations) were computed to summarize educators’ responses to questions about which reading and math assessment types they would assign the students in the vignettes, what factors they would consider in making the assignment decisions, and the ranked order of importance of these factors.

Results

Results of this study will be presented in the following sequence: (a) educators’ perceptions of the familiarity and clarity of their state alternate assessment eligibility guidelines and the extent of their training on test-type assignments; (b) which math and reading assessments the special education teachers chose to assign to each of the students in the vignettes; (c) what factors educators considered in making the assignment decisions; and (d) the rank in order of importance of these factors.

Familiarity and Clarity of Guidelines, Extent of Assignment Training

The study showed that 98.21% of the special education teachers were familiar with their state eligibility guidelines, and 99% of the teachers said that they had received test-type assignment training. Special education teachers were asked, “In general, are the state eligibility guidelines clear in helping the IEP team decide on the test-type assignment?” Responses were summarized qualitatively, and four themes emerged. These were (a) a straightforward “yes”; (b) a straightforward “no”; (c) a “mixed response,” such as one educator’s statement that the state eligibility guidelines were clear and helpful to a certain degree but not sufficient to cover all students; and (d) an “unspecified opinion,” a response tangential to the question. Responses varied moderately by state (Table 1). For instance, 55.1% of the Midwestern educators stated that the eligibility guidelines were clear, while only 40.4% and 38.8% of the other educator groups (in the West and the Northeast, respectively) reported clarity in the guidelines.

Table 1.

Examples of Survey Respondents’ Comments to Open-Ended Question, “In General, Are the State Eligibility Guidelines Clear in Helping the IEP Team Decide on the Test-Type Assignment?”

Theme of response	State	Examples of respondents’ comments
Response includes a clear statement of “yes”	Midwest 55.1% (n = 70)	“Yes, they seem to be very clear on where kids should be placed.”
	West 40.4% (n = 57)	“The state eligibility guidelines are clear. I just wish there was a middle option for the students that don’t completely meet the criteria in the guidelines, but are still scoring below grade level in one or two areas.”
	Northeast 38.8% (n = 19)	“Yes, as clear as they can be, given the unique nature of each child.”
Response includes a clear statement of “no”	Midwest 22.8% (n = 29)	“No, my experience is that the interpretation varies on the individual special education teacher.”
	West 33.5% (n = 47)	“No, I don’t believe so. Adaptive skills are considered, but to what extent?”
	Northeast 30.5% (n = 15)	“No, there should be an option for students with significant behavioral concerns to access the AA-AAS.”
A mixed response is given, such as “mostly,” “somewhat,” or “in general”	Midwest 6.3% (n = 8)	“Mostly. I feel there are some students that just don’t fit the guidelines so it is hard to make a decision.”
	West 12.1% (n = 17)	“I feel that the guidelines are somewhat clear, however, I feel that each student can present such diversity that some do not fit the guidelines and assessment needs to be looked at in a different manner altogether.”
	Northeast 20.4% (n = 10)	“Somewhat, but they are still subject to interpretation.”
Response is “unspecified opinion,” respondents give an opinion or personal experience rather than a “yes” or “no”	Midwest 15.7% (n = 20)	“The gap between qualifying for Alternate Assessment and [AA-MAS] is too great. Many of the students that I have worked with in SPED fall through this gap. I service students that have some skills in reading and math but are not able to function in the regular classroom.”
	West 14.2% (n = 20)	“I do have students who don’t perform well in IQ tests or state test—their minds work well, but bodies don’t connect. I don’t know how clear guidelines are for students who are intelligent, but fail to perform on cue.”
	Northeast 10.2% (n = 5)	“The IEP team does not decide. The student’s placement determines which tests are used for assessments.”

Note. Special education teachers from the Midwestern state made up 40% (n = 127) of the 317 respondents, the Western state made up 44.4% (n = 141), and the Northeastern state made up 15.4% (n = 49). AA-AAS = alternate assessment–alternate achievement standards; AA-MAS = alternate assessment–modified achievement standards. SPED = special education.

Vignette 1: Samuel

Vignette 1 concerned Samuel, a sixth-grade student with a learning disability (Table 2). The differing alternate assessment systems used in the state the special education teachers taught in seemed to affect their test-type decisions. In the Western state, which did not have an AA-MAS option, 8.5% of teachers assigned Samuel to the math AA-AAS, but none of the teachers from the Northeastern and Midwestern states made this assignment. In addition, 70.1% of educators from the Midwestern state, 59.6% from the Western state, and 73.5% from the Northeastern state assigned Samuel to the general math assessment with accommodation, while 11.8% and 16.3% of the educators in the Midwestern and Northeastern states (respectively) assigned him to the AA-MAS. For the reading assessment, 79.4% of the teachers from the Midwestern state and 83.7% of those from the Northeastern state assigned Samuel to the AA-MAS reading assessment, and 8.7% of those from the Midwestern state, 47.5% from the Western state, and 2% from the Northeastern state assigned him to the AA-AAS.

Table 2.

Survey Vignette 1: Samuel’s Background Information.

Category	Description
Present grade level
Math	At grade level
Reading	4th or 5th grade level
Performance scores (General assessment, 2010)
Math	Proficient
Reading	Less than proficient
School district
Math	50th or 60th percentile
Reading	1st percentile
Classroom
Math	Has para-professional support
Reading	Doing very well because the reading quizzes he took were not at his grade level
Classroom accommodations	Read aloud, extended time
Classroom modifications	4th or 5th grade reading materials, a reduced number of classroom quiz items
Classroom setting
Math	General classroom
Reading	50% resource room and general classroom
Characteristics
Cognitive	IQ below average, above grade-level recall, long-term, and auditory memory
Behavioral	Distractible, wanders off task very easily
Emotional	Easily frustrated
Social	Good social skills, gets along with his peers
Family life	No concerns

The factors that special education teachers considered when deciding which test assignment was appropriate for Samuel were similar in frequency across the three states (Table 3), but there were differences in the ranking of these factors based on the state. For teachers of all three states, Samuel’s present level/learning progress was the most important consideration. Formative assessments/classroom quizzes were second in importance for teachers from the Western and Northeastern states, and classroom modifications/adaptations were ranked second by Midwestern teachers. Samuel’s scores on the previous year’s assessment were third in importance for Midwestern teachers and fourth for Northeastern teachers. IQ score was ranked fourth and disability category fifth by special education teachers from the Western state. For the reading assessment, again, present level/learning progress was the most important consideration for teachers in all three states. Formative assessments/classroom quizzes were ranked second by teachers from the Western and Northeastern states. Need for intensive instruction was ranked second by teachers from the Midwestern state, third by teachers from the Western state, and fourth by teachers from the Northeastern state. The 1% and 2% AYP cap were a concern only to teachers from the Northeastern state and were ranked fifth.

Table 3.

Frequency of Factors Most Affecting Test-Type Decisions.

		Midwestern State		Western State		Northeastern State
Test	Factor	n	%	n	%	n	%
Samuel, math	Present level/learning progress	113	89.0	126	89.4	46	93.9
	Scores on previous year’s state assessment	103	81.1	90	63.8	40	81.6
	Classroom modifications/adaptations	102	80.3	93	66.0	40	81.6
Samuel, reading	Present level/learning progress	119	93.7	129	91.5	46	93.9
	Scores on previous year’s state assessment	104	81.9	90	63.8	38	77.6
	Classroom modifications/adaptations	113	89.0	114	80.9	41	83.7
John, math	Present level/learning progress	117	92.1	118	83.7	43	87.8
	Scores on previous year’s state assessment					37	75.5
	Classroom modifications/adaptations	108	85	105	74.5	37	75.5
	Behavioral characteristics	88	69.3	105	74.5
John, reading	Present level/learning progress	118	93.7	116	82.3	46	93.9
	Scores on previous year’s state assessment					37	75.5
	Classroom modifications/adaptations	106	84.1	103	73	38	77.6
	Behavioral characteristics	90	71.4	104	73.8
	Classroom setting			78	55.3
Susan, math	Present level/learning progress	116	91.3	121	85.8	48	98
	Scores on previous year’s state assessment	104	81.9			36	73.5
	Classroom modifications/adaptations	108	85	101	75.9	41	83.7
	Formative assessments/classroom quizzes			83	58.9
Susan, reading	Present level/learning progress	115	90.6	119	84.4	48	98
	Scores on previous year’s state assessment	107	84.3	81	57.4	36	73.5
	Classroom modifications/adaptations	105	82.7	97	68.8	41	83.7

Vignette 2: John

In vignette 2, John is described as a sixth-grade student with an emotional/behavioral disorder (Table 4). The majority of Midwestern teachers (53.5%) and Northeastern teachers (65.3%) chose to assign John to the AA-MAS math assessment, while the majority of Western teachers (67.4%) chose the AA-AAS math assessment. For the reading assessment, the majority of Midwestern teachers (62.2%) and Western teachers (80.1%) chose the AA-AAS, whereas the majority of Northeastern teachers (63.3%) chose the AA-MAS.

Table 4.

Survey Vignette 2: John’s Background Information.

Category	Description
Present grade level
Math	3rd grade level
Reading	2nd grade level
Performance scores (General assessment, 2010)
Math	Proficient
Reading	More than proficient
School district
Math	20th percentile
Reading	1st percentile
Classroom
Math	Good scores (receives an “A” using 3rd grade materials)
Reading	Good scores (receives an “A” using 1st grade materials)
Classroom accommodations	Read aloud, hand script
Classroom modifications	Reduced number of classroom test items and assignments
Classroom setting
Math	Self-contained or resource room due to his problem behaviors
Reading	Self-contained or resource room due to his problem behaviors
Characteristics
Cognitive	IQ below average
Behavioral	Distractible, problem behaviors (animal sounds, noise, running in hallway)
Emotional	Becomes frustrated with challenging material, runs away or acts out
Social	A loner
Family life	No concerns

Of the 14 factors teachers could select as considerations when making assessment decisions for John, the number one factor across all three states was present level/learning progress for reading and math (Table 3). Classroom modifications/adaptations were a commonly cited factor by teachers in the two states where AA-MAS was implemented (Midwestern and Northeastern), and behavioral characteristics were cited by teachers from Western and Midwestern states. With regard to the top five factor rankings for the math assessment, special education teachers in all three states ranked John’s present level/learning progress as the most important factor and his behavioral characteristics as the second. John’s IQ score ranked third for the Western and Northeastern teachers and need for intensive instruction was ranked third by Midwestern teachers. John’s disability category ranked fifth in importance for teachers in the Northeastern state. For the reading assessment decision, special education teachers from all three states considered present level/learning progress as the most important factor. Behavioral characteristics were the second most important factor for teachers from the Midwestern and Western states. Formative assessments/classroom quizzes were ranked third by Western and Northeastern teachers, while John’s need for intensive instruction was third for Midwestern teachers.

Vignette 3: Susan

As shown in Table 5, Susan is described as an eighth-grade student with a learning disability. For the math assessment, 73.2% of the teachers in the Midwestern state and 71.4% of those in the Northeastern state assigned Susan to the AA-MAS. In the Western state, 57.4% of the teachers assigned her to the general assessment with accommodation and 36.9% assigned her to the math AA-AAS.

Table 5.

Survey Vignette 3: Susan’s Background Information.

Category	Description
Present grade level
Math	4th grade level
Reading	7th grade level
Performance scores (General assessment, 2010)
Math	Less than proficient
Reading	Proficient
School district
Math	Below average
Reading	Below average
Classroom
Math	B or C student with para-educator support
Reading	B or C student with para-educator support
Classroom accommodations	Read aloud, separate setting
Classroom modifications	Para-educator support in math and reading, reduced number of classroom test items and assignments
Classroom setting
Math	Resource room (with pull out)
Reading	General education
Characteristics
Cognitive	IQ average, poor memory retention
Behavioral	No concerns
Emotional	Easily scared, high anxiety level, often dwells on family situation and cannot think in class
Social	Easily loses control
Family life	Single-parent household, tough home life, parent is very ill

No special education teacher in the Midwestern and Northeastern states assigned Susan to the reading AA-AAS, but a small percentage of Western teachers (7.1%) did make this assignment. The majority of teachers in all the three states assigned Susan to the general reading assessment with accommodation: 80.3% of the Midwestern teachers, 78.7% of the Western teachers, and 71.4% of the Northeastern teachers.

Of the 14 factors teachers could select as considerations when making assessment decisions, the most frequently cited among teachers in all three states when deciding Susan’s math test assignment were her present level/learning progress and classroom modifications/adaptations in math (Table 3). Special education teachers in the Midwestern and Northeastern states also considered Susan’s scores on the previous year’s state assessment, but educators in the Western state cited her formative assessments/classroom quizzes. For the reading assessment, the most frequently cited factors of teachers from all the three states were her present level/learning progress, scores on the previous year’s state assessment, and classroom modifications/adaptations. For special education teachers of all three states, Susan’s present level/learning progress was ranked as the most important consideration in deciding her math assessment. Special education teachers in the Northeastern and Western states gave second and third highest priority to formative assessments/classroom quizzes, but the Midwestern state ranked need for intensive instruction second. Teachers in the Western state considered Susan’s emotional characteristics the second most important factor, while this factor ranked third for teachers in the Northeastern state. Score on the previous year’s math assessment was not in the top five factors for the Western teachers.

For the reading assessment, once again present level/learning progress was the top-ranked factor for teachers of all three states. Formative assessments/classroom quizzes were second for teachers from the Western and Northeastern states and third for those of the Midwestern state. Susan’s scores on the previous year’s reading assessment were the second and fourth most important factors for the Midwestern and Northeastern states, respectively. Disability category was important (fourth) to teachers in the Western state.

Discussion

The purpose of this study was to explore the effect of diverse alternate assessment systems and eligibility guidelines on the test-type decisions made for students with disabilities in the three states. The findings are consistent with the concerns of previous studies on the differences among eligibility guidelines across states (Thurlow, 2004), the lack of specificity in using students’ general or instructional characteristics (Cho & Kingston, 2011; Filbin, 2008; Musson et al., 2010; Thurlow, 2004), and the vulnerability of the eligibility guidelines to subjective judgment or interpretation (Cho & Kingston, 2012).

The students in this vignette survey had mild disabilities (e.g., learning disabilities, emotional and behavioral disorders). Even so, some special education teachers assigned these students to the AA-AAS. This finding is similar to that of a recent case study: “Why IEP Teams Assign Low Performers with Mild Disabilities to the Alternate Assessment Based on Alternate Achievement Standards” (Cho & Kingston, 2012). This study found that special educators of students with other health impairments, learning disabilities, and emotional behavioral disorders believed that their students qualified as having a “most significant cognitive disability” because they needed adaptations/modifications or intensive instruction, or had different information processes, poor academic performance, slow response, and so on. Technically, these students did not have a most significant cognitive disability. These special educators were unable to distinguish mild disability from the most significant cognitive disabilities because of a lack of knowledge. The participants of the present study are similarly lacking in that knowledge.

This survey extends the current research in this area. Insufficient knowledge of the definition of most significant cognitive disabilities can cause inappropriate assignment of students with mild disabilities to the Alternate Assessment System. The probability of the three students with mild disabilities being assigned to AA-AAS and treated as students with significant cognitive disabilities was dependent on the Alternate Assessment System they were in. Special education teachers in the Western state, where the AA-MAS had not been implemented, freely assigned the three students with mild disabilities to the AA-AAS, unlike their colleagues in the other two states. For Samuel’s math assessment, 0%, 8.5%, and 0% of the teachers assigned him to the AA-AAS in the Midwestern, Western, and Northeastern states, respectively. For Samuel’s reading assessment, 11%, 47.5%, and 2% of the teachers assigned him to the AA-AAS in the Midwestern, Western, and Northeastern states, respectively. John’s assignment to the math AA-AAS was made by 28%, 67%, and 20% of the teachers, while his assignment to the reading AA-AAS was made by 62%, 80%, and 37% of the teachers in the Midwestern, Western, and Northeastern states, respectively. Susan was assigned to the math AA-AAS by 8%, 37%, and 4%, and to the reading AA-AAS by 0%, 7%, and 0% of the teachers in the Midwestern, Western, and Northeastern states, respectively. This phenomenon reflects the many problems in diagnosing specific learning disabilities that the ability-achievement discrepancy and the response to intervention have, including their lack of empirical support (Flanagan, Ortiz, & Alfonso, 2008; Ofiesh, 2006; Reynolds & Shaywitz, 2009) and the potential to vary greatly from district to district and across states (Machek & Nelson, 2010).

Special education teachers’ responses to open-ended questions about the clarity of eligibility guidelines should give researchers, policymakers, and the field reason to consider using disability category and IQ scores as factors to identify students with the most significant cognitive disabilities and prevent students with nonintellectual significant physical disabilities or mild disabilities from being categorized as having significant cognitive disabilities. For example, several studies have reported that students with the most significant cognitive disabilities participating in the AA-AAS experience notable difficulties in expressive and receptive communication and often require augmentative/alternative communication systems (Almond & Bechard, 2005; Kearns et al., 2011; Towles-Reeves, Kearns, Kleinert, & Kleinert, 2009). However, these indicators also accurately describe some students with orthopedic impairments; expressive and receptive language disorders are commonly observed among individuals with cerebral palsy who do not have intellectual disabilities (Pellegrino, 2002). In addition, students with significant physical disabilities such as orthopedic impairment often have significant limitations to their adaptive skills.

Special education teachers in the current study had mixed approaches in considering IQ and adaptive skills. Some explained that low functional/adaptive skill was not a sufficient reason to place a student with high IQ in the AA-AAS. Others considered IQ scores unreliable indicators, or believed that low functional/adaptive skill mediated IQ to the point that such a student should be assigned to the AA-AAS. One educator complained that the eligibility guidelines were not clear on how to assign students who were intelligent but failed to perform. There has been considerable debate on the usefulness of IQ scores in evaluating students’ intellectual ability, although scholars and policymakers have been discouraged from using IQ scores for this purpose (Ofiesh, 2006; Reynolds & Shaywitz, 2009). In some cases, IQ scores are necessary to differentiate students with mild disabilities from those with significant cognitive disabilities. However, it would be problematic for IQ score to be the sole factor in deciding the most significant cognitive disabilities because intellectual abilities are not valid measures of intellectual potential for all students.

Almost half of the special education teachers responded that eligibility guidelines were clear and helpful, whereas many others raised concerns about placing all students with disabilities into appropriate test-type assignments based on the current eligibility guidelines. Therefore, it is not surprising that there was little unanimity in the test-type assignments for the students in the vignettes. When making test-type assignments for their own students, special education teachers in all the three states revealed that they considered factors that are not included in eligibility guidelines or exclusionary criteria (e.g., the 1 or 2% caps, restricted read-aloud accommodation, emotional characteristics, behavioral characteristics, or classroom settings). This finding is similar to the findings in a case study by Cho and Kingston (2012).

As reported, there are inappropriate political reasons (e.g., the 1 or 2% caps or the read-aloud accommodation policy), administrative pressures, and a lack of knowledge of the definition of significant cognitive disabilities that lead to inappropriate assessment assignments. The field should define and standardize the intended population for the AA-AAS. In addition, policymakers should rethink the current system, which leaves each state to formulate its own eligibility guidelines, and instead encourage states to adopt common terminology. However, if the guidelines rely on instructional and general characteristics (e.g., alternate achievement standards, intensive instruction, significant cognitive disabilities, adaptive skills), common terminology in the guidelines would still be insufficient for IEP teams to identify students with the most significant cognitive disabilities. The use of general and instructional characteristics will still result in IEP teams making subjective interpretations. It would be desirable to use explicit descriptors in the guidelines to specify the appropriate student characteristics for the intended population. Students with significant cognitive disabilities are a heterogeneous group, and it would be very challenging to specify the appropriate student characteristics for the intended population within the scope of one paper on eligibility guidelines.

Limitations

Although this study increases our knowledge, it is important to note two caveats. Two states were unable to apprise researchers of the precise number of special education teachers who received invitations to the survey. Therefore, researchers could not accurately determine the response rate. In addition, one state’s response rate was low (40%); therefore, our findings may not be representative of all special education teachers. Recent studies indicate that a low response rate does not guarantee lower survey accuracy and instead simply indicates a risk of lower accuracy. For example, in their examination of whether lower response rates were associated with less unweighted demographic representativeness of a sample, Holbrook, Krosnick, and Pfent (2007) found that lower response rates were only minimally less accurate. These researchers assessed the results of 81 national surveys with response rates between 5% and 54%.

There is some evidence that demonstrates that web-based surveys can be hampered by limited access for certain demographic groups, which restricts generalizability (Skitka & Sargis, 2006). Participants in this study may have struggled with the test-type decision more than other special educators would, so this sample of participants may not represent the population. In addition, the present study covered only three states, so the findings may not generalize to all other states. Future research should survey more states to ascertain whether these findings apply to special education teachers in other states as well.

Implications and Practice

Because the AA-MAS option will soon be phased out, low achievers with mild disabilities will again face increased likelihood that they will be assigned to the AA-AAS unless we refine eligibility guidelines. It is important to address the need for a new, systematic approach to identifying the intended population for the AA-AAS because inconsistencies in the current test-type assignment system hinder efforts to achieve better educational outcomes for all students with disabilities. To improve the current system, it is important to make unified, objective, and scientifically based eligibility guidelines across the 50 states. One way to achieve this is by introducing the use of probabilistic decision-tree modeling as a tool to be used by IEP teams. The probabilistic decision tree is developed using active learning algorithms based on expert judgments concerning the characteristics of students with the most significant cognitive disabilities, and on extensive literature regarding the key factors: (a) disability category, (b) IQ level, (c) communication level, (d) capability of communication devices in academic areas, and (e) other related student characteristics that differentiate nonintellectual disability and mild disability from the most significant cognitive disability.

Assessment type and assessment outcomes guide instruction and determine curriculum. Inappropriate assignment of students with mild disabilities to the AA-AAS threatens the opportunities of these students to learn their grade-level academic content.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research and/or authorship of this article.

References

Almond

Bechard

(2005). In-depth look at students who take alternate assessments: What do we know now? (Report prepared by Colorado Department of Education and Measured Progress). Retrieved from http://www.measuredprogress.org/c/document_library/get_file?uuid=412baede-489b-46d3-a679-ca9a0b9cd4a0&;groupId=10157

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Cho

H-J.

Kingston

(2011). Capturing implicit policy from NCLB test type assignments: Analysis of characteristics and performance of students taking alternate assessments based on modified achievement standards. Exceptional Children, 78, 58-72.

Cho

H-J.

Kingston

(2012). Why IEP teams assign low performers with mild disabilities to the alternate assessment based on alternate achievement standards. The Journal of Special Education. Advance online publication. doi:10.1177/0022466911435416

Filbin

(2008). Lessons from the initial peer review of alternate assessments based on modified achievement standards. Washington, DC: U.S. Department of Education, Office of Elementary and Secondary Education, Student Achievement and School Accountability Program.

Flanagan

D. P.

Ortiz

S. O.

Alfonso

V. C.

(2008). Response to intervention (RTI) and cognitive testing approaches provide different but complementary data sources that inform SLD identification. NASP Communique, 36(5), 16-17.

Holbrook

Krosnick

Pfent

(2007). The causes and consequences of response rates in surveys by the news media and government contractor survey research firms. In Lepkowski

J. M.

. (Eds.), Advances in telephone survey methodology (pp. 499-678). Hoboken, NJ: Wiley.

Individuals with Disabilities Education Act Amendments of 1997, Pub. L. No. 105-117, 20 U.S.C. § 1400 et seq.

10.

Individuals with Disabilities Education Improvement Act of 2004, Pub. L. No., 108-446, 20 U.S.C. § 1400 et seq.

11.

Kearns

J. F.

Towles-Reeves

Kleinert

H. L.

Kleinert

J. O.

Thomas

M. K-K.

(2011).Characteristics of and implications for students participating in alternate assessments based on alternate academic achievement standards. The Journal of Special Education, 45, 3-14.

12.

Machek

G. R.

Nelson

J. M.

(2010). School psychologists’ perceptions regarding the practice of identifying reading disabilities: Cognitive assessment and response to intervention considerations. Psychology in the Schools, 47, 230-245.

13.

Musson

J. E.

Thomas

M. K.

Towles-Reeves

Kearns

J. F.

(2010). An analysis of state alternate assessment participation guidelines. The Journal of Special Education, 44, 67-78.

14.

No Child Left Behind Act of 2001, Pub. L. No., 107-110, 20 U.S.C. § 6301 et seq.

15.

Ofiesh

(2006). Response to intervention and the identification of specific learning disabilities: Why we need comprehensive evaluations as part of the process. Psychology in the Schools, 43, 883-888.

16.

Pellegrino

(2002). Cerebral palsy. In Batshaw

M. L.

(Ed.), Children with disabilities (pp. 443-467). Baltimore, MD: Paul H. Brookes.

17.

Quenemoen

(2009). Identifying students and considering why and whether to assess them with an alternate assessment based on modified achievement standards. In M. Perie (Project Manager), Considerations for the alternate assessment based on modified achievement standards (AA-MAS): Understanding the eligible population and applying that knowledge to their instruction and assessment (pp. 17-50). New York: Comprehensive Center, New York State Education Department.

18.

Reynolds

C. R.

Shaywitz

S. E.

(2009). Response to intervention: Ready or not? Or, from wait-to-fail to watch-them-fail. School Psychology Quarterly, 24, 130-145.

19.

Skitka

L. J.

Sargis

E. G.

(2006). The internet as psychological laboratory. Annual Review of Psychology, 57, 529-555.

20.

Thompson

S. J.

Erickson

Thurlow

M. L.

Ysseldyke

J. E.

Callender

(1999). Status of the states in the developmentof alternate assessments (Synthesis Report No. 31). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://www.cehd.umn.edu/NCEO/OnlinePubs/archive/Synthesis/Synthesis31.html

21.

Thompson

S. J.

Thurlow

M. L.

(2000). State alternate assessments: Status as IDEA alternate assessment requirements take effect (Synthesis Report No. 35). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://www.cehd.umn.edu/NCEO/OnlinePubs/archive/Synthesis/Synthesis35.html

22.

Thompson

S. J.

Thurlow

M. L.

Johnstone

C. J.

Altman

J. R.

(2005). 2005 State special education outcomes: Steps forward in a decade of change. Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/2005StateReport.htm

23.

Thurlow

M. L.

(2004). How state policies and practices for alternate assessment impact who is included in NAEP state assessments. National Center on Educational Outcomes. Minneapolis: University of Minnesota. Retrieved from http://www.nagb.org/publications/conferences/thurlow.pdf

24.

Towles-Reeves

Kearns

Kleinert

H. L.

Kleinert

J. O.

(2009). An analysis of the learning characteristics of students taking alternate assessments based on alternate achievement standards. The Journal of Special Education, 42, 241-254.

25.

U.S. Department of Education. (2005). Alternate achievement standards for students with the most significant cognitive disabilities: Non regulatory guidance. Washington, DC: Office of Elementary and Secondary Education, U.S. Department of Education.

26.

Warlick

Olsen

(1999). Who takes the alternate assessment? State criteria. Lexington: University of Kentucky.