State Systemic Improvement Planning: Impact on System and Student Outcomes

Abstract

In the wake of reports of continued gaps between youth with and without disabilities in regard to graduation rates and postschool outcomes, the U.S. Department of Education (USDOE), Office of Special Education Programs (OSEP) began examining their process for monitoring state implementation of the Individuals with Disabilities Education Improvement Act (IDEA). OSEP’s revised accountability system, known as Results-Driven Accountability, better aligns accountability systems to support states in improving results for infants, toddlers, and youth with disabilities and their families. Currently required from states, is a comprehensive multiyear State Systemic Improvement Plan (SSIP) focused on improving results for students with disabilities. The purpose of this article is to describe the phases of the SSIP and provide an example of how this new accountability system is working in the Republic of the Marshall Islands (RMI). As with all other states, territories, and freely associated states, RMI is required to develop and implement an SSIP.

Keywords

collaboration policy school reform/restructuring system(s) change transition

The purpose of the Individuals with Disabilities Education Improvement Act (IDEA, 2004) is

to ensure that all children with disabilities have available to them a free appropriate public education (FAPE) that emphasizes special education and related services designed to meet their unique needs and prepare them for further education, employment, and independent living. (d)(1)(A)34 CFR §300.1(a)

In an effort to better understand the in-school and postschool experiences (i.e., education, employment, and independent living) of youth with disabilities, the U.S. Department of Education (USDOE) contracted with Stanford Research International (SRI) to conduct the first longitudinal study of youth with disabilities (i.e., National Longitudinal Transition Study [NLTS]; Wagner et al., 1993). Youth ages 13 to 21 years were followed for 5 years, and data were collected on such factors as (a) individual and family characteristics (e.g., demographics); (b) independent functioning (e.g., financial independence and functional abilities); (c) social experiences (e.g., socializing with friends and belong to social groups); (d) employment experiences (e.g., rates of employment types of jobs and wages); and (e) postsecondary education (i.e., participation in vocational schools, 2-year, 4-year colleges). Results of the study were eye opening. After 4 years of high school, youth with disabilities were earning half of the required credits needed for graduation. Large numbers of youth with disabilities (i.e., 38%) were dropping out of high school. Of the 67% of youth who did graduate, few (less than one third) went on to postsecondary education, approximately 57% were competitively employed, and 37% lived independently (Wagner et al., 1993).

Since 1993, in-school and postschool outcomes for students with disabilities continue to lag behind those of their peers without disabilities. In academics, recent results from the 2015 National Assessment of Educational Progress (NAEP) indicated that in 12th-grade math, only 5% of students with disabilities were proficient (compared to 23% of students without disabilities), while an additional 17% were at the basic level (compared to 39% of all students). In 12th-grade reading, 10% were proficient and 26% were at the basic level, compared to 33% and 36% of youth without disabilities, respectively. As a result, it is not surprising that the national graduation rate for students with disabilities is only 63%, while it is about 83% for students without disabilities (Institute for Education Sciences, National Center for Education Statistics [IES, NCES], 2017). In fact, in recent data from the USDOE summaries of State Performance Plan/Annual Performance Report (SPP/APR) Indicators B-1 (graduation rates) and B-2 (drop-out rates) reported in 2016 (for federal fiscal year [FFY] 2014), states reported graduation rates for students with disabilities ranging from approximately 25% to 85% (M = 62.1%) and dropout rates from 0.5% to 22.7% (M = 4.6%). While these percentages are trending in the right direction, positive school completion rates continue to lag behind those of students without disabilities. Given these dismal data, it is not surprising that the postschool outcome data (i.e., enrolled in higher education, competitively employed, enrolled in other education, or employed in other employment) for students with disabilities continue to remain poor (77.5%; USDOE, 2016), and they indicate gaps between students with and without disabilities in postsecondary education and employment (Newman et al., 2011).

The U.S. Department of Labor (USDOL; 2017) indicated that only 17.9% of individuals with disabilities were employed compared to 65.3% of individuals without disabilities. In a national survey conducted by the Institute for Community Inclusion, findings indicated that only 50% of youth with disabilities attend 4-year colleges or universities, 40% attend 2-year colleges, and 10% attend trade/technical schools (National Council on Disability, 2012). According to results of the latest longitudinal study, NLTS 2012, Lipscomb et al. (2017) reported that many youth with disabilities are still not taking appropriate steps to prepare for college and careers.

In the wake of reports of continued gaps between youth with and without disabilities in regard to graduation rates and postschool outcomes, the USDOE Office of Special Education Programs (OSEP) began examining its process for monitoring state implementation of the IDEA (2004). Prior to 2012, states reported progress using quantifiable indicators in priority areas such as (a) provisions of Free, Appropriate Public Education (FAPE) and Least Restrictive Environment (LRE); (b) Child Find, effective monitoring, use of resolution sessions, mediation, voluntary binding arbitration, and system of transition services; and (c) disproportionate representation caused by inappropriate identification of racial and ethnic groups in special education and related services. Included under IDEA part B were 20 indicators for which data were collected annually. These indicators were an accountability measure and could be considered a “blueprint” for state improvement (Smith, 2017). The first eight indicators monitored FAPE in the LRE. Indicators 9 and 10 monitored disproportionality. Indicator 11 monitored effective general supervision of Child Find. Indicators 12 through 14 monitored general supervision of effective transition. Indicators 15 through 19 monitored general supervision of dispute resolution. The remaining indicator monitored data collection and submission (OSEP, 2009).

This former OSEP accountability system collected data on states’ compliance with procedural requirements (e.g., Participation in Statewide Assessment [Indicator 3]; LRE [Indicator 5]), results (e.g., graduation [Indicator 1], and drop-out [Indicator 2]); however, states were identified as needing different levels of assistance based on compliance alone (OSEP, 2012; USDOE, 2014). As a result of its ongoing work with states, OSEP noted significant improvements in compliance rates over the years but recognized that educational outcomes of students with disabilities (e.g., reading and math performance, graduation rates, and postschool outcomes) were not improving at the same rates (USDOE, 2014). In preparation for a more focused accountability system that emphasized improving the outcomes of youth with disabilities, OSEP worked with states to improve data timelines, data quality, and compliance. This change in accountability systems, now known as Results-Driven Accountability (RDA), raised expectations for special education, a need that was long overdue.

Results-Driven Accountability

The RDA system is an effort led by OSEP to better align accountability systems to support states in improving results for infants, toddlers, and youth with disabilities and their families (OSEP, 2012). RDA aligns with missions of the USDOE Office of Special Education and Rehabilitative Services and is based on the premise that the results states want to achieve drive the actions that states take. With RDA, USDOE committed to providing the support states, districts, schools, principals, and teachers needed to improve the academic and functional achievement of youth with disabilities. USDOE developed seven core principles to guide the work: (a) partner with stakeholders, (b) transparency, (c) drive to improve results, (d) protection of children and families, (e) differentiated incentives and supports to states, (f) target resources and reduction of burden, and (g) responsive to needs (Ryder, 2015). The new accountability system included 17 Part B (i.e., Special Education) indicators in which states would collect data and include in their SPP/APR. Now included in the SPP/APR is a comprehensive multiyear State Systemic Improvement Plan (SSIP, Part B Indicator 17) focused on improving results for students with disabilities.

State System Improvement Plan

SPP/APR Part B Indicator 17 is different from the other 16 indicators of compliance and results designed to measure whether states are carrying out the intent of IDEA (2004). Part B Indicator 17 is a multiyear plan focused on improving results for students with disabilities (Corr, 2015). The SSIP consists of three phases and follows a “plan, do, study, act” model (see Figure 1; Park et al., 2013). An important aspect of the SSIP is that it is guided by stakeholder (e.g., school personnel, families, youth with disabilities, and community members) feedback across all phases. This practice-led approach of the SSIP, in theory, should improve implementation, sustainability, and dissemination of results. According to Candy (2006), “practice-led research is concerned with the nature of practice and leads to new knowledge that has operational significance for that practice,” (p. 1); in this case, provision of special education services for youth with disabilities. Practice-led research is carried out by practitioners to develop generalizable knowledge to inform the field. To this end, states must describe how they involve stakeholders in developing, implementing, and evaluating the SSIP.

Figure 1.

Plan, Do, Study, Act model (Republic of Marshall Islands [RMI] Public School System [PSS], 2017, p. 2).

Phase I

Phase I of the SSIP completed in federal fiscal year (FFY) 2013 aligned with the “Plan” portion of the plan, do, study, act model, and focused on data analysis and identifying evidence-based practices (EBPs). In Phase I, states and their stakeholder groups were charged with identifying a State-Identified Measurable Result (SIMR) for children with disabilities. To do this, states and their respective stakeholder groups analyzed key data to identify root causes contributing to low performance of youth with disabilities. Key data included data from previous SPP/APRs and IDEA section 618 data collections, among other data available in the state (e.g., Vocational Rehabilitation and Carl Perkins). This process involved multiple meetings in each state to present, review, and analyze these data. Once the SIMR was identified, states then conducted an analysis of state policies, procedures, communication systems, professional capacity, organizational structure, and other infrastructure components to determine the capacity of the state, district, and schools to scale-up and sustain the use of EBPs to improve in-school and postschool outcomes of youth with disabilities. States were required to analyze infrastructure to determine if improvements were needed in systems of governance, fiscal oversight, quality standards, professional development/technical assistance, data capacity, and accountability. Once states identified strengths and areas of need, they worked with stakeholders to determine appropriate improvement strategies to address identified needs and develop a theory of action. The theory of action was a critical aspect of SSIP Phase I because it assisted state teams in moving beyond selecting programs or solutions they believed would produce the intended results to examining how the improvement strategies selected during Phase I would actually make change happen (i.e., improve SIMR). Phase I encouraged states to deliberately consider interventions rather than jump to action to quickly investing time, money, and other resources in activities that might not help a state realize its goals. States fully examined what was happening with students with disabilities in their respective states and thoughtfully determined which issues identified are actually problems and what strategies might actually work to solve the problems (Abercrombi et al., 2018; Organizational Research Services, 2004).

Phase II

In Phase II of the SSIP conducted in FFY 2014, states developed a multiyear plan to address needs identified in Phase I aligned with the proposed theory of action. The focus of this phase was to build a state’s capacity to support local education agencies in the implementation of EBPs that will support improvement in the SIMR. Phase II built upon information learned in Phase I, providing a detailed plan for how to turn theory into action. The result of Phase II was a detailed plan including steps necessary to accomplish the identified improvement activities, resources needed to accomplish activities, timelines for implementation, and measures needed to evaluate implementation.

Phase III

Phase III of the SSIP, initiated in FFY 2015 and continuing until FFY 2018, focused on implementation of the plan. During this phase, states implemented the improvement activities delineated in Phase II and collected data on effectiveness. Phase III lasted 3 years, during which states reported annually on the results of the ongoing evaluation, extent of progress, and any revisions needed to the improvement activities or evaluation plan.

SSIP in the Republic of the Marshall Islands

Entities (e.g., Republic of the Marshall Islands [RMI], Guam, Federated States of Micronesia) receive IDEA Part B funding under the Freely Associated States provision of IDEA (2004). As required by IDEA (2004), these entities must submit USDOE-approved SPP/APRs, biannual data reports, and an annual application for funds. The educational systems in many of these entities differ vastly from those of the 50 U.S. states. Geographic location, linguistic and cultural differences, among other barriers, make implementation of U.S. policies difficult. It is important to understand how implementation of U.S. federal policy occurs within the unique context of each of the Pacific entities to help inform future iterations of IDEA and guidance provided by the OSEP.

In FFY 2013 RMI—like all other states, territories, and freely associated states—was required to develop an SSIP. To explore implementation of the SSIP in the RMI, archival data were reviewed and summarized (e.g., SPP/APR; SSIP Phase I, II, II reports; Ministry of Education policy papers). Detailed descriptions of data derived from documents are provided to assist readers in generalizing the information to their own context. Historical and factual data related to the Marshall Islands were also retrieved from various websites (i.e., Marshall Islands Public School System, U.S. Department of State).

The RMI is a sovereign nation consisting of approximately 70 square miles of land situated west of the international dateline, just north of the equator and spread across approximately 750,000 square miles of ocean (U.S. Department of State, 2018). Equitable access to public education in the Marshall Islands is limited. There are 24 inhabited atolls with 78 public elementary schools and 5 public high schools. In the 2017–2018 school year, approximately 14,031 students were enrolled in the public schools (i.e., 10,736 primary students; 3,295 secondary students; Public School System [PSS], 2018). Access to secondary education in particular requires students to relocate to one of the five atolls with a high school, which in some cases is hundreds of miles away and requires students to move away from immediate family and live in a dormitory or with extended family members. The primary language of instruction in Grades K through 6 is Marshallese (native language), and in secondary grades the primary languages of instruction are English and Marshallese, with Marshallese being used approximately 50% of the time during instruction (RMI, 2015).

SSIP Phase I in RMI

Prior to 2013, special education operated under the Ministry of Education. In 2013, the RMI enacted the Public School System Act, which mandated development of an autonomous PSS. This legislation established a nine-member National Board of Education, with five elected members from geographical districts that have ultimate control of and authority over the PSS. Within the newly reorganized PSS, Special Education operates under the Division of Instructional Services. The special education director reports to the associate commissioner of education for instructional services, who then reports to the commissioner of education. A special education specialist is assigned as a liaison to a specific region (i.e., Northern District, Eastern District, Southern District, Central District, Kwajalein/Mejatto, Western District). Each special education specialist focuses on certain content areas (e.g., fiscal, preschool education, K–8 education, data management, middle/high school). High school principals report to an associate commissioner for secondary education. The special education specialist for secondary education and transition acts as a liaison between the high schools and the associate commissioner.

Special education services are available on all 24 inhabited atolls in 72 of 78 public elementary schools and all five public high schools. Approximately 5% of students enrolled in the PSS (i.e., 700 students) are receiving services under IDEA (2004). Each year, more and more students are identified as needing services via Child Find activities and public awareness campaigns. Students with disabilities in the Marshall Islands are supported by approximately 125 special education teachers, a program director, five program specialists, and three support staff (Ministry of Education Republic of Marshall Islands, 2013).

Similar to other states and territories receiving Part B IDEA funds, strategic planning in the Marshall Islands was not a new endeavor. Since the 1980s, the Republic of Marshall Islands Ministry of Education had been engaging in strategic planning to decrease the gaps between where they were and where they wanted to be (Ministry of Education Republic of Marshall Islands, 2013). Schools have been required to develop and update annual school improvement plans; however, these efforts had not had lasting effects. In 2012, the Ministry of Education designed a new local accreditation program for all public schools in which schools were evaluated on leadership, teacher performance, data management, curriculum and student learning, campus, classrooms, facilities, and school improvement planning. Schools were rated on a 4-point Likert-type scale (Level 4 = exceed standards; Level 3 = complies with standards, Level 2 = substantially complies with standards and has plan to correct deficiencies; Level 1 = does not meet standards). Schools not sufficiently meeting the standards (Levels 1, 2) were required to develop an action plan to address areas of need identified by the evaluation team. This new accreditation program was designed to provide a research-based continuous school improvement model appropriate to the local context of the Marshall Islands. This process was focused on the school as a whole and did not specifically address needs identified in special education (Ministry of Education Republic of Marshall Islands, 2013).

In 2014, the RMI PSS convened stakeholders (e.g., middle and high school teachers, principals, representatives from the College of Marshall Islands, parent groups, Special Education Advisory Council), to analyze graduation and other related data (e.g., transition components of the individualized education program (IEP), postschool outcomes, attendance) to determine contributing factors for the low graduation rates of youth with disabilities attending high schools in the RMI (i.e., SSIP Phase I). During the infrastructure analysis, the PSS and stakeholders identified barriers and leverage points for improvement and focused the SIMR on increasing the number of youth with a disability graduating high school with a standard diploma in Majuro and Ebeye schools. The team identified five major strands of improvement activities aligned with the MOE’s 2013–2016 Strategic Plan: (1) increasing parent involvement, (2) increasing the number of students transitioning from middle to high school, (3) improving the quality of teaching, (4) improving the quality of student transition plans, and (5) improving student attendance.

SSIP Phase II in RMI

During school year 2015–2016 (i.e., Phase II), the PSS and stakeholder team developed an improvement plan to guide their work in (a) improving infrastructure to better support high schools to implement EBPs in secondary transition, (b) supporting high schools in implementing EBPs, and (c) evaluating efforts to improve graduation rates for youth with disabilities in high school. The evaluation plan followed the Plan, Do, Study, Act Model to ensure the work they were doing was successful in achieving the outcomes they desired. The RMI implementation and evaluation matrix included in the report outlined a 5-year plan to improve graduation rates among youth with disabilities in Majuro and Ebeye high schools. The team developed both formative and summative evaluation questions to guide the evaluation plan.

According to the RMI SSIP Phase III Year 1 report (RMI PSS, 2017), “the evaluation plan is aligned to the RMI Public School System Theory of Action’s strands (e.g., improving attendance, improving quality of transition services and IEPs, improved parent engagement) and the logic model input and outputs” (see RMI PSS SSIP Phase III Year 1 Report, p. 6). “Formative and summative evaluation questions focus on short, medium, and long-term outcomes that will inform the measurement of SSIP implementation and impact of achieving RMI’s State-Identified Measurable Result of increasing graduation rates” (p. 4). See Table 1 for formative and summative evaluation questions.

Table 1.

Republic of Marshall Islands State Systemic Improvement Plan Formative Evaluation Questions.

Formative questions	Summative questions
1. Was there an increased number of parents involved with schools? 2. Was the cadre of parent educators an effective way to increase parent knowledge and skills to support schools in implementing action plans? 3. Did the revised policies and procedures and the transition guidance document increase teachers and administrators understanding of skills needed to provide quality instruction in special education, specifically secondary transition requirements? 4. Did the new policies and procedures for advancement of students with disabilities increase teacher, administrator, family, and community awareness of requirements for promotion from Grade 8 to high school? 5. Did the public awareness campaign effective in increasing community awareness of the importance of attending school and persisting through high school? 6. Has the PSS provided sufficient support to high schools in order for high school teachers to provide appropriate transition related instruction, services, and supports to youth with disabilities?	7. Was there an increase in quality of IEPs as evident by the total score on the Republic of the Marshall Islands IEP Evaluation Rubric? 8. Was there an increased number of youth receiving appropriate secondary transition services and supports (Indicator 13)? 10. Was there an increase in the number of students with disabilities attending school on a regular basis (<5 days absent a year)? 11. Was there an increased number of youth graduating with a high school diploma or equivalent (SPP/APR Indicator 1; (RMI PSS, 2017, p. 5)?

Formative questions

Summative questions

1. Was there an increased number of parents involved with schools?
2. Was the cadre of parent educators an effective way to increase parent knowledge and skills to support schools in implementing action plans?
3. Did the revised policies and procedures and the transition guidance document increase teachers and administrators understanding of skills needed to provide quality instruction in special education, specifically secondary transition requirements?
4. Did the new policies and procedures for advancement of students with disabilities increase teacher, administrator, family, and community awareness of requirements for promotion from Grade 8 to high school?
5. Did the public awareness campaign effective in increasing community awareness of the importance of attending school and persisting through high school?
6. Has the PSS provided sufficient support to high schools in order for high school teachers to provide appropriate transition related instruction, services, and supports to youth with disabilities?

7. Was there an increase in quality of IEPs as evident by the total score on the Republic of the Marshall Islands IEP Evaluation Rubric?
8. Was there an increased number of youth receiving appropriate secondary transition services and supports (Indicator 13)?
10. Was there an increase in the number of students with disabilities attending school on a regular basis (<5 days absent a year)?
11. Was there an increased number of youth graduating with a high school diploma or equivalent (SPP/APR Indicator 1; (RMI PSS, 2017, p. 5)?

Source. RMI PSS (2017, p. 5).

Note. IEP = individualized education program; SPP/APR = State Performance Plan/Annual Performance Report; PSS = Public School System.

SSIP Phase III in RMI

Implementation of the SSIP began school year 2016–2017 (i.e., Phase III, Year 1). As part of SSIP efforts, the PSS formed school implementation teams at two middle schools and three high schools on Majuro and Ebeye, which consisted of middle and high school principals, special education teachers, general education teachers, school counselors, and parents. The PSS also sought input from students with disabilities in the Marshall Islands through focus groups. Professional development was provided to the school implementation teams on the Plan, Do, Study, Act Model. In addition, the PSS supported school teams in developing an action plan for change aligned with their school’s overall school improvement plan.

RMI sought support in the development and implementation of the SSIP from OSEP. The state lead from OSEP’s Monitoring and State Improvement Planning Division (MSIP), OSEP’s Performance Accountability Implementation Team (PAIT), and other staff from OSEP’s Research to Practice Division (RTP), supported RMI PSS in the implementation of the SSIP through the Differential Monitoring and Supports Systems (OSEP, 2017), a result of RDA. The differential monitoring system (DMS) was designed to help mitigate risk and use federal resources more wisely. Based on a state’s need, OSEP differentiates supports to states providing either universal, targeted, or intensive supports. A designation of intensive simply means the state would benefit from a higher level of engagement from OSEP to assist with the identification of problems and solutions to support improved performance (OSEP, 2017). During each phase of the SSIP, members of OSEP’s MSIP and RTP teams supported RMI with universal, targeted, and intensive supports across different areas (e.g., graduation, SSIP, and participation in statewide assessment). Supports included, but were not limited to, providing tools and specific guidance around data analysis and infrastructure analysis, developing a theory of action, and determining other OSEP-funded technical assistance (TA) centers that would provide beneficial services to the Marshall Islands. Members of the MSIP and RTP teams met with RMI monthly to discuss progress and provide additional support as needed.

During each phase of the SSIP RMI, the school implementation teams also received assistance from two OSEP-funded Technical Assistance and Dissemination (TA&D) Centers (i.e., National Center for Systemic Improvement [NCSI] and National Technical Assistance Center on Transition [NTACT]). NCSI supports states in building capacity in data use, knowledge utilization, systems change, and communication and collaboration. The NCSI has provided both targeted and intensive technical assistance to RMI PSS specifically around improving data collection systems to improve data use and communication within and across agencies to support more systematic use of data to improve graduation rates of youth with disabilities in the RMI. In addition to organizing regular meetings with RMI staff and other collaborators, NCSI also provided customized tools designed to support each phase of the SSIP and guidance regarding building capacity within the RMI to implement the SSIP. The NTACT provided intensive technical assistance focused on building capacity of the PSS to support schools in the implementation of EBPs to improve school completion, including (a) quality individualized education program (IEPs), (b) collaborative teaching, (c) family engagement, (d) school attendance, and (e) quality of instruction. The NTACT provided both onsite and virtual professional development and coaching to support implementation of the improvement activities outlined in the SSIP. The NCSI and NTACT worked collaboratively to support the RMI in their mission to improve graduation rates of youth with disabilities. They provided joint technical assistance leveraging the expertise and resources of each of the centers to maximize impact for the Marshall Islands.

RMI PSS SSIP Preliminary Outcomes

As a result of implementing the SSIP, RMI has made notable accomplishments including, but not limited, to organizational changes, policy changes, and improved outcomes for students with disabilities. According to PSS staff:

The State Systemic Improvement Plan (SSIP) has impacted the way we approach special education in the Marshall Islands. Since the development of the SSIP, we have made many major improvements in how we approach Special Education Services. The SSIP process has helped us understand the importance of focusing on outcomes of students rather than merely compliance with the IDEA Part B Indicators. This shift is reflected in some of the policy and procedural changes made these past few years during SSIP implementation. In addition, to the shift of focus from compliance to outcomes, we have also built capacity within our system to deliver quality instruction and services to students with disabilities in the Marshall Islands. The SSIP and OSEP’s Differential Monitoring System (DMS) have been key elements for this change process. (RMI PSS, 2018, p. 37)

By the end of Year 1 of implementation, results of SSIP evaluation measures indicated RMI PSS had (a) improved infrastructure to provide and ensure FAPE (e.g., improved data collection systems and dedicated staff), (b) increased knowledge of teachers and other PSS staff regarding predictors of postschool success, EBPs, and transition assessment, (c) increased number of schools developing and implementing targeted action plans for change, and (d) increased reading and math skills for middle school students with disabilities (RMI PSS, 2017, p. 1).

Organizational changes

One organizational change resulting from the SSIP was the development of school improvement plans focused on needs of students with disabilities in schools, regardless of the school’s level of determination outlined in the 2012 local accreditation system described earlier. Each school’s improvement plan aligned with the SSIP theory of change. There has also been an increase in collaboration across divisions within the PSS as a result of SSIP implementation including special education, assessment, and the secondary education and career development division. The PSS hired a staff focused on secondary transition to support school implementation teams in implementing action plans and to be the liaison between schools and other divisions. As the PSS staff noted in the Phase III Year 2 report (RMI PSS, 2018):

The SSIP requirement for schools to develop and implement school improvement plans has made a great impact on schools. Team members are uniting together to accomplish their plans. The school plans made a connection to shape schools’ relationships with their students with a disability, regular education teachers, school’s principals, parents, SEAC members, and other community partners. School plans have involved all members and ensured all members had major roles in implementation. Data that have been collected on the implementation of activities included in the plans are leading schools to make better decisions on how they can make Special Education services higher quality in our schools. (p. 38)

Another organizational change was the provision of sufficient supports to high schools in order for high school teachers to provide appropriate transition-related instruction, services, and supports to youth with disabilities. As described in the RMI SSIP report, RMI PSS has designed and delivered professional development aligned with teacher need. According to evaluation data reported, “teachers were overall satisfied with the training and felt the content was relevant and useful to their work (mean score 4.2 out of 5) and positively contributed to their school improvement plans” (RMI PSS SSIP Phase III Year 2 Report, p. 5). Teachers also perceive the professional development and other supports being provided is increasing their knowledge of predictors of postschool success, EBPs, and transition assessment. “Results of the evaluation of the workshop indicated school implementation teams gained more knowledge of Evidence Based Practice (mean score out 4.7 out of 5)” (RMI PSS SSIP Phase III Year 2 Report, p. 10).

Policy changes

In addition to organizational changes, the PSS has clarified policies to support students in transitioning from middle to high school and high school to adulthood. As indicated in the SSIP Phase III Year 2 report, divisions within the PSS conducted a root cause analysis to determine barriers to students moving from middle school to high school and persisting through to graduation. It was determined that

results of the MISAT and other PSS promotion requirements were barriers to students with disabilities moving from eighth grade to high school. PSS has since removed these barriers to allow for students with disabilities to be promoted from grade eight to nine based on meeting required actions denoted in the IEP. (p. 7)

It is important to note that these changes in promotion requirements do not equate to decreased expectations for students with disabilities. There has simultaneously been a concerted effort to increase the participation and success of all students in the MISAT, resulting in an increase in participation from 25% (SY 2015–16) to 89% (SY 2017–18) and subsequent increases in performance in reading and math (RMI PSS SPP/APR). There is a reality of geographical and structural capacity barriers to the high schools serving all students entering who are high school age. To deal with these realities, these policies were established and other programs (e.g., pre-nine) were developed to better serve students during this transition period. PSS has also clarified the policy regarding due dates for IEPs, ensuring that all IEPs were submitted by May 30 each year in preparation for the next school year. This clarification has resulted in teachers who are prepared with “Quality of IEPs for implementation in the upcoming school year.” (p. 37).

Student outcomes

Most important are the changes in student outcomes that RMI is seeing as a result of SSIP implementation. Translating research findings into everyday practices of teachers in the classroom is not easy for schools and teachers. It involves a host of complex and interrelated issues that can sometimes be difficult to decipher. There is a science to implementation (Cook & Odom, 2013; Eccles & Mittman, 2006). Schools and teachers need to be able to assess the relevance and fit of the intervention to the target environment, ensure individuals implementing the practice have sufficient time and knowledge of the practice, and ensure the context in which the practices will be implemented is sufficient (e.g., available resources, staffing, coaching, training, administrative support (Cook & Odom, 2013; Fixen et al., 2005). As part of SSIP activities, the PSS has built the infrastructure to successfully support schools. Improvements in infrastructure included ongoing professional development and coaching provided by PSS staff with support from NTACT and NCSI, improved data collection systems, and improved communications within and across schools in the Marshall Islands.

The RMI School implementation teams have been engaging in the Plan, Do, Study, Act model as outlined in the RMI PSS SSIP Phase II Report since 2016–2017 school year. Each school implementation team developed a SMART goal and developed a plan to achieve the goal (see Figure 2 for school SMART goals). School implementation teams monitor progress on the school improvement plan quarterly. Although the school implementation teams have encountered their share of challenges (e.g., staff turnover, lack of consistent data collection systems, and lack of parent engagement) in their efforts to adopt and implement EBPs to improve outcomes of youth, they continue to persist through the Plan, Do, Study, Act cycles, celebrating successes and adjusting their plans as needed. Below highlights some of the accomplishments, as reported in the RMI SSIP Phase III Year 2 report.

Figure 2.

School implementation team SMART goals.

Marshall Island High School (MIHS) implemented Power School to systematically collect attendance data. The average number of days students with disabilities were absent from school in the first quarter was 2.9 out of 48 (mean attendance 94%). The average days in the second quarter was 4.3 out of 33 days (mean attendance 87%). MIHS noted in the Phase III report that the data from second quarter were skewed due to a few students with excessive absences (>10 days). To better understand attendance for the whole school and identify individuals to provide more intensive supports, they ran the analysis without the outliers. When these outliers were removed from analysis, the average number of days missed during the second quarter was 2.3 out of 33 days (a decrease of 0.6 days; mean attendance = 93%). As a result of a deeper data analysis, MIHS reported implementing individualized interventions (e.g., home visits to discuss attendance policy and learn reasons students were not attending school) for students with excessive absences and their families to support students in attending school (RMI PSS SSIP Phase III Year 2 Report), which in theory should improve the overall attendance rate for the school. According to the Phase III Year 2 report, the home visits are yielding additional information to inform intervention selection for individual students (e.g., student located from outer islands living with relatives who lack general interest in student’s progress in school, pressure of peers to stay out of school).

Majuro Middle School (MMS) developed a tutoring program for reading and mathematics for students with disabilities. This program was initiated in the 2016–2017 school year and includes the use of peer tutoring and other EBPs in reading and math (e.g., explicit instruction). In 2017 school year, students participating in the program increased both reading and math performance as measured by curriculum-based measures. Also notable was the number of Grade 8 students with disabilities whose performance on MISAT (state-wide assessment) improved. In 2015–2016, one Grade 8 student met the cut-off score for high school entrance. In 2016–2017, five Grade 8 students met the cut-off score. In the 2017–2018 school year students’ performance in math had improved at mid-year; however, reading scores had not. MMS staff identified additional strategies to incorporate into the reading program to support continued student growth.

Laura High School (LHS) focused improvement efforts on increased parent engagement. They have successfully increased the number of parents engaged in IEP meetings from 50% to 65%. Although they have not quite met their goal, LHS has been successful in educating a cohort of parents who have now agreed to take a leadership role in supporting other parents in the community. They have officially elected officers for their LHS parent organization. As indicated in the Phase III year two report, the first item on the LHS parent organization agenda is to help other parents of students with disabilities understand their rights under IDEA (2004).

Kwajalein High School, Ebeye Middle School, and Ebeye Elementary (Kwajalein Atoll Schools) decided to partner together to achieve the same goal, to improve attendance rates during state-wide testing, ultimately impacting student proficiency. Prior to participating as a school implementation team with RMI PSS SSIP efforts, the attendance rate during statewide assessment for students with disabilities in the Kwajalein Atoll Schools was 21%. The low participation rate did not provide sufficient data for the schools to assess proficiency rates. The focus of their action plan for the first few years of SSIP implementation was attendance during testing. As of the Phase III Year 2 report, the Kwajalein Atoll schools achieved 100% participation rate during statewide testing. With an accurate picture of proficiency rates, they have shifted focus to performance:

The special education and general education teachers are co-planning each of the instructional lessons. In some cases, the teachers have developed modified lesson plans for students with disabilities that include more scaffolding and additional supports. To promote reading proficiency, they have implemented school-wide activities such as reading comprehension daily activity school-wide and 3-5 vocabulary daily. Special education and general education teachers are working together to ensure each student with a disability receives the appropriate accommodations outlined in the IEP during all class activities and assessments and are monitoring the effectiveness of those accommodations. Kwajalein schools are also implementing Achieve 3000 and have conducted two MI-SAT trials with students in preparation for the upcoming assessment to teach test-taking skills as well as familiarize students with the assessment format. (RMI PSS SSIP Phase III Year 2 Report, p. 14)

School Year 2013–2014 was a baseline year for RMI PSS in terms of outcome data collection. It was then the PSS began to monitor graduation rates for all students with disabilities in Marshall Islands, but particularly those students in Majuro and Ebeye, as these schools were the focus of the SSIP. Overall, there has been an increase in graduation rates for students with disabilities in the Marshall Islands (i.e., 37.5% in FFY 2014, 46.2% in FFY 2016). Analysis of the data, specifically for Majuro and Ebeye schools, indicated a notable increase (i.e., 17% in FFY 2014, 46.2% in FFY 2016). The only students graduating with a standard diploma in FFY 2016 were from Majuro and Ebeye schools. While not the focus of their SSIP, RMI has also observed changes in postschool outcomes for students with disabilities. For the first time, RMI has met targets for SPP/APR Part B Indicator 14 Measures A, B, and C. In the past, most students were enrolled in other education or in other employment. Outcome data from FFY 2015 indicated students are now enrolling in higher education (i.e., 21.4%) and becoming competitively employed (i.e., 35.7%), in addition to the other education and employment options (i.e., 42.9%).

Limitations and Recommendations for Future Research

While a relatively rich description of SSIP implementation in the Marshall Islands is provided, it is not without limitations. Qualitative techniques were used (e.g., document analysis), but this was not a true qualitative case study. Although a case study design is a relatively flexible research method, descriptions here lack many elements to be considered methodologically rigorous (Brantlinger et al., 2005). Qualitative research methods are prone to researcher bias and as Brantlinger et al., (2005) indicated, it is important to be forthright about perspective. Data reviewed and reported in this study came from publicly available sources. The views portrayed are those of stakeholders involved in writing the various reports, specifically the SSIP Phase II and Phase III reports. Authors of this manuscript represent diverse perspectives, including individuals from the Marshall Islands. All authors support improved graduation rates and want to see the Marshall Islands succeed in their goals; however, we have attempted to illustrate SSIP implementation in the Marshall Islands based solely on the written reports versus our own assumptions, beliefs, and values.

Another limitation is the lack of data sources for triangulation. In a qualitative research study, evidence from multiple data sources would be included in the analysis to attempt to establish trustworthiness and credibility. Although this study included multiple written documents and websites as data sources, it could benefit from additional data collection through interviews or focus groups with teachers or other professionals involved with implementation of the SSIP to verify information included in reports. Future research should consider multiple data sources and follow steps for systematic data analysis to confirm the hypothesis that the SSIP process is supporting systems change. Last, focusing only on the Marshall Islands PSS limits the generalizability of the results. However, context-dependent knowledge is valuable in this particular situation. Although the context is unique in the Marshall Islands, many of the actions taken as a result of the SSIP (e.g., Plan, Do, Study, Act) are universal strategies and can inform policy and practice elsewhere.

Implications for Policy and Practice

The RMI reflects a unique history and culture as a freely associated state of the United States. Its citizens read, write, and speak both Marshallese and English, and the curriculum reflects an expectation of mastery of American and Marshallese language, history, and government. The geography of the RMI dictates access to economic prosperity, educational opportunities, and services. Despite the socio-cultural difference of the RMI from mainland states, implications for policy and practice can be discerned from their SSIP work. Through the initial needs assessment and infrastructure analysis process required in Phase I of the SSIP process, three key recommendations can be gleaned:

Aligning improvement efforts with other federal and state initiatives is key to a successful implementation effort.

Changes to the system and substantive improvements necessitated a continuous review and analysis of data, following the Plan, Do, Study, Act model. This step required leaders to become experts in the collection, analysis, and use of quality data for planning but also in translating and disseminating that knowledge to others. So, creating a culture that models and supports the use of data-driven decision making at every level from the state to the classroom is recommended. State and local education policies and procedures should facilitate and potentially even incentivize data-driven decision making to support positive outcomes for students with and without disabilities.

A persistent engagement of stakeholders across all levels and directions of the system is important. Key stakeholders include decision makers at the highest levels, representatives of potential partner agencies and community entities, and consumers of the services and outputs of the system. This engagement of stakeholders at all decision points (e.g., interpreting data and identifying next steps) is important in any systems change effort; however, within the social and cultural structure of the RMI this was novel and empowering.

Phase II of the SSIP process can inform additional implications for policy and practice. The development of an evaluation plan proved important in the RMI. The state, a local district, and a school must monitor improvement efforts closely and follow the evaluation plan systematically to know when progress is being made, or not. Another factor important to success with a systems change initiative was that RMI identified external supports from (a) the monitoring and practice divisions in OSEP and (b) specific technical assistance centers to provide both process and content expertise. Having coordination—not silos—of support was important to forward movement. This support, additionally, was systematically arranged to build the capacity of RMI’s PSS staff and other stakeholders to develop expertise. TA providers initially delivered resources such as content knowledge, a structure for meetings, and data collection tools, and they modeled how to use these resources for RMI staff. Gradually, RMI staff used the content and tools with coaching from TA providers in Phase III of the SSIP work. It is important for federal policy to model the expectations for state and local education agencies. As with the recent reauthorization of Workforce Innovation Opportunities Act, federal regulations should explicitly require coordination both within and among agencies (e.g., Education and Vocational Rehabilitation) to support improved outcomes for students. Finally, Phase III of the SSIP process has identified two additional implications. Intra-agency and interagency collaboration, including clarifying roles and responsibilities and regularly scheduled reviews of progress were valued and central to improvement efforts and recommended. Also, as gaps in services or barriers are identified through adherence to a continuous improvement model, a commitment to building the infrastructure to support effective instructional practice implementation through professional development is essential.

Conclusion

It is apparent from the results of the SSIP Phase III evaluation data for RMI that the SSIP process has assisted them in developing the capacity to better support their schools in implementing EBPs to support improved student outcomes. The Plan, Do, Study, Act cycles specifically have helped the PSS and school implementation teams maximize their resources and more strategically consider the interventions that would lead to the desired outcomes they were trying to achieve, increased number of youth with disabilities graduating with a standard diploma. The collaborative technical assistance provided by OSEP, NCSI, and NTACT has proven effective in helping build RMI’s capacity to plan to support improved student outcomes.

Footnotes

Authors’ Note

This document was developed by The National Technical Assistance Center on Transition (NTACT), Charlotte, NC, funded by Cooperative Agreement Number H326E140004 with the U.S. Department of Education, Office of Special Education Programs and the Rehabilitation Services Administration. Opinions expressed herein do not necessarily reflect the position or policy of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Department of Education.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Abercrombie

Boswell

Thomasoo

(2018). Thinking big: How to use theory of change for systems change. New Philanthropy Capital. https://www.thinknpc.org/publications/thinking-big/

Brantlinger

Jimenez

Klingner

Pugach

Richardson

(2005). Qualitative studies in special education. Exceptional Children, 71(2), 195–207. https://doi.org/10.1177/001440290507100205

Candy

(2006). Practice based research: A guide. CCS Report, 1, 1–19. https://www.creativityandcognition.com/resources/PBR%20Guide-1.1-2006.pdf

Cook

B. G.

Odom

S. L.

(2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79, 135–144.

Corr

(2015). Results driven accountability. RPTAC Region 4 Conference. https://www2.ed.gov/about/offices/list/osers/osep/rda/index.html

Eccles

M. P.

Mittman

B. S.

(2006). Welcome to implementation science. Implementation Science, 1, 1–3.

Fixen

D. L.

Naoom

S. F.

Blasé

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature (FMHI Publication #231). The National Implementation Research Network, Louis de la Parte Florida Mental Health Institute, University of South Florida. http://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-MonographFull-01-2005.pdf

Individuals with Disabilities Education Improvement Act of 2004, P. L. No. 108-446, 20 U.S.C. § 1400 sec. 602 (34).

Institute for Education Sciences, National Center for Education Statistics. (2017). Common core of data: Public high school 4-year adjusted cohort graduation rate (2014-2015). https://nces.ed.gov/programs/coe/indicator_coi.asp

10.

Lipscomb

Haimson

Liu

A. Y.

Burghardt

Johnson

D. R.

Thurlow

M. L.

(2017). Preparing for life after high school: The characteristics and experiences of youth in special education. Findings from the National Longitudinal Transition Study 2012. Volume 2: Comparisons across disability groups: Full report (NCEE 2017-4018). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/projects/evaluation/disabilities_nlts2012.asp

11.

Ministry of Education Republic of Marshall Islands. (2013). Special Education policies and procedures. https://pss.edu.mh/special-education-documents/

12.

National Council on Disability. (2012). Section 2: Cross-system focus to improve outcomes across the lifespan for people with ID/DD. http://www.ncd.gov/publications/2012/Apr222012/section2/#S221

13.

Newman

Wagner

Knokey

A.-M.

Marder

Nagle

Shaver

Wei

with Cameto

Contreras

Ferguson

Greene

Schwarting

(2011). The post-high school outcomes of young adults with disabilities up to 8 years after high school: A report from the National Longitudinal Transition Study-2 (NLTS2) (NCSER 2011-3005). SRI International. https://nlts2.sri.com/reports/2011_09_02/index.html

14.

Office of Special Education Programs. (2009). Part B SPP/APR indicator/measurement table. https://sites.ed.gov/idea/files/policy_speced_guid_idea_bapr_bmeatab2.pdf

15.

Office of Special Education Programs. (2012). Results-driven accountability in special education summary. https://www2.ed.gov/about/offices/list/osers/osep/rda-summary.pdf

16.

Office of Special Education Programs. (2017). Differentiated monitoring and support system. https://osep.grads360.org/#communities/pdc/documents/14982

17.

Organizational Research Services. (2004). Theory of change: A practical tool for action, results, and learning. Annie E. Casey Foundation. http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf

18.

Park

Hironaka

Carver

Nordstrum

(2013). Continuous improvement in education. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.

19.

Public School System. (2018). School enrollment 2017-2018. Republic of the Marshall Islands. https://pss.edu.mh/documents/

20.

Republic of Marshall Islands Public School System. (2017). RMI SSIP phase III year 1 report. https://osep.grads360.org/#report/apr/2017B/publicView?state=MH&ispublic=true

21.

Republic of Marshall Islands Public School System. (2018). RMI SSIP phase III year 2 report. https://osep.grads360.org/#report/apr/2017B/publicView?state=MH&ispublic=true

22.

Republic of the Marshall Islands. (2015). Marshall Islands public school system language education policy. https://drive.google.com/file/d/1UZVGou2_BFQk6X_1IRL8jxoEWdE8u8QX/view

23.

Ryder

(2015). Results-driven accountability. Presentation on state leadership call. https://www.cec.sped.org/~/media/Files/CAN%20Documents/CAN%202014/Ruth%20Ryder.pdf

24.

Smith

(2017, Fall). Special education data: In the era of accountability and district improvement [conference session]. Western Regional Resource Center Regional Data Manager Meeting. Salem, Oregon. https://slideplayer.com/slide/4570310/

25.

U.S. Department of Education. (2014). New accountability framework raises the bar for state special education programs. https://www.ed.gov/news/press-releases/new-accountability-framework-raises-bar-state-special-education-programs

26.

U.S. Department of Education. (2016). Part B State performance plan/annual performance report 2016 indicator analyses. https://osep.grads360.org/#communities/pdc/documents/12827

27.

U.S. Department of Labor. (2017). Persons with a disability: Labor force characteristics–2016. https://www.bls.gov/news.release/disabl.nr0.htm

28.

U.S. Department of State. (2018). U.S. relations with the Marshall Islands: Bilateral relations fact sheet bureau of East Asian and Pacific affairs. https://www.state.gov/u-s-relations-with-marshall-islands/

29.

Wagner

Blackorby

Cameto

Hebbeler

Newman

(1993). The transition experiences of young people with disabilities: A summary of findings from the National Longitudinal Transition Study of Special Education Students. SRI International.