Abstract
This study confirmed the reliability and validity of the Quality Indicators of Exemplary Transition Programs Needs Assessment–2 (QI-2). Quality transition program indicators were identified through a systematic synthesis of transition research, policies, and program evaluation measures. To verify reliability and validity of the QI-2, we administered rigorous methods including a content analysis, an expert review, and instrument field test. Forty-seven indicators were categorized into seven domains: (a) transition planning, (b) transition assessment, (c) family involvement, (d) student involvement, (e) transition-focused curriculum and instruction, (f) interagency collaboration, and (g) systems-level infrastructure. The QI-2 was found to be reliable and valid for use by transition stakeholders, districts, and states for evaluating quality of transition programs and identifying areas for program improvement.
A major goal of special education is to help students to prepare for a meaningful adult life. With sufficient preparation and support, students with disabilities can achieve desired outcomes such as integrated employment, postsecondary education, and participation in social and community life. Transition planning and service provision was strengthened with the reauthorization of the 2004 Individuals With Disabilities Education Act (IDEA; 2004). Specifically, the individualized education program (IEP) must include transition planning and services beginning when youth with disabilities are 16 years old or younger, if needed. Transition planning must target measurable postsecondary goals focusing on education/training, employment, and, where appropriate, independent living. These long-term goals must be based upon age-appropriate transition assessments. IDEA emphasizes a results-oriented process, which strengthens accountability toward postschool outcomes of youth with disabilities. Over the past two decades, positive improvements in postschool outcomes have been demonstrated in integrated employment, accessing postsecondary education, living independently, and fully participating in the community (Blackorby & Wagner, 1996; Newman, Wagner, Cameto, Knokey, & Shaver, 2010).
Despite explicit mandates, postschool success has not been achieved for youth with disabilities as compared with youth without disabilities. A recent report from the National Longitudinal Transition Study–2 (NLTS-2) demonstrates that youth with disabilities were more likely to earn lower salaries than their peers without disabilities (Newman et al., 2011). These researchers found that youth with disabilities were less likely to enroll in or complete postsecondary education programs compared with same-age youth without disabilities. In addition, youth with disabilities were less likely to have a checking account or a credit card and were also less likely to live independently than peers in the general population. Overall, these differences reflect limitations for youth with disabilities to fully engage in community life.
To confront these challenges, attention has turned to investigating promising and evidence-based transition practices leading to meaningful adult outcomes. Two systematic reviews from the National Secondary Transition Technical Assistance Center (NSTTAC) have directed attention to effective secondary special education and transition practices. The first study by Test, Fowler, and colleagues (2009) identified evidence-based practices to effectively teach transition-related skills. For example, community-based instruction, backward chaining, and systems of least to most prompts were identified as effective methods of instruction when teaching functional life skills. To increase students’ participation in IEP meetings, the Self-Advocacy Strategy, the Self-Directed IEP, and Whose Future Is It Anyway? have been determined to be evidence-based practices. To date, 65 evidence-based practices have been identified by Test and colleagues in the areas of (a) promoting student-focused planning, (b) teaching student skill development, (c) facilitating family involvement, and (d) improving program structure.
A second review by Test, Mazzotti, and colleagues (2009) identified predictors of postschool success. Sixteen in-school experiences were identified that correlate with improved postschool outcomes. Major predictors of success include (a) inclusion in general education, (b) paid work experience, (c) self-care/independent living, and (d) ongoing student support from transition programs. These efforts have driven the establishment of high-quality transition programs that affect and increase postschool outcomes.
Descriptions of evidence-based practices point to the importance of identifying indicators that measure the quality of transition programs and services; however, little research has been directed toward measuring specific elements of comprehensive transition programs. Current research has begun to identify indicators of programmatic success, given that a variety of state and district transition programs have emerged, including some that have identified disproportionate access to transition services (Baer, Daviso, McMahan Queen, & Flexer, 2011). Although educators do implement evidence-based transition interventions with students with disabilities (Test, Fowler, et al., 2009), the extent of a transition program’s success may be affected by factors both within and outside of school (Benz, Lindstrom, Unruh, & Waintrup, 2004). Diverse characteristics of transition programs and the contextual factors that affect success are essential aspects to be evaluated and tracked if we are to ensure high-quality transition programs.
Among the few program measurements that have been developed and published, Kohler’s (1996) Taxonomy for Transition Programming has been used as a guiding framework for planning, organizing, and evaluating transition programs. The Taxonomy was developed through multiple evaluation efforts that identified research-supported transition activities and program elements (Kohler, 1993, 1996). The Taxonomy includes five conceptual domains: (a) student-focused planning (e.g., IEP development, student participation in developing goals), (b) student development (e.g., life skills and employment instruction, vocational/career curricula, work experiences), (c) interagency collaboration (e.g., collaborative funding and staffing, shared transition service delivery), (d) family involvement (e.g., family training, active involvement in decision making and policy development), and (e) program structure (i.e., delivery structure of transition services). These five domains work as a framework that has recently been used to evaluate transition programs and plan for effective changes.
Two other existing transition program measurements have been developed that align with the Taxonomy: TransQual (Brewer, 2006) and Quality Indicators of Exemplary Transition Programs Needs Assessment (QI; Morningstar, 2006). These instruments assess programs and identify needed areas for improvement. In TransQual, 76 indicators were identified based on New York State regulations in the areas of (a) educational program structure, (b) interagency and interdisciplinary collaboration, (c) family involvement, (d) student involvement, (e) student development, and (f) IEP compliance performance. The QI (Morningstar, 2006) includes 40 indicators in the areas of (a) transition planning, (b) family involvement, (c) student involvement, (d) outcome-oriented curriculum and instruction, (e) inclusion in school, (f) interagency collaboration, and (g) transition assessment. As described previously, a third transition program measurement developed by Test, Mazzotti, and colleagues (2009) uses 16 predictors of postschool success in a checklist of indicators to evaluate the level in which transition programs align with the predictors.
Existing transition program evaluation measures do not sufficiently or comprehensively address IDEA regulations, emerging research findings, and predictors of success. Most measures were published prior to the emergence of evidence-based research results of practices and predictors. Furthermore, critical contextual issues concerning cultural diversity, assistive technology, and statewide accountability measures, although perhaps not yet sufficiently established as predictive of transition outcomes, are essential to consider when evaluating secondary transition programs. Therefore, we reexamined the QI (Morningstar, 2006) to reflect current federal legislation as well as research-based findings. The purpose of this study was to identify and validate indicators of quality transition programs to enhance and validate a revised Quality Indicators of Exemplary Transition Programs Needs Assessment–2 (QI-2; Morningstar, Erickson, Lattin, & Lee, 2012).
Method
Participants
Transition practitioners and stakeholders from school districts in the Midwest participated in the validation of the QI-2. Two phases of data collection were completed. The first sample of participants was a convenience sample of attendees at two state summer professional development workshops. This initial sample included 102 participants. The second group of participants completed an online version of the QI-2 between September and April, 2013. From this phase, 366 transition practitioners and stakeholders were added for a total of 468 participants. Almost half (49%, 244) were special education teachers, 21% (110) were transition coordinators or work experience/vocational coordinators, 27% (136) held other school-related positions (e.g., school psychologist, general education teachers, administrators, related services), and 1.5%, respectively (7 each), reported to be adult agency staff and parents.
Instrument Development
Four steps were used to ensure validity of the revised instrument: (a) identifying program quality indicators through multiple sources, (b) categorizing and organizing indicators into domains, (c) conducting an expert review, and (d) conducting a field test of sampling.
Identification of transition program quality indicators
Possible indicators were identified from a comprehensive review of published program evaluation instruments as well as recent transition research. Four instruments were reviewed: Taxonomy for Transition Programming (Kohler, 1996), TransQual (Brewer, 2006), an earlier version of the QI (Morningstar, 2006), and Predictors of In-School and Post-School Success (NSTTAC, 2013). This yielded a total of 281 distinct indicators. Current transition-related literature was examined to determine new and emerging issues, practices, and research syntheses. The 32 transition evidence-based practices identified by Test, Fowler, et al. (2009) were included as were other research syntheses of transition practices including interventions of functional life skills, social and communication (Alwell & Cobb, 2006), transition planning (Cobb & Alwell, 2008), and self-determination (Cobb, Lehmann, Newman-Gonchar, & Alwell, 2008). As a result, the comprehensive review yielded a total of 350 indicators.
Categorization and organization of indicators
To reduce the number of indicators, a content analysis data reduction method was completed (Miles, Huberman, & Saldana, 2014). Indicators were first grouped based on similarities of content and focus. A matrix was created that began with the seven domains from the earlier 2006 version of the QI. An individual and aggregated similarity and difference matrix (Namey, Guest, Thairu, & Johnson, 2008) was constructed to examine each indicator. During this comparison, if indicators revealed similar intent, they were combined and organized by domain. They were then reworded based upon scale development guidelines (Devellis, 2003), whereby ambiguity and complexity were diminished by removing double-barreled items, using action statements, reviewing grammatical structures, and reducing lengthy items. New indicators were created and added within a given domain as they emerged from new research and literature. Once completed, domains were renamed to better reflect associated indicators. In this way, seven domains were constructed: (a) transition planning, (b) transition assessment, (c) family involvement, (d) student involvement, (e) transition-focused curriculum and instruction, (f) interagency collaboration and community services, and (g) systems-level infrastructure. A final review of domains and indicators was then conducted for clarity. The process resulted in identification of 48 transition program quality indicators
Expert review
Once the indicators were established, three university faculty with expertise in secondary special education and transition services participated in an individual interview. This was to ensure that items were consistently understandable across respondents. We used a question-and-answer process, an established approach to developing an instrument (Fowler, 1995). The experts were asked to paraphrase their understanding of items, define terms, share misperceptions, and rate how confident they were that each indicator reflected the specific domain. We took systematic notes during this iterative process. In this way, indicators were reviewed for clarity, redundancy, alignment, and grammatical ease. As a result, we excluded one indicator, yielding a final list of 47 grouped across the seven domains: (a) transition planning, (b) transition assessment, (c) family involvement, (d) student involvement, (e) transition-focused curriculum and instruction, (f) interagency collaboration, and (g) systems-level infrastructure (see Table 1).
Quality Indicators of Exemplary Transition Programs Needs Assessment–2 (QI-2).
Source. Morningstar, Gaumer Erickson, Lattin, and Lee (2013). Morningstar, M.E. Erickson, A.G., Lattin, D.L. & Lee, H. (Revised June, 2012). Quality indicators of exemplary transition programs needs assessment summary [Assessment tool]. Lawrence, KS. University of Kansas, Department of Special Education. Retrieved from www.transitioncoalition.org
Note. QI = Quality Indicator; IEP = individualized education program. MAPS = Making Action Plans
Field test of sampling and data collection procedures
Following the expert review, a field test of the instrument was conducted. Participants were asked to rate each indicator using a 4-point Likert-type scale with 0 (not achieved), 1 (partially achieved), 2 (mostly achieved), or 3 (completely achieved) to determine the current status of each QI-2 indicator for their program, school, or district. The online version of the QI-2 averages domain scores, automatically calculates individual indicator scores, and reports them to respondents. Furthermore, the QI-2 collects demographic data (e.g., role, state, geographic area, professional development related to transition).
Data Analysis
Factor analysis is commonly used in psychometric evaluations of instruments (Floyd & Widaman, 1995) as a strong test for validity of constructs. Because the QI-2 was developed using theory and research, a confirmatory factor analysis (CFA) was completed using Mplus version 6.12 (Muthén & Muthén, 1998–2010) with maximum likelihood estimation. A CFA was completed to examine the latent structure of the instrument through patterns among item–factor relationships (Brown, 2006). MPlus is a statistical modeling software package that provides flexible analytical tools such as CFA that rely on the concept of latent variables so that data structures become understandable and interpretable. The sample was of adequate size to conduct the CFA (Comrey & Lee, 1992). The benefit of using a CFA approach is the ability to quantify model fit, and it is recommended that multiple fit indices be used (Thompson, 2004). Three of the most commonly reported CFA indices were used during data analysis: (a) chi-square, (b) comparative fit index (CFI), and (c) root mean square error of approximation (RMSEA).
Chi-square represents the degree to which the specified model reproduces the observed interrelationships of the items being examined. In this case, a statistically significant finding means that the model does adequately reproduce the observed patterns of relationships. Because chi-square is affected substantially by large sample sizes, reliance on it exclusively often leads to conclusions of poor model fit with large samples. Consequently, other measures are considered as well (Thompson, 2004). The CFI is a comparative measure of goodness-of-fit of the specified model relative to a baseline independence model. CFI values above .90 indicate acceptable and excellent fit. The RMSEA is a measure of lack of fit, so values below .08 indicate a model estimate that reproduces the population covariance sufficiently and thus has acceptable and excellent fit (Browne & Cudeck, 1993). In addition, internal reliability analysis was conducted using the SPSS. Coefficient alpha estimates for overall indicators and for each domain were calculated.
Results
The purpose of this study was to identify and validate indicators of quality transition programs for the revised QI-2. We used a CFA to validate constructs, along with an internal reliability analysis. The means for each of the 47 indicators ranged from 1.23 to 2.37 (M = 1.81, standard deviation [SD] = 0.6). Correlation estimates ranged from moderate to strong positive relationships. Table 2 provides the item means, SDs, and item-total correlations. The model fit statistics from the CFA showed an acceptable fit. The chi-square value was significant, χ2(1,013, N = 468) = 3,246.78, p < .001. The RMSEA was 0.069 showing acceptable model fit with a 90% confidence interval (CI) of [0.066, 0.071]. The CFI was .86, which did not meet the .90 threshold. The first initiated model analysis found nine strong correlations between indicators. For example, indicators “A comprehensive data collection system is in place that includes academic, behavioral, transition, and postsecondary outcomes data” and “Comprehensive data systems are used to evaluate secondary programs and transition services” were found to highly correlate. Therefore, adjustments for residual correlations were completed. The final model resulted in an improved and excellent fit. The chi-square value was significant, χ2(1,006, N = 468) = 2,548.957, p < .001. The RMSEA was 0.057 with a 90% CI = [0.054, 0.060]. Likewise, the CFI improved to .90. The standard factor loadings of each indicator to the domain were above .50 and significant at .001 level, indicating good convergent validity. Table 3 provides the standard factor loadings and standardized error variances across the seven domains.
Item Descriptive Statistics.
Estimated Variances for Domains.
Internal reliability analyses were also conducted using Cronbach’s alpha. The overall estimate of reliability for all 47 indicators produced a .97 coefficient alpha. The coefficient alpha estimates for each domain were transition planning (.88), transition assessment (.90), family involvement (.89), student involvement (.92), transition-focused curriculum and instruction (.88), interagency collaboration and community services (.89), and systems-level infrastructure (.88). These psychometric estimates indicated high overall reliability and high internal consistency for each domain (Green & Salkind, 2008).
Discussion
This study revised the QI-2 (Morningstar et al., 2012). The 47 indicators were grouped into seven domains based on a comprehensive review of research evidence and theory. The validity and reliability of the QI-2 was established through the comprehensive review and data reduction methods as well as utilization of a CFA, thereby establishing the QI-2 as a well-constructed and reliable measure for transition program evaluation. A brief review of the domains is described next.
Transition Planning
High-quality transition programs prioritize effective transition planning. Transition planning is the centerpiece for aligning transition services and developing postschool goals for youth with disabilities. The transition planning domain includes eight indicators, of which six align with transition requirements in IDEA (2004). The additional two indicators describe a transition planning approach that incorporates student- and family-centered methods, including addressing the needs of culturally diverse families. By putting students and families at the center of planning, transition becomes individualized, comprehensive, and collaborative (Bui & Turnbull, 2003; Keyes & Owens-Johnson, 2003; Menchetti & Garcia, 2003; Michaels & Ferrara, 2005).
Family Involvement
Family members play a consistent supportive role throughout a student’s life (Geenen, Powers, & Lopez-Vasquez, 2005; Hetherington et al., 2010; Morningstar, Turnbull, & Turnbull, 1995). The indicators, therefore, include family culture, six aspects of family involvement, and approaches to promote family involvement in the transition planning process. An essential strategy is to share transition-related information and resources with family members. The National Center on Secondary Education and Transition (NCSET; 2011) suggests practices for parent–professional collaboration, including (a) creating family mentoring program, (b) developing a survey to ask families how they would like to be involved, and (c) working with culturally specific community organizations that already have established relationships with families. Increased attention has been directed toward family-friendly IEP meetings, such as having the meeting outside of the school building, holding the meeting when and where the family is primarily available, as well as including a language interpreter or advocate during the meeting (Cho & Gannotti, 2005; Geenen et al., 2005). In addition, pre-IEP planning activities have emerged as a strategy to facilitate family input prior to the meeting. Boone (1992) showed the effectiveness of teaching parents concepts of transition planning and their roles during the IEP meetings through parent training. Family members often become the primary support network as the student moves into adult life; therefore, having them involved during transition planning is essential.
Student Involvement
Promoting student involvement in transition is an established evidence-based predictor of improved postschool outcomes (Test, Mazzotti, et al., 2009). Students with higher levels of self-determination were more likely to be engaged in employment and independent living after graduation (Morningstar et al., 2010; Wehmeyer & Schwartz, 1997). Therefore, quality transition programs provide curriculum and strategies that facilitate students’ skills in the areas of decision making, goal setting, problem solving, and self-advocacy. Evidence-based curricula identified as effective in teaching self-determination skills include (a) teaching academic, social, and job-specific skills (Moore, Cartledge, & Heckaman, 1995; Wolgemuth, Cobb, & Dugan, 2007); (b) self-monitoring for functional life skills (Mahon & Bullock, 1992; Todd & Reid, 2006); and, (c) Self-Determined Learning Model of Instruction (Wehmeyer et al., 2012). In addition, opportunities to make real-life choices should be provided to students in school (Brewer, 2006) and at home (Morningstar, 2006). Another quality indicator focuses on teaching students to direct their own IEP meetings by implementing published evidence-based curricula: (a) the Self-Directed IEP (Martin, Marshall, Maxson, & Jerman, 1996), (b) the Self-Advocacy Strategy (Van Reusen, Bos, Schumaker, & Deshler, 1994), and (c) Whose Future Is It Anyway? (Wehmeyer et al., 2004).
Transition-Focused Curriculum and Instruction
Transition-related knowledge and skills learned in secondary schools are now established predictors leading to successful adult roles (Test, Mazzotti, et al., 2009). Research has shown that students who are included in general education academic courses were more likely to be engaged in postsecondary education, employment, and independent living (Baer et al., 2003). To do this, quality programs incorporate effective instructional methods when teaching academics. An additional indicator of effective transition-focused instruction includes support for appropriate accommodations in the general educational settings.
A balance between teaching academics and transition-specific content has also been recognized as an indicator of quality. Ensuring that content essential for adult independence (e.g., career development, independent living skills) must be considered. Given that research supports enrollment in occupational and vocational education coursework as well as work-based learning experiences (Carter, Austin, & Trainor, 2012; Migliore, Mank, Grossi, & Rogan, 2007), career development and employment skills should be taught in school and work-based settings (Walker & Bartholomew, 2012). Likewise, teaching social and interpersonal skills should also be included in comprehensive transition programs. Quality indicators found in the QI-2 address the importance of independent living skills to support multiple adult roles as well as instruction in social and interpersonal skills.
Interagency Collaboration and Community Services
Interagency collaboration is a critical indicator of postschool success (Kohler, 1996; Test, Mazzotti, et al., 2009). School-business partnerships are recognized as an essential factor in student career development (Carter et al., 2009). Given that access to community agencies is a likely predictor of postsecondary education and employment (Bullis, Davis, Bull, & Johnson, 1995), establishing a process for communicating with outside agencies is critical. This includes establishing procedures for referring students to agencies prior to exiting school. Noonan, Morningstar, and Erickson (2008) identified 11 strategies that promote interagency collaboration at the local level. Building a collaborative and trusting relationship among schools and agencies is essential. It includes discussing anticipated service needs of students as well as developing interagency agreements to determine how information will be exchanged, resources will be shared, and services coordinated. Meeting with students and families also promotes interagency collaboration and, therefore, quality programs provide parents with accurate and timely resources about services.
Systems-Level Infrastructure
Systems-level infrastructures support efficient and effective delivery of transition-focused education and services (Kohler & Rusch, 1996). Program infrastructures are required to maintain comprehensive support for effective transition programs and policies (Stodden & Leake, 1994). Emerging issues in schools such as inclusion, dropout, and cultural diversity must be addressed within secondary and transition programs. This includes co-teaching in general education, programmatic indicators of inclusive settings, and supporting both general educators and students with disabilities in inclusive classrooms (Murawski & Hughes, 2009; Scruggs, Mastropieri, & Mcduffie, 2007). Incorporating dropout prevention models for students at risk of dropping out of school is another quality indicator that promotes student engagement leading to positive outcomes (Christenson & Thurlow, 2004). Evaluating the impact of transition programs on students’ learning is another critical indicator of systems-level infrastructures leading to positive outcomes. This requires supporting data-based management systems to assess achievement outcomes as well as behavioral, nonacademic, and transition competencies. Finally, school districts that support a professional responsible for coordinating transition programs and services significantly enhance effective transition programs (Morningstar & Clavenna-Deane, 2014).
Using the QI-2 for Change
This study confirms that the QI-2 measurement is a valid, reliable instrument that can be used to strengthen the quality of transition programs. Thus far, the QI-2 has been used by both individuals and groups to identify strengths and gaps in transition programs supporting youth with disabilities. Practitioners have used the QI-2 for improvement to their own programs. Individuals who have been introduced to the QI-2 through a training, workshop, or the website (www.transitioncoalition.org) have completed it to identify specific professional development needs they may have such as identifying evidence-based curricula for implementation.
Preservice practitioners have also used the QI-2 to first gauge the quality and comprehensiveness of transition programs in which they work (Morningstar & Clavenna-Deane, 2014). Once participants complete the QI-2, they identify a specific domain and often an individual indicator within the domain for improvement. For example, a participant who prioritized the domain “transition assessment” as a critical area for improvement, and the indicator “A wide variety of formal and informal transition assessments are available to use with students” as the specific target, would review research related to this improvement focus, and identify, obtain, and organize transition assessments for use in his school transition program.
District and community transition teams have also used aggregated data from the QI-2 as a data-based method for planning. District teams consisting of school personnel (e.g., teachers, administrators, school psychologists, related services providers, guidance counselors), community agencies and organizations (e.g., vocational rehabilitation counselors, intellectual and developmental disabilities agency personnel), as well as other stakeholders (e.g., parents) have completed the QI-2. This use of the QI-2 was primarily done in preparation for attending training and technical assistance events (e.g., transition summer institute) or as a gap analysis to identify specific areas for team-based program improvement. When used in this way, team members completed the QI-2 independently and received an aggregated report of their results by domain and indicator. Teams then worked together to identify a domain and perhaps one or two specific QI-2 indicators around which to develop an action plan for improvement.
Finally, the QI-2 has been used by state departments of education (SEAs) to examine aggregated data as a method of data analysis for informing comprehensive systems of professional development. In this way, SEAs determined a critical domain area based on statewide data to target for a broad professional development and technical assistance implementation plan. Furthermore, in some cases, the SEAs disaggregated data by specific regions of the state to develop a regionalized picture of transition needs to prioritize improvements. In a few instances, SEAs used the QI-2 to report change over time, for example, in annual performance reports (APR).
Conclusion
Effective transition programs must be designed to address a wide range of transition education and services for students with disabilities to ensure positive adult outcomes. Evaluating the quality of transition programs is necessary for ongoing enhancements and improvement. Using a highly reliable and valid measure, such as the QI-2 (Morningstar et al., 2012) allows practitioners, programs, districts, and states to evaluate their progress toward quality transition practices. The QI-2 can be used to guide practitioners and stakeholders to evaluate the current status of their transition programs, and identify the most needed areas for improvement and plan for change. It can monitor the transition program over time, identifying the level at which the program is performing as well as areas of needed enhancements.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.