Abstract
This study pilot tested an employability skills rubric (ESR) to support the summary of performance (SOP) requirement in Individuals With Disabilities Education Improvement Act. The ESR consisted of three performance areas: (a) Responsibility, (b) Problem-Solving Skills, and (c) Interpersonal Skills. The study sought to establish (a) interrater reliability, (b) the degree of agreement among raters, and (c) assessment of feedback regarding the ESR. The instrument was pilot tested with 19 local educational agencies. Most items had correlations in the moderate range and large effect sizes. There was a high level of internal consistency for each rubric and respondents indicated that the ESR identified important skills and appropriate measures related to the SOP.
Transition planning and successful postschool outcomes to employment continue to challenge the field. Transition from secondary settings to successful postschool outcomes for individuals with disabilities has remained a topic of concern (Test, Aspel, & Everson, 2006). The National Longitudinal Transition Study–2 (NLTS2) data reported a decline in the level of employment (62%–56%) for youth with disabilities between the years of 1990 and 2005 (Newman, Wagner, Cameto, Knokey, & Shaver, 2010). More recent NLTS2 data found that 71% of youth with disabilities were engaged in paid employment as reported from Wave 4 interview (2007) data (Stanford et al., 2011). These findings align with the employment rate reported for the general population same-age peers (Stanford et al., 2011). Current economic conditions, punctuated by high levels of unemployment, have heightened concerns about overall employment. Unemployment of youth (ages 16–19) and young adults (ages 20–24) have been a concern in the current recession (Borbely, 2009), as individuals with disabilities have a higher probability of experiencing financial difficulties during their early years (first 2 years of employment or postsecondary education) of transition to adult life (Kochhar-Bryant & Izzo, 2006).
Promoting postsecondary success for students exiting high school has been advocated as a new reality for American youth (Gray, 2000). One of Gray’s Five Premises for Success specifically states “every student should graduate from high school with a postsecondary plan that has a high probability of success” (p. 124). The Individuals With Disabilities Education Improvement Act (IDEA) of 2004 (PL 108-446) addresses the critical issue of postsecondary success for youth with disabilities in transition services and school-based individualized education program planning. Specifically IDEA established transition services as follows: (a) a coordinated set of activities for a child with a disability that—(1) Is designed to be within a results-oriented process, that is focused on improving the academic and functional achievement of the child with a disability to facilitate the child’s movement from school to post-school activities, including postsecondary education, vocational education, integrated employment (including supported employment), continuing and adult education, adult services, independent living, or community participation. (§300.43)
One tool designed to help students achieve their postsecondary goals is the summary of performance (SOP). IDEA requires an SOP for a child with a disability at the secondary level whose eligibility terminates because of graduation with a regular diploma or reaching the maximum age of eligibility for special education services as established by State law: Local educational agencies [LEAs] must provide the child with a summary of the child’s academic achievement and functional performance, which shall include recommendations on how to assist the child in meeting the child’s postsecondary goals. (§300.305(e)(3)) must provide the . . . ” are correct. (Federal Register, 2006)]
While IDEA requires LEAs to provide students with an SOP to assist them in the pursuit of their postsecondary goals, it does not mandate the specific content or information to be included in the SOP. This is left to the judgment of states and LEAs to determine (U.S. Department of Education [DOE], 2010). Madaus, Shaw, Miller, Banerjee, and Vitello (2011) found that nationally, states and LEAs varied in their implementation of the SOP. Essential elements of the SOP include academic skills, social skills, independent living skills, career and vocational data, work experience, and self-determination skills (Lamb, 2007). In a review of the SOP literature, Richter and Mazzotti (2011) found that employment and postsecondary education were two postschool adult outcomes of interest. The authors recommended state education agencies create efficient SOP forms that address the federal requirements while providing adequate training to LEAs. Kochhar-Bryant (2007) indicated employers and adult providers need relevant information from school-based transition assessments. The SOP should be organized as a “clear, understandable, and usable document that facilitates student self-determination” (Kochhar-Bryant, 2007, p. 162) toward employment and postsecondary education.
Madaus et al. (2011) suggested that students are disadvantaged when SOP documentation fails to address the mandate of IDEA. The SOP can provide an important link to Section 504 of the Rehabilitation Act and the Americans With Disabilities Act (ADA) by providing employers and postsecondary education with key information concerning disability eligibility and needed accommodations (Dukes, Shaw, & Madaus, 2007; Shaw, 2006). Leconte (2006) indicated employability skills recommended by the Secretaries Commission on Achieving Necessary Skills (SCANS) can also be an important part of the SOP, providing an accurate profile of the student related to employability. Madaus, Bigaj, Chafouleas, and Simonsen (2006) indicated that within the SOP “the present level of performance section offers a chance to integrate data to provide a current picture of the student’s strengths and needs” (p. 92). These authors suggested that a student’s level of self-advocacy skills, attendance, behavior, production, and quality of work are all important areas to be covered in the SOP related to employment. Other areas of importance included time management, problem solving, communication, access help/support, social skills, and personal safety (Madaus et al., 2006).
Transition services, as mandated by IDEA, are intended to assist students with disabilities during their school years with “an array of activities aimed at increasing employability” (Test et al., 2006, p. 223). Employment skills needed for successful workforce participation require educators to “teach future employees how to make decisions, how to solve problems, how to learn, how to think a job through from start to finish, and how to work with people to get the job done” (Carnevale, Gainer, & Villet, 1990, p. 237). Several authors have addressed the importance of transition services in assisting youth with disabilities in moving to postschool employment (Flexer, Baer, Luft, & Simmons, 2008; Harvey & Koch, 2004; Sabornie, & deBettencourt, 2009; Test et al., 2006; Wehman, 2001).
The employability skills rubric (ESR) represents one midwestern state’s early efforts to develop a skills-based performance measure focused on research-based employability skills needed for employment. The ESR was developed as a rating instrument aimed at assessing employability skills as part of the SOP documentation. Employability skills performance assessment is a valuable part of the SOP that can facilitate successful movement toward postschool employment. The intent of this study was to validate the instrument. Therefore, the purpose of this study was to determine whether the ESR could be used as a skills-based performance rubric to support a student’s SOP documentation as a means of promoting successful transition to postschool goals.
Method
Participants
The state’s DOE served as the facilitator of the ESR pilot study with LEA recruitment and dissemination of study materials. Email Listserv communication was used to recruit participating LEAs. This was an open call for interested LEAs in being involved with the pilot study. LEA recruitment included a letter of invitation from the state’s DOE Director of Exceptional Learners sent to all special education directors across the state representing the state’s 362 school corporations (LEAs). The ESR was explained and sites were invited to volunteer to participate. Packet materials were included as part of the recruitment process for sites indicating a willingness to participate. LEAs participating in the ESR pilot study were asked to (a) identify five students per high school, (b) assign two raters familiar with the identified student to rate student performance using the ESR instrument, and (c) encourage LEA raters to provide feedback about the utility of the ESR instrument as part of the SOP process.
Nineteen LEAs agreed to participate in the pilot study. Participants included special education teachers, general education teachers, school psychologists, speech therapists, para-educators, and others (e.g., administrators, guidance counselors). A six-digit coding sequence was used to manage study participation and facilitate data analysis. The code was developed by the researchers but was administered by the state’s DOE. None of the LEA school sites, school personnel raters, or students selected for ESR rating purposes was known to the researchers.
There were 194 rater respondents who participated in the ESR study. From these participants, there were 178 rater respondents with usable data as identified by matched pairs for data analysis (n = 89 pairs). Respondents identified as ESR raters for this study represented the following demographic characteristics: (a) the majority of study respondents were female (77%), (b) most respondents indicated that they were 51 years old or older (36.3%), (c) almost half (47.5%) identified themselves as special education teachers, and (d) several respondents (13.6%) indicated their position as other (see Table 1). Position types most often identified under “other” included transition services coordinator/specialist, student services coordinator, job coach, school corporation vocational coordinator, or school principal/assistant principal. Data reported in Table 1 represent those providing demographic information as part of the ESR study. There were two categories with high levels of nonresponse (gender and age).
Employability Skills Rubric Pilot Study Respondent Rater Demographic Characteristics
Note: Percentages are based on those respondents who provided information concerning demographic questions SPCED = Special education.
The study was intended to provide insight into ESR instrumentation and “usability” concerning SOP reporting. Students identified for ESR ratings were selected by their high schools and were anonymous to researchers. Students most often identified by LEAs as participants included (a) 46.1% students identified with a specific learning disability (LD), (b) 19.9% students identified with an emotional disability (ED), (c) 18.2% students identified with a cognitive disability (CD), (d) 6.8% students identified with autism, and (e) 9.0% identified in all other disability categories. Other student demographic data included (a) 68.5% students who were male, (b) 31.5% students who were female, (c) 88.8% students who were Caucasian, (d) 11.2% students who were all other races, (e) 15.7% students who were age 16, (f) 33.1% students who were age 17, (g) 25.3% students who were age 18, and (h) 25.9% students who were ages 19 to 21 years old.
Instrumentation
The state’s DOE in the spring of 2004 created a transition work group made up of individuals from LEAs, institutions of higher education, community agencies, parent groups, and DOE Center for Exceptional Learners. All members were identified as key transition stakeholders from across the state. The intent of this group was to explore and advise DOE on transition services and related issues for youth with disabilities as they transitioned from school to postsecondary adult life. This state-level transition work group became known as the “Employability Work Group” (EWG). One of the tasks of the EWG was to develop an assessment measure to support DOE efforts related to SOP. The EWG created a subgroup to develop the ESR. The development of the ESR was actively supported by DOE working with key personnel from the Indiana state improvement grant (INSIG). The ESR was developed with significant stakeholder input and research. The ESR was designed to provide soft skills summative ratings and analysis of individual performance in three key areas associated with employability, including Responsibility, Problem-Solving Skills, and Interpersonal Skills.
Several sources were reviewed to develop the ESR. Primary resources included but were not limited to the following: (a) National Work Readiness Credential (2006), (b) ACT WorkKeys (2006), (c) Secretary’s Commission on Achieving Necessary Skills (SCANS) Report (U.S. Department of Labor, 1991), and (d) Life-Centered Career Education: A Competency-Based Approach (Brolin, 1997). In addition, ESR design incorporated transition concepts emphasized in Life Beyond the Classroom: Transition Strategies for Young People With Disabilities (Wehman, 2001) and Integrating Transition Planning into the IEP Process (West et al., 1999).
Broad themes represented in workforce development and transition literature concerning employability skills included communication, listening, expression, and speaking (SCANS, 1991; West et al., 1999; WorkKeys, 2006). Also included, related to employability, were (a) the ability to follow directions, (b) ability to work with others, (c) work habits, (d) organizational skills, (e) time management, (f) appropriate work behaviors, (g) getting along with employer/coworkers, and (h) self-management (SCANS, 1991; Wehman, 2001; West et al., 1999; WorkKeys, 2006). Thinking skills, interpersonal skills, and personal qualities as suggested by SCANS (1991) and the National Work Readiness Credential (2006) were relevant. These included (a) decision making, (b) problem solving, (c) conflict resolution, (d) knowing how to learn, (e) reasoning, (f) responsibility, and (g) self-esteem among others. These skill areas were represented in the ESR as part of the rubric measures.
The formative work done by Brolin’s (1997) Life-Centered Career Education Competencies was also represented in the ESR. Areas related to Personal-Social Skills and Occupational Guidance and Preparation with specific competencies identified under (#10) Achieving Self-Awareness, (#11) Acquiring Self-Confidence, (#12) Achieving Socially Responsible Behavior, (#13) Maintaining Good Interpersonal Skills, (#15) Making Adequate Decisions, (#16) Communicating With Others, (#19) Exhibiting Appropriate Work Habits and Behavior, (#20) Seeking, Securing, and Maintaining Employment, (#22) Obtaining Specific Occupational Skills with many of the associated subcompetency areas were directly related to the ESR rubric areas Responsibility, Problem-Solving Skills, and Interpersonal Skills.
The ESR design used a rubric rating scale based on a zero to five (0–5) with 0 indicating the skill was not evident and 5 indicating the highest level of skill/proficiency development. The rating scales were designed to meet the specific measures of the skill/performance being assessed for each of the three ESR skill/performance areas. The final instrument was sent to seven reviewers with expertise in transition (LEA Directors of Special Education, Transition Coordinators, Vocational Rehabilitation (VR), Institutions of Higher Education Special Education faculty, and Indiana DOE (IDOE) Center for Exceptional Learners personnel) within the state for comment on content and construct validity prior to the ESR pilot study. Revisions were made to ESR items concerning readability and jargon that might be confusing based on reviewers’ feedback. ESR items were edited providing clarification, limiting jargon, and focusing on nonacademic employability skills.
A major focus of the EWG was to develop an entry-level work skills assessment identifying “soft skills” necessary for any type of entry-level employment. The EWG subgroup charged with designing the ESR emphasized work readiness and employability skills from workforce development resources and transition skills from the transition literature. The resulting ESR instrument was a compilation of knowledge and skills organized under three central domains (responsibility, problem solving, interpersonal skills). These areas of employability skills were deemed critical for employment by EWG stakeholders (e.g., VR counselors, disability service providers, LEA transition coordinators, DOE personnel). The final ESR pilot tested included the following skill/performance areas: (a) ESR Rubric 1, Responsibility—which had eight rating items, (b) Rubric 2, Problem-Solving Skills—which had eight rating items, and (c) Rubric 3, Interpersonal Skills—which had seven rating items.
Examples of ESR rating items and scales for these three rubrics include the following: (a) ESR Rubric 1, Responsibility—Manages time: Level 0 = not evident, Level 1 = recognizes who can help accomplish a task, Level 2 = requests and accepts assistance from appropriate sources, Level 3 = identifies resources available to support work, Level 4 = requests resources to develop self-sufficiency, and Level 5 = identifies and requests resources to enhance organizational productivity; (b) ESR Rubric 2, Problem-Solving Skills—Defines a problem: Level 0 = not evident, Level 1 = acknowledges a problem that has been made apparent, Level 2 = recognizes when there is a simple problem in a familiar setting, Level 3 = provides some information related to a problem situation, Level 4 = provides a detailed description of a complex problem, and Level 5 = provides a detailed description of a complex problem and the related circumstances; and (c) ESR Rubric 3, Interpersonal Skills—Uses effective communication skills to interact with others: Level 0 = not evident, Level 1 = acknowledges what others have to say, Level 2 = initiates interactions and engages in discussions, Level 3 = alters style or manner of communication to be appropriate for given situations, Level 4 = evaluates the effectiveness of communication and corrects misunderstanding, and Level 5 = develops positive relationships and maintains effective communication with others.
The ESR pilot study included a section of six open-ended questions. The open-ended questions included an opportunity for a more in-depth inquiry into the usability of the ESR for SOP purposes. Respondents were asked to provide feedback concerning the ESR addressing the following questions: (a) comprehensiveness of employability skills/measures, (b) nonacademic skills which should be included, (c) items or skills that were not needed or were unimportant, (d) terms confusing to transition stakeholders, and (e) value of ESR to include in SOP reporting.
Data Collection Procedures
Data for the ESR pilot study were collected electronically utilizing inQsit, a software data collection and information management tool. Data collection used a coding sequence where each student selected by his or her LEA was assigned a unique identifier starting with #1 through #5, and each individual student’s high school personnel rater was assigned a code of #1 for the first rater and # 2 for the second rater by the participating LEA. The student and school personnel completing the ESR ratings were completely anonymous to the researchers but coding allowed the researchers to match students and raters for accurate data analysis. The 19 LEA sites were provided with PDF files of the ESR and a link to inQsit from the state’s DOE webpage. Study participants were asked to log in using their assigned code, rate student performance levels using the ESR instrument, and provide additional feedback to open-ended questions related to usage and comprehensiveness of the ESR instrument/items for the purposes of SOP reporting.
Data Analysis Procedures
The pilot study for the ESR provided insight into interrater reliability, internal consistency, and LEA respondents’ feedback on the ESR instrument as part of the SOP documentation considered as part of the SOP reporting process. Cronbach’s alpha was used to assess internal consistency as a measure of reliability for reach ESR rubric. Cronbach’s alpha provides an estimate of internal consistency reliability related to an item on a measure with all other items on a measure (Gay & Airasian, 2000). In general terms, Cronbach’s alpha for individual measures should be .80 or higher. Intraclass correlation (ICC) was used as a measure of interrater reliability because it measures consistency of a set of data when it has multiple groups. ICC coefficients can range from 0.0 to 1.0, with a higher ICC statistical measure indicating a small level of variance (more agreement) between raters’ scores. ICC coefficient was used to assess the level of rater agreement for ESR rubrics as single-measure and multiple-measures estimates.
Additional measures used to assess the level of interrater agreement for ESR item analysis included Pearson’s r correlation coefficients, Cohen’s kappa statistics, and itemized descriptive percentage of agreement (obtained from cross-tab analysis). In general terms, Pearson product–moment correlation coefficient (r) measures the linear relationship between two variables X and Y. Cohen’s kappa measures interrater agreement. Pearson’s r and Cohen’s kappa (κ) statistics range from −1.0 to +1.0, with a Pearson’s r of +1.0 indicating a perfect positive linear relationship and a Pearson’s r of −1.0 being a perfect negative linear relationship (Jaeger, 1993). Gay and Airasian (2000) suggested that correlation coefficient ranges of ± .35 to .65 show moderate relationships, but that a correlation coefficient of ± .50 is typically not adequate for group or individual predictions, “although a combination of several variables in this range might yield a reasonably satisfactory prediction” (p. 324). Cohen’s kappa interpretations indicate 0.0 to 0.20 slight agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and 0.81 to 1.00 almost perfect agreement. Kappa controls for chance agreement where other measures do not. The reader is cautioned when interpreting Cohen’s kappa results that treat data as nominal and do not fully account for ratings such as those used in rubrics that have an order or level (continuum/progression; Landis & Koch, 1977). As such, the Cohen’s kappa statistics presented here are used as a guide to supplement the overall analysis reported. Data for the ESR study were analyzed using the Statistical Program for Social Sciences (SPSS 14.0). The open-ended questions were summarized as descriptive data.
Results
Internal Consistency for the ESR
Reliability concerning internal consistency for each ESR was reported using Cronbach’s alpha. The Cronbach’s alpha statistics for Rubric 1, Responsibility, was 0.934, Rubric 2, Problem-Solving Skills, was 0.957, and Rubric 3, Interpersonal Skills, was 0.918. Each rubric exceeded basic Cronbach’s alpha levels (≥.80) suggested by Gay and Airasian (2000) indicating a high level of internal consistency and reliability among identified rubric items for each of the specific rubrics for the ESR.
Interrater Reliability for the ESR
Table 2 reports ICC coefficients for each ESR rubrics. The results included data reported as single-measure and average-measure (two raters) estimates with the associated 95% confidence levels with lower and upper bounds. The Rubric 1, Responsibility was found to be statistically significant with a large effect size (F = 3.48, p < .001). ICC results for single-measure estimates were (r + .554) compared with those of average-measure estimates (two raters; r ≠ .713). The Rubric 2, Problem-Solving Skills, was found to be statistically significant with a large effect size (F = 4.76, p < .001). ICC results for single-measure estimates were (r + .654) compared with those of average-measure estimates (two raters; r ≠ .790). The Rubric 3, Interpersonal Skills, was also found to be statistically significant with a large effect size (F = 5.56, p < .001). ICC results for single-measure estimates were (r + .698) compared with those of average-measure estimates (two raters; r ≠ .822). Together, the results indicated solid interrater reliability for the ESR rubrics with stronger ICC levels reported when using average-measure estimates compared with single-measure estimates.
Pilot Data Results for Employability Skills Rubrics 1-2-3 Intraclass Correlations
Note: r − is the single-measure estimate and r ≠ is the average-measure estimate.
p < .05. **p < .01. ***p < .001.
Rubric 1: Responsibility
The interrater reliability results for itemized ESR 1, Responsibility, are reported in Table 3. Six of the eight items on this rubric had Pearson’s r correlations that were in the moderate range and all items had large effect sizes. Item 10, Manages Emotions (r = .59, p < .001), had 43.2% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .30, p < .001). Manages Personal Resources (Item 11, r = .52, p < .001) had 39.1% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .24, p < .001). Cohen’s kappa statistic indicated fair agreement (κ = .25, p < .001) for Item 12, Follows Policies and Procedures (r = .49, p < .001), with 39.8% rater agreement. Item 14, Utilizes Resources (r = .53, p < .001), had 39.1% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .21, p < .001). Manages Materials and Equipment (Item 15, r = .38, p < .001) had 33.3% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .15, p < .01). Cohen’s kappa statistic indicated slight agreement (κ = .18, p < .01) for Item 16, Manages Goals (r = .44, p < .001), with 35.3% rater agreement.
Pilot Data Results for Employability Skills Rubric 1: Responsibility
p < .05. **p < .01. ***p < .001 indicator of effect size.
Two items had Pearson’s r correlations that were not within the moderate range (small). Item 17, Takes Responsibility for Learning (r = .34, p < .01), had 34.1% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .14, p < .05). Item 13, Manages Time (r = .32, p < .01), had 33.7% rater agreement and a Cohen’s kappa statistic indicating slight agreement (κ = .11, p > .05 = ns). The Cohen’s kappa data reported for Item 13 was found to be not significant.
Rubric 2: Problem-Solving Skills
Table 4 reports the interrater reliability results for itemized ESR 2, Problem-Solving Skills. All eight items on this rubric had Pearson’s r correlations in the moderate range and had large effect sizes. Item 18, Recognizes Problems (r = .57, p < .001), had 41.9% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .27, p < .001). Defines a Problem (Item 19, r = .48, p < .001) had 41.4% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .20, p < .001). Cohen’s kappa statistic indicated fair agreement (κ = .23, p < .001) for Item 20, Analyzes Cause and Effect (r = .41, p < .001), with 38.6% rater agreement. Item 21, Uses Information to Solve Problems (r = .68, p < .001), had a Cohen’s kappa statistic indicating fair agreement (κ = .30, p < .001) with 46.6% rater agreement. Analyzes Potential Strategies to Solve Problems (Item 22, r = .60, p < .001) had 48.8% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .33, p < .001). Cohen’s kappa statistic indicated fair agreement (κ = .27, p < .001) for Item 23, Considers Solutions to Problems (r = .42, p < .001), with 42.4% rater agreement. Acts to Solve Problems (Item 24, r = .48, p < .001) had 32.6% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .15, p < .01). Item 25, Evaluates Solutions to Problems (r = .47, p < .001), had 33.7% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .17, p < .01).
Pilot Data Results for Employability Skills Rubric 2: Problem-Solving Skills
p < .05. **p < .01. ***p < .001 indicator of effect size.
Rubric 3: Interpersonal Skills
The results regarding interrater reliability for itemized ESR 3, Interpersonal Skills, are reported in Table 5. Pearson’s r correlations were found to be in the moderate range and all items had large effect sizes. Item 26, Appreciates the Perspectives of Others (r = .57, p < .001), had 38.4% rater agreement and Cohen’s kappa statistic indicating fair agreement (κ = .23, p < .001). Respects Property and Privacy (Item 27, r = .57, p < .001) had 44.3% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .20, p < .01). Item 28, Accepts Guidance and Directives From Others (r = .49, p < .001), had 39.1% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .18, p < .01). Cohen’s kappa statistic indicated fair agreement (κ = .21, p < .001) for Item 29, Communicates Information (r = .48, p < .001), with 37.2% rater agreement. Item 30, Uses Effective Communication Skills to Interact With Others (r = .43, p < .001), had 36.4% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .20, p < .001). Cohen’s kappa statistic indicated fair agreement (κ = .23, p < .001) for Item 32, Manages Conflict (r = .63, p < .001), with 40.1% rater agreement.
Pilot Data Results for Employability Skills Rubric 3: Interpersonal Skills
p < .05. **p < .01. ***p < .001 indicator of effect size.
Item 31, Contributes as a Member of the Group (r = .52, p < .001), had 29.5% rater agreement and Cohen’s kappa statistic indicating slight agreement (κ = .10, p > .05 = ns). The Cohen’s kappa data reported for Item 31 was found to be not significant.
Results of Open-Ended Questions and Feedback Concerning ESR Items
The pilot study provided respondents an opportunity to give feedback regarding the ESR. There were 110 respondents who provided feedback on the ESR study’s open-ended questions. Respondents agreed that ESR items identified a comprehensive collection of employability skills and measures (89% [n = 98] of those providing responses on this item). Respondents who did not agree (11% [n = 12] of respondents) indicated that the ESR should also address (a) the needs of GED students, (b) postsecondary academic planning, and (c) more items regarding social skills and personal interactions.
Respondents felt ESR rubrics did not need to identify additional nonacademic skills to be included in the collection of skills and measures developed in the ESR (86% [n = 94] of those providing information on this question). Respondents who did not agree (14% [n = 16] of respondents) suggested that the ESR should include the following additional elements: (a) self-determination; (b) vocational training for 2 years; (c) attendance; (d) physical skills; (e) communication, especially in group settings; (f) behavior in terms of work completion; and (g) conflict with peers (conflict-resolution skills).
Study respondents agreed the ESR rubrics did not contain any skills and/or items that were not needed or were perceived to be unimportant (98% [n = 107]). Respondents agreed ESR items contained terms and information that would not be confusing to transition stakeholders (i.e., parents, students, employers, and teachers). In all, 85% of those providing information on this question (n = 92) felt that ESR syntax and terminology were appropriate for this audience. Respondents who did not agree (15% of respondents [n = 18]) indicated the ESR should revisit some item wording, jargon, item clarity, and simplification. The open-ended feedback provided important constructive feedback related to the ESR.
Overall, respondents agreed the items and the collection of employability skills and measures developed for the ESR had value as part of the SOP report (85% [n = 92] of those providing responses on this question). Respondents’ feedback for those who did not agree indicated that further development of the current version of the ESR was potentially needed to improve the measure as a meaningful part of the SOP (15% of respondents [n = 18]).
Discussion
The ESR was developed as an assessment tool related to employment skills to support SOP reporting documentation as deemed important by a state-level transition group from one midwestern state. The study sought to establish internal consistency, interrater reliability, raters’ degree of agreement, and respondents’ feedback regarding the ESR for SOP purposes. The ESR included three specific rubrics that assessed performance areas focusing on responsibility, problem-solving skills, and interpersonal skills. Results indicated all three ESR rubric had high levels of internal consistency (Cronbach’s alpha) and strong ICC. While these results are promising, data indicated that each rubric had higher correlations with more than a single rater; two or more raters are preferred. Most ESR rubric items had Pearson’s correlations (r) in the moderate range with percentages of rater agreement in the 40% range. Cohen’s kappa (κ) analysis indicated fair agreement among raters for most ESR items (caution noted with reported κ). Effect size for all items were considered large and respondents indicated the ESR identified important skills and appropriate measures related to employability. Data suggested skills and measures associated with the ESR can make a valuable contribution to the SOP. Some respondents suggested additional clarification or revision on some items within certain rubrics items of the ESR. Feedback to open-ended questions and statistical analysis indicated the ESR is viable but the current version could potentially be improved. There were suggestions from the open-ended questions (i.e., clarity, wording, and reducing jargon) that could improve the ESR as a measure of employability for SOP documentation.
There were positive results concerning the ESR as a measure of employability skills. The findings suggested most ESR items were important measures associated with the domains of responsibility, problem-solving skills, and interpersonal skills. In addition, findings pointed to possible limitations from some ESR components (e.g., Item 13, Manages Time; Item 17, Takes Responsibility for Learning; and Item 31, Contributes as Member of the Group). The data from these items suggest that either the items and/or the rating scales were problematic or that ESR raters needed to be better trained in using the ESR. Further revisions to the current version of the ESR are potentially needed to establish the ESR as an effective measure of employability skills related to the SOP. Overall, feedback suggested that the intent and purpose of the ESR is viable and has the potential to make a positive contribution to the SOP process.
The ESR is an assessment of employability skills designed as part of the SOP that can help educators meet the IDEA requirement of assisting students toward more positive postschool employment outcomes (Gray, 2000; Newman et al., 2010; Richter & Mazzotti, 2011; Shaw, 2006; Stanford et al., 2011). As suggested by Leconte (2006) and SCANS (1991), the ESR included soft skills essential for youth with disabilities seeking entry-level employment. Next, as suggested by Kochhar-Bryant (2007), Madaus et al. (2006), and Madaus et al. (2011), the ESR can provide transition stakeholders, including students, with assessment of employment skills in a usable format for SOP purposes. Finally, as recommended by Dukes et al. (2007), Kochhar-Bryant (2007), Leconte (2006), and Shaw (2006), the SOP, including elements like the ESR, can provide an accurate profile of the students’ skills and can be used to inform employers about performance, work-related skills, and possible needs for accommodations.
Limitations and Suggestions for Future Research
First, the ESR represents a compilation of multiple-source data concerning employment, employability, and job/work skills as developed by the one midwestern state. As a result, utility may be limited by the beliefs, perspectives, and vested interest of the group who created the ESR as a measure of employability skills in support of SOP documentation designed to assist individuals with disabilities as they transition to postsecondary employment.
Second, the ESR pilot study included only 19 participating LEAs with a relatively small number of matched-pairs ESR raters (n = 89) used in the analysis. Students and LEA raters identified for inclusion in the ESR pilot study were selected by LEA settings as recruited by the state’s DOE with no direct involvement from the researchers. Student selection and subsequent LEA rater participants for the ESR pilot study was not directed by the researchers; thus all participants (students selected to be rated using the ESR and LEA raters who best knew these students) were completely anonymous to the researchers and independent by study design (limiting selection bias). As a result, the pool of students and associated LEA raters identified for the ESR may have influenced study results. While there were study requirements that a) students identified for participation needed to be of transition age, b) their LEA raters needed to be knowledgeable of those students selected for participation, and c) ESR raters be able to accurately assess the student’s employability skill level, no formal training was provided to LEA participants beyond the information provided by the state’s DOE in LEA recruitment materials and those posted to inQsit concerning the ESR study process. The ESR pilot test needed to include more LEAs and rater respondents to ensure breadth and depth of instrument and item analysis. In future research, it is recommended that researchers be more directly involved in the implementation phase of the project (i.e., LEA sampling, specified student participation, identified LEA respondent raters). Working directly with LEA recruitment, student participant identification, and school personnel ESR rater selection can potentially enhance study findings through more specifically defined research methods. Research is also needed to determine the level of preparation and training of LEA personnel needed to provide accurate usage of ESR measures as part of the pilot study process. Researcher involvement with LEA personnel providing adequate comprehensive training in the usage of the ESR is important (Richter & Mazzotti, 2011). Training could be conducted in webinar format and should include informational modules for LEA support. An introduction webinar could be designed to provide background information on the SOP and the specific design of the ESR. Training should emphasize the ESR is a measure of employability skills focused on soft skills important for employment. Modules should be developed to address each specific ESR rubric with examples and explanation of the identified skill performance and various measures. Student vignettes with utility of rating examples are suggested.
Third, study results report multimeasures of interrater reliability of which the Cohen’s kappa results reported are done with caution. Kappa statistics are presented as supplemental in support of the overall interrater analysis. Results focused on Pearson’s r correlations and raters’ percentage of agreement are measures that more specifically address ESR item analysis. Specific recommendations made from open-ended feedback need to be considered in potentially revising the ESR. These include (a) adding items concerning attendance, work completion, physical skills, and social skills—peer interactions; (b) simplification of wording and reducing jargon; and (c) clarification of rating item(s) with explanation and examples. Revising the ESR and conducting a more comprehensive pilot test of the instrument can be a valuable next step.
Fourth, further research needs to focus on validating the ESR and addressing the utility of the instrument with transition stakeholders (e.g., youth with disabilities, LEA transition personnel, adult agency providers [VR, WorkOne], and employers). Finally, research efforts need to confirm the ESR has the potential of meeting the IDEA mandate in facilitating movement to postschool employment for youth with disabilities.
Implications for Practice
IDEA specifies an SOP is required to assist students in meeting their postsecondary goals but provides limited guidance in that process. As a result, states have flexibility in determining the content of their SOP and establishing appropriate SOP elements. States and LEAs should consider including a rubric-based skills performance rating measure similar to the ESR to appropriately support employment outcomes with “recommendations on how to assist the child in meeting the child’s postsecondary goals” (§300.305(e)(3)).
The ESR is supported by research within the transition literature (Brolin, 1997; Test et al., 2006; Wehman, 2001) and workforce development literature (Carnevale et al., 1990; Gray, 2000; SCANS, 1991). The ESR can serve as performance/skills documentation of a student’s abilities, as articulated in the transition component of the IEP, that Test et al. (2006) suggested needs to represent a multitude of transition activities designed to increase one’s employability. The ESR addresses important issues in support of a student’s movement to postschool employment as articulated by Kochhar-Bryant (2007), Lamb (2007), Leconte (2006), and Madaus et al. (2006). The intent of this study was to validate the instrument and propose that other states and/or LEAs consider using the ESR, adapting the tool for their purposes as a part of their SOP documentation and reporting process.
Current economic conditions indicate continued challenges for those involved with the transition process, especially concerning youth with disabilities seeking employment in today’s job market. The ESR can provide youth with disabilities transitioning to postsecondary employment “added value” related to the SOP because the ESR is a skills-based employability assessment directly focused on the types of employability skills required in today’s workforce. The ESR can provide transition stakeholders with performance measures related to essential employment skills suggested by Carnevale et al. (1990) and SCANS (1991).
The SOP, incorporating an ESR component, provides an appropriate vehicle for youth with disabilities in meeting Gray’s (2000) Premises for Success, indicating all students leave high school with a postsecondary plan geared toward success. States or LEAs wanting to adopt the ESR as a measure of employability skills for SOP purposes should consider having a transition stakeholders group similar to the EWG review the instrument. States and LEAs that adopt the ESR should adapt it for their needs and should consider an external review by a panel of experts with pilot testing of their version of the instrument.
Recommendations made by Richter and Mazzotti (2011) are echoed here. There needs to be consistency when using the ESR in adopting the instrument with adequate training provided to LEAs using the tool. Study findings suggest that raters need to be given adequate training on ESR measures concerning the usage and utility for the SOP. Training could be done through webinars and in-service professional development as legitimate options. Raters at the local level need to have a good knowledge base of student skills and performance levels in order for ESR results to be purposeful as part of a student’s SOP documentation. Higher ICC correlations were reported for when multiple knowledgeable raters assessed a student’s employability skill level when using the ESR tool. States and LEAs need to recognize that the ESR is a component of the SOP documentation supporting students’ postschool goals of accessing appropriate gainful employment. Also, as suggested by Richter and Mazzotti, students need to be aware of the potential of the ESR as part of the SOP to promote self-advocacy in gaining access to employment.
Conclusion
The SOP has the potential to be an important component of a student’s documentation as part of their transition process. One way to meet this goal is by having the SOP include skills-based rubrics like the ESR as evidence-based assessment of student’s achievements, skills, and transition needs. Our intent with the ESR pilot study was to provide states and/or LEAs with an instrument to assess these important employment skills.
Results of this study indicated the ESR had good interrater reliability and internal consistency. As a result, the pilot study indicated the ESR has the potential to provide constructive and instructive information for SOP purposes. We hope that states and LEAs will include some form of rubric-based skills performance assessment similar to the ESR as part of their SOP documentation and reporting process, especially in facilitating successful employment outcomes for youth with disabilities. The study provides specific employability skills performance assessment as part of the SOP that can facilitate successful movement toward postschool employment.
Footnotes
Acknowledgements
The authors would like to express appreciation to Career Development for Exceptional Individuals (CDEI) Coeditor Dr. David Test for his assistance and editorial suggestions during the revision process leading toward publication of this manuscript.
Authors’ Note
The opinions expressed are those of the authors and do not represent views or positions of the Indiana Department of Education or the U.S. Department of Education. The primary author was the external evaluator of the state’s State Improvement Grant, and this project was a result of efforts focused on transition. The secondary author was employed as the coordinator of the Indiana Post-School Follow-up Study, a project funded in part through the State Improvement Grant. The third author was employed as the Codirector of the Indiana State Improvement Grant.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The author(s) declared that they served as members on the state’s Employability Work Group during the study period, the research was supported by the state’s Department of Education Center for Exceptional Learners, and technical matter and analysis in this manuscript were submitted as part of a technical report to Indiana Department of Education.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by the Indiana Department of Education, Indiana University Project 01704-004-PY02 (CFDA # 84.323A, federal pass through funds from the U.S. Department of Education).