Abstract
This article describes the development, implementation, and second-year evaluation of Project Autism Competencies for Endorsement (ACE), a web-based professional development (PD) program that is designed to train teachers currently working in the field to meet the unique and diverse needs of children with autism spectrum disorders (ASDs). A description of the rationale for the project and an overview of the program of study are followed by the presentation of the second-year evaluation activities and their results. Preliminary findings of this study suggested that the web-based PD program was effective in helping teachers develop and improve their competencies, knowledge, and skills. Moreover, teachers felt comfortable using the teaching strategies they learned in the program and reported some application of these strategies in their practices. Also, most teachers reported their satisfaction with the web-based courses offered in the program. Implications for the future research and practice are presented.
Keywords
Introduction
Autism spectrum disorders (ASDs) are complex developmental disabilities characterized by deficits in social interaction, verbal and nonverbal communication, and repetitive behaviors or interests (Leblanc, Richardson, & Burns, 2009; Simpson, de Boer-Ott, & Smith-Myles, 2003; Suhrheinrich, 2011). The autism spectrum includes Autistic Disorder, Asperger Syndrome, Pervasive Developmental Disorder: Not Otherwise Specified, Rett Syndrome, and Childhood Disintegrative Disorder (Leblanc et al., 2009; Simpson et al., 2005). It is generally accepted that ASDs occur as a result of abnormalities in brain structure and functioning (http://www.autism-society.org/about-autism/causes/), and can often be reliably diagnosed by the age of 3, and in some cases as early as 18 months (Shaw & Hatton, 2009; Webb & Jones, 2009).
The past three decades have witnessed dramatic increases in the number of children identified with ASDs (Leblanc et al., 2009; Morrier, Hess, & Heflin, 2011; Simpson & LaCava, 2008). For example, between 1990 and 2000, the rate of growth for the U.S. population at large was 13%, and the rate of growth for non-ASDs related disabilities was 16%. The number of children diagnosed with ASDs, however, grew by an astounding 173% (http://www.autism-society.org/about-autism/facts-and-statistics.html). The Centers for Disease Control and Prevention estimates that 1 in 68 children in the United States has an ASD (Baio, 2014).
As a result of the increase in the number of children identified with ASDs, the number of children receiving special education services under this category has also increased (Morrier et al., 2011). According to the 29th Annual Report to Congress on the Implementation of the Individuals With Disabilities Education Act (U.S. Department of Education, 2010), the number of students receiving special education services under the ASDs category increased 775% from 1993 to 2005. Although there are logical explanations to account for this increase (e.g., improvements in identification and diagnosis and development of assessments and screening tools), it remains that ASDs affect a large number of children.
Historically, there has been a shortage of teachers and other related service providers (teachers, hereafter) trained to work with children with ASDs (Robertson & Singleton, 2010; Sindelar, Brownell, & Billingsley, 2010). With the increase in the number of children identified with ASDs, this shortage of teachers is significant and will likely continue to grow in coming years (Frey, 2009). Given that teachers are one of the most critical factors accounting for development of skills in children with ASDs, the need to ensure availability of highly qualified teaching staff is vital (Bellini, Henry, & Pratt, 2011; Morrier et al., 2011; Simpson, Myles, & LaCava, 2008). Furthermore, Simpson et al. (2008) noted that personnel issues in ASDs are not only quantitative but also qualitative. Because the growth of knowledge, practices, and technology in the area of ASDs has been very rapid (Berry, Petrin, Gravelle, & Farmer, 2011), programs focusing on professional development (PD) for teachers must provide teachers with up-to-date, adequate, and in-depth opportunities to learn about evidence-based practices to work with children with ASDs (Kretlow & Bartholomew, 2010; Morrier et al., 2011; Simpson et al., 2008; Suhrheinrich, 2011). Providing valuable and effective PD opportunities to teachers, however, is a challenge for the field of education. Many teachers have limited opportunities to take courses at institutions of higher education to increase their knowledge and skills. The problem becomes even more critical for the teachers who work with children with ASDs in rural areas (Huai, Braden, White, & Elliott, 2006; Knapczyk, Frey, & Wall-Marencik, 2005). One answer to this challenge is the use of web-based PD programs related to special education. Web-based PD programs are convenient and flexible, and as such, ideally suited for teachers without access to institutions of higher education. They also make teaching and learning possible any time and in any place (Huai et al., 2006; Knapczyk et al., 2005; Steinweg, Davis, & Thomson, 2005).
Description of the Project ACE
Rationale
In response to the identified shortage of teachers to work with children with ASDs, several faculty members of the College of Education at the University of Florida applied for and received funding from the U.S. Department of Education, Office of Special Education and Rehabilitative Services to implement a web-based PD program. Project Autism Competencies for Endorsement (ACE) was developed based on four criteria: (a) dramatic national and state (Florida) increase in the number of children with ASDs, (b) critical shortage of qualified teachers to instruct children with ASDs, (c) newly mandated state of Florida endorsement for teachers working with children with ASDs, and (d) current obstacles to recruiting a diverse teaching body.
The nature and magnitude of the gaps and weaknesses in current services for children with disabilities and the shortage of teachers in the state of Florida necessitated the preparation of highly qualified teachers to meet the unique needs of this population. The goal of this program was to prepare teachers who were able to (a) meet state endorsement requirements for teachers instructing children with ASDs; (b) fill vacancies in programs serving children with ASDs, including inclusive education programs; (c) demonstrate knowledge of evidence-based pedagogy to work with children with ASDs; (d) link assessment of social, cognitive, and behavioral skills to effective interventions; and (e) diversify the teacher population serving children with ASDs.
Project ACE aimed to address teacher shortage through recruiting a diverse group of teachers throughout the state of Florida. Over a 4-year period, the project planned to provide 80 teachers (20 per year) with the knowledge of scientifically based pedagogy through the use of web-based technology and field experience. In the second year of the project, a second section of the courses was offered through the Florida Personnel Development Partnership (PDP; formally known as Comprehensive System of Personnel Development). Through this grant, 60 additional teachers were funded for 3 years (20 per year). Therefore, the total number of teachers who were trained throughout the project was 140.
Project ACE was designed as an in-service PD program to assist teachers to gain content and practical knowledge to meet the state autism endorsement requirements. Therefore, teachers currently working in the field with children with ASDs comprised the student body. Teachers accepted to Project ACE completed course requirements (four web-based courses and two field experiences) within a 1-year period.
Program of Study
The content of the project was developed based on core competencies outlined by the Florida Department of Education for the ASDs Endorsement (Jones, Conroy, & Emery, 2005), effective educational practices for children with ASDs defined in the extant literature (Dunlap, Iovannone, & Kincaid, 2008; Koegel, Shirotova, & Koegel, 2009; Mesibov & Shea, 2011; Simpson et al., 2003), and the report of the National Research Council (2001) on educating children with ASDs.
The content of the project emphasized six core components identified as essential for implementing an effective educational program for children with ASDs: (a) individualized supports and services for children with ASDs and their families, (b) systematic instruction, (c) structured environments, (d) specialized curriculum focusing specifically on social and communication skill development, (e) functional approach to behavior problems, and (f) family involvement (Dunlap et al., 2008).
In the first semester, teachers enrolled in two courses: Nature of Autism and Interventions for Communication and Social Development. The Nature of Autism course provided an overview of ASDs, which included the most recent definition, characteristics, and etiology of ASDs; an introduction to evidence-based strategies and practices; classroom structure; legal issues; and the need for family involvement. The Interventions for Communication and Social Development for Students With Autism course was designed to assist teachers to understand the communication and social skills of children with ASDs including their relationship with challenging behaviors. The focus was on classroom-based strategies for promoting effective communication including the use of assistive technology and alternative and augmentative communication.
In the second semester, teachers enrolled in Assessment and Diagnosis course and engaged in the first of two field experiences. The Assessment and Diagnosis course provided information related to: assessment for eligibility, program development, and progress monitoring. An introduction to functional behavior assessment, classroom-based assessment, data collection, data-based decision making, and program development are also included. Field experience provided participants with opportunities to apply the knowledge and practice the skills and strategies learned through the Assessment and Diagnosis course. For the field experience, each practitioner was asked to identify a target child and work with the same child throughout the semester to demonstrate competencies regarding assessment and diagnosis of children with ASDs.
During the last semester in the program, teachers enrolled in the Behavior Management and Positive Behavior Support course and completed the second part of the field experience. The Behavior Management and Positive Behavior Support course provided in-depth information on functional behavior assessment strategies and research-based strategies to reduce inappropriate behavior while facilitating appropriate behavior. The course emphasized the relationship between communication, social skills, and behavior; ethical issues of applying interventions; and the principles of applied behavior analysis. Field experience for this semester was designed to provide teachers with opportunities to practice strategies regarding managing behaviors of children with ASDs and to apply knowledge and skills with respect to implementing functional behavior assessment and positive behavior support. Similar to the first part of the field experience, teachers were asked to identify a target child to work with throughout the semester to demonstrate competencies regarding behavior management and positive behavior support.
Two semester-long field experiences were designed to help teachers translate their understanding of theory, research, and knowledge related to assessment, diagnosis, and behavior management of children with ASDs into practice by working with a target child with an ASD throughout the semester. Teachers funded through the PDP did not take the two field experience courses. The Florida Endorsement in Autism program requires a bachelor’s degree with an additional 12 semester hours of coursework and the PDP funds a maximum of 12 semester hours of coursework. Each of four courses offered in the Project ACE is 3 credit hours, totaling 12 credit hours. To provide similar experiences to teachers funded through PDP, however, the content of field experience courses were integrated into the Assessment and Diagnosis and Behavior Management and Positive Behavior Support courses for these participants. Instead of working with a target child throughout the semester, these teachers were provided with case studies to demonstrate competencies focused on assessment, diagnosis, and behavior management of children with ASDs.
Purpose
The purpose of this article is to describe the development, implementation, and second-year evaluation of a web-based in-service PD program (Project ACE) designed to train teachers currently working in the field to meet the unique and diverse needs of children with ASDs. The evaluation study described in this article was approved by the Institutional Review Board (IRB) of the University of Florida.
Following research questions were addressed during program evaluation activities:
Method
Course Instructors
A total of five instructors taught the four courses and facilitated the activities related to field experiences during the second year of the project. These included three instructors with doctoral degrees in special education and two instructors who were doctoral students in special education or school psychology programs. All five instructors had experience in working with children with ASDs and teaching ASDs-related courses in undergraduate and graduate levels. Each instructor had primary responsibilities of preparing course materials, facilitating online discussions, grading assignments and quizzes, and providing written feedback to students.
Program Participants
Recruitment
To disseminate information about the project and reach a large number of potential participants, project staff contacted administrators in school districts around the state (e.g., district superintendents, personnel directors, or special education directors), designed a web page that provided information about the project and application process, and displayed booths at the state education conferences. In addition, information about the project and application process was disseminated by the state PDP and the Center for Autism and Related Disabilities (CARD). To be accepted into the program, applicants needed to meet the following criteria: (a) have a bachelor’s degree, (b) be employed in a position serving children with ASDs, (c) have basic computer word processing knowledge and skills, and (d) have access to the Internet and an email account. Project ACE participants were provided with tuition support for all courses in the program and stipends for book and other course materials. Participants were also offered funding to attend to the annual CARD conference to provide them with opportunities for ongoing PD and to network with other teachers working with children with ASDs.
A total of 53 applications were received during the second year of the project. Of those, 7 applicants did not meet the criteria to be accepted into the project. Forty of remaining 46 applicants were accepted into the program on a first-come-first-serve basis: The first 20 teachers were funded through Project ACE and the last 20 teachers were funded through PDP. The remaining 6 applicants (all female and Caucasian) were recommended to apply to the project again during the following year.
Characteristics
Upon their acceptance into the program in the second year of the project, each participant was asked to complete an IRB-approved consent form to participate in the study. Of the 40 teachers enrolled in Project ACE and PDP, 36 agreed to participate in this study and 33 completed all evaluation activities. The data presented in this article are based on the responses of those 33 participants. Of the 33 participants, 17 were in Project ACE group and 16 were in the PDP group. Most participants were female, Caucasian, and working as special education teachers. Approximately half of the participants had a master’s degree. As shown in Table 1, participants funded through Project ACE (who took two field experience courses) and PDP (who did not take two field experience courses) showed similar demographic characteristics. The 7 participants who did not complete the consent form or evaluation activities were female. With respect to ethnicity, five were Caucasian, one was Latino American, and one was African American.
Demographic Information for Participants Funded Through Project ACE (N =17) and PDP (N =16).
Note. Participants funded through Project ACE attended two field experience courses, while participants funded through PDP did not. ACE = Autism Competencies for Endorsement; PDP = Personnel Development Partnership.
Course Content and Delivery
As described earlier, program participants completed four web-based courses and two field experiences within a period of a year. The aim of each course and field experience is provided under the Program of Study section above. Specific content of each of four courses, number of weekly discussions, written assignments, and quizzes are presented in Table 2.
Course Content and Frequency of Activities.
Note. A syllabus for each course can be obtained from the first author. ASD = autism spectrum disorders.
All four courses and two field-based experiences were delivered entirely online using web-based course delivery system at the University of Florida, and participants were not required to attend any face-to-face meetings or classes on campus during the program. For each course offered throughout the project, the instructor prepared a syllabus that outlined the content of the course, resources needed, course requirements and policies, assignments, and assessments. The content of each course was delivered using learning modules that lasted 1 or 2 weeks. Each module had four instructional elements: (a) narrative or PowerPoint presentation and readings, (b) discussions, (c) assignments, and (d) quizzes. At the beginning of each module, the instructor uploaded materials related to each of these instructional elements.
Narrative presentation and readings
The narrative presentation and readings were designed to provide participants with general information about the specific topics covered in each module. These were supported by additional resources including case studies, pictures, graphs, videos, or other teaching materials. Such presentation of course content that incorporates some features of hypermedia and takes advantage of web-based environment enhances the effectiveness of online instruction and student learning (Anderson, Sheng, & Sweet, 2010; De Villiers, 2007).
Online discussions
Participants were required to engage in online discussions at least twice during a module by posting comments or questions about the topics covered in the present or previous (when appropriate) modules in response to the instructor’s and other participants’ comments and questions. Online discussions provided participants with opportunities to communicate with each other, share information, and discuss key topics. Discussions were asynchronous allowing participants to post their responses at their convenience during the specified time period in each module. At the end of each module, the instructor provided written feedback to participants about their comments and questions through the online course management system.
Assignments
Two types of written assignments were given to participants throughout the semester. The first group of assignments was designed to evaluate participants’ knowledge acquisition and comprehension of the content of course modules, as well as their ability to analyze and synthesize knowledge learned throughout the semester. For example, during the Nature of Autism course, participants were asked to identify one unusual or controversial intervention associated with ASDs, provide a detailed description of the intervention, analyze of the intervention using the criteria described in the course, and make suggestions regarding its use.
The second type of assignment was developed to help participants apply new knowledge and skills to their current practice. For example, during the Behavior Management and Positive Behavior Support for Students with Autism course, participants were asked to develop a behavior intervention plan by working with a target child or using the case study provided by the course instructors. This assignment allowed participants to apply knowledge and skills they learned during the course to a real-life example. Course instructors provided written feedback on assignments through the online course management system.
Quizzes
An additional measure of knowledge acquisition included brief quizzes on the content of each module. Generally, quizzes contained four types of questions including multiple-choice, true-false, short answer, and short essay. An automated grading system was used whenever possible. For short answer and essay questions, scoring criteria along with space for participant response and instructor comments were provided. Instructor comments were made available to participants once the instructor reviewed and graded each quiz.
Evaluation Instruments
Three instruments were developed or adapted for the study and completed by participants at three different intervals: pre–post, within program, and post-only. These instruments included (a) the Competency-Based Self-Assessment (pre–post), (b) the Use of Knowledge in Practice Survey (post-only), and (c) the Satisfaction Survey (3 times throughout the program). These instruments were administered through a web-based survey system.
Competency-based self-assessment
Participants evaluated their own knowledge and skills about ASDs and serving children with ASDs and their families at the beginning and end of the program through a 45-item researcher-developed instrument. The instrument used a 5-point Likert-type scale with response options that reflect levels of competence (1 = not competent, 2 = somewhat competent, 3 = competent, 4 = very competent, and 5 = extremely competent). The items included in the instrument were developed based on the core competencies outlined by the Florida Department of Education for the ASDs Endorsement (Jones et al., 2005) and were grouped under six broad competency categories: (a) overview of ASDs, (b) assessment and evaluation, (c) instructional strategies, (d) communication, (e) family involvement, and (f) legal issues.
Use of Knowledge in Practice Survey
Two weeks before the end of their last semester in the program, participants assessed their levels of comfort in using evidence-based instructional practices while working with children with ASDs and the frequency in which they used these practices through a 41-item researcher-developed instrument. The participants rated their level of comfort in using practices they learned throughout the program with a 4-point Likert-type scale (1 = not comfortable at all, 2 = somewhat comfortable, 3 = comfortable, and 4 = very comfortable). They also rated the frequency with which they used these practices in their classrooms while working with children with ASDs using a 4-point scale (1 = never, 2 = occasionally, 3 = often, and 4 = very often). Broadly, the strategies were grouped under five categories: visual teaching methods (e.g., use of individual picture schedule, activity checklist, choice board, or labeling objects), classroom management strategies (e.g., use of structured, predictable routines, arranging classroom environment to prevent challenging behaviors, or adapting tasks and materials), strategies to support communication development (e.g., modeling appropriate language, providing communication opportunities by structuring play activities, or using alternative communication systems), strategies to support social skill development (e.g., using peer-mediated social interaction strategies, supporting peer interaction through play, or using social stories), and challenging behavior intervention strategies (e.g., conducting functional behavior assessment, or developing, implementing, and evaluating positive behavior support plans). These strategies were developed based on the literature on effective instructional practices for children with ASDs (e.g., Dunlap et al., 2008; Mesibov & Shea, 2011; Odom, Collet-Klingenberg, Rogers, & Hatton, 2010; Simpson, 2005) and were addressed in one or more of the courses throughout the program.
Satisfaction Survey
Participants completed a 10-item satisfaction survey 3 times throughout the program (i.e., one per semester) to evaluate their satisfaction with three aspects of web-based courses: technology, relationships, and learning. The instrument used a 5-point Likert-type scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). This survey was adapted from Chen, Klein, and Minor (2008).
Data Analysis
The Competency-Based Self-Assessment pre-test and post-test data were analyzed using paired-samples t-tests for participants in Project ACE (field experience) and PDP (no field experience) groups separately at competency category level to determine within-group changes in the participants’ knowledge and skills as a result of their participation in the program. To compare Project ACE and PDP groups on differences between pre-test and post-test, independent samples t-tests of gain scores were conducted at competency category level. Post-test data obtained through the Use of Knowledge in Practice Scale were analyzed using independent samples t-tests at teaching strategy category level. The same analysis technique was used to determine between-group differences (Project ACE versus PDP) for comfort level and frequency of use. Similarly, independent samples t-tests were used to investigate between-group differences in participants’ satisfaction at item level across three courses. When using Likert-type scales, it is recommended to calculate and report Cronbach’s alpha coefficient for internal consistency reliability at sub-scale or scale level (Gliem & Gliem, 2003). For each administration of the instruments used in the present study, a Cronbach’s alpha reliability coefficient was calculated and reported at overall scale level.
Results
The intended outcomes of Project ACE and therefore the program evaluation study reported in this article include (a) enhanced knowledge and skills on ASDs and working with children with ASDs and their families, (b) improved comfort in using teaching strategies while working with children with ASDs, (c) enhanced application of knowledge and skills learned throughout the courses into practice, and (d) understanding participants’ satisfaction with a web-based course delivery model. Thus, the results of the current study are presented based on these intended outcomes.
Perceived Impact in Knowledge and Skills
The paired-samples t-tests results showed that participants who were funded through Project ACE and received two field experience courses significantly increased their competency levels in each of six competency areas from pre-test to post-test (p-values = .000 for overview of ASDs, assessment and evaluation, communication, and instructional strategies; p-value = .014 for family involvement; and p-value = .005 for legal issues). Similarly, participants funded through PDP who did not take the field experience courses also significantly increased their competency levels in all the six competency areas (p-values = .000 for overview of ASDs, assessment and evaluation, communication, and instructional strategies; p-value = .010 for family involvement; and p-value = .004 for legal issues). The independent samples t-tests of gain scores showed no significant difference between participants who were funded through Project ACE and PDP for any of the six competency categories (p-values > .364 for all competency categories). Table 3 illustrates the mean pre-test and post-test scores for each competency category across the two groups and t-values for within- and between-group comparisons. Overall, Cronbach’s alpha coefficient for internal consistency reliability was .98 for both pre-test and post-test.
Competency-Based Self-Assessment Results by Item-Categories.
Note. Participants rated their competency level using a 5-point scale. In this scale, 1 represents not competent and 5 represents extremely competent. Participants funded through Project ACE attended two field experience courses, while participants funded through PDP did not. Mean differences were calculated by subtracting pre-test scores from the post-test scores. ACE = Autism Competencies for Endorsement; PDP = Personnel Development Partnership; MD = mean difference; ASD = autism spectrum disorders.
t values were based on comparison of gain scores and between-group comparisons were not statistically significant for any of the six competency categories.
Pre–post comparisons are significant at p < .05.
Comfort in and Frequency of Using Teaching Strategies
As shown in Table 4, the mean comfort level was 3.39 or above for participants who were funded though Project ACE and 3.31 or above for participants funded through PDP across all five teaching strategy areas. This indicates that participants across the two groups felt comfortable using teaching strategies they learned throughout the program. Although participants funded through Project ACE had a mean comfort level score higher than the mean comfort level score of those funded through the PDP in all five teaching strategy areas (Mdifference = 0.04 for visual methods, 0.09 for classroom management, 0.07 for communication skills, 0.08 for social skills, and 0.06 for challenging behaviors), the difference in mean scores between the two groups was not statistically significant for any of the five areas (p-values > .597 for all teaching strategies).
Participants’ Comfort Level in Using Teaching Strategies and Frequency of Use.
Note. Data reported in the table are the mean ratings and standard deviations. For comfort level, 1 represents not comfortable at all and 4 represents very comfortable. For frequency of use, 1 represents never and 4 represents very often. Participants funded through Project ACE attended two field experience courses, while participants funded through PDP did not. Mean differences were calculated by subtracting PDP scores from Project ACE scores. ACE = Autism Competencies for Endorsement; PDP = Personnel Development Partnership; MD = mean difference.
None of the comparisons are statistically significant.
Investigation of the frequencies with which teachers used these strategies showed that with the exception of strategies to support communication skill development (Project ACE, M = 3.70 and SD = 0.38; PDP, M = 3.62 and SD = 0.48), participants funded through Project ACE and PDP did not use teaching strategies very frequently in their classrooms. Mean score for frequency of use was 2.72 or below across teaching strategies and groups. Mean frequency of use score for four of the five teaching strategy areas was higher for participants funded through Project ACE when compared with those who were funded through PDP. These areas included visual teaching methods (Mdifference = 0.02), classroom management (Mdifference = 0.12), communication skills (Mdifference = 0.08), and challenging behaviors (Mdifference = 0.20). Mean frequency of use score for social skills was higher for the participants funded through PDP by .06 when compared with the participants funded through Project ACE. The differences between groups across teaching strategy areas, however, were not statistically significant (p-values > .268 for all teaching strategies). Table 4 shows the mean comfort level and frequency of use scores for each teaching strategy area across the two groups and t-values for between-group comparisons. Overall, Cronbach’s alpha coefficient for internal consistency reliability was .98 for comfort level scale and .97 for frequency of use scale.
Participant Satisfaction
As shown in Table 5, participants were generally positive about many aspects of web-based courses. Moreover, mean item scores were similar across groups (i.e., Project ACE and PDP) and courses. In Course 1 (Nature of Autism), a statistically significant mean difference across groups was observed only for one item (Item 3 = Building relationships with other classmates was difficult). Although participants from both groups indicated that it was difficult to build relationships with other classmates, participants funded through PDP (M = 4.62, SD = 0.65) had significantly higher mean score than those funded through the Project ACE (M = 3.79, SD = 1.19), t(25) = −2.27, p = .035. Cronbach’s alpha coefficient for internal consistency reliability was .96 for Course 1.
Means and Ranges for Satisfaction Survey Items Across Three Courses.
Note. In satisfaction Survey, 1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree or disagree, 4 = somewhat agree, and 5 = strongly agree. Participants funded through Project ACE attended two field experience course, while participants funded through PDP did not. Mean differences were calculated by subtracting PDP scores from Project ACE scores. ACE = Autism Competencies for Endorsement; PDP = Personnel Development Partnership; MD = mean difference; ASD = autism spectrum disorders.
Between-group comparisons are significant at p < .05.
In Course 2 (Interventions for Communication and Social Development), a statistically significant mean difference across groups was observed for two items (Item 9 = It was difficult to set aside time to go online for class and Item 10 = As a result of this course, my knowledge and skills in working with children with ASDs have increased). For both items, participants funded through PDP (MItem 9 = 3.31, SDItem 9 = 0.48; MItem 10 = 4.38, SDItem 10 = 0.87) had a higher mean item score than those funded through Project ACE (MItem 9 = 2.88, SDItem 9 = 0.49; MItem 10 = 3.71, SDItem 10 = 0.77), Item 9 t(28) = −2.39, p = .024, and Item 10 t(28) = −2.26, p = .032. Cronbach’s alpha coefficient for internal consistency reliability was .97 for Course 2.
In Course 3 (Assessment and Diagnosis), a statistically significant mean difference across groups was also observed for two items (Item 4 = I missed the face-to-face interaction with the instructor and Item 9 = It was difficult to set aside time to go online for class). For Item 4, the mean item score for participants funded through Project ACE (M = 4.18, SD = 0.95) was statistically significantly higher than those reported for participants funded through PDP (M = 3.09, SD = 0.94), t(26) = 2.96, p = .007. For Item 9, the mean item score for participants funded through PDP (M = 3.55, SD = 0.52) was statistically significantly higher than those reported for participants funded through Project ACE (M = 3.00, SD = 0.61), t(26) = −2.43, p = .022. For all other items across courses, although mean item scores differed across groups, the difference between groups was not statistically significant. Cronbach’s alpha coefficient for internal consistency reliability was .97 for course three.
Discussion
The present article describes the development, implementation, and second-year evaluation of a web-based in-service PD program (Project ACE). Project ACE was designed to help teachers learn knowledge and skills to work with children with ASDs and their families and apply these knowledge and skills in their practices.
The evaluation of PD programs, whether they are web-based or not, has critical implications for meeting the unique needs of children with disabilities and their families (Maddox & Marvin, 2013; Snyder, Denney, Pasia, Rakap, & Crowe, 2011). Given the increasing number of children with ASDs, the dearth of data on outcomes of web-based PD programs focused on influencing knowledge and skills of teachers working with children with ASDs and their families, and the need for identifying the processes used to influence teachers’ knowledge and skills, findings from this and similar PD evaluation studies are important. The evaluation activities were designed to investigate the impact of participating in Project ACE on teachers’ knowledge, skills, and practices as well as their satisfaction with web-based courses offered throughout the program. The preliminary findings reported in this article can be used to refine Project ACE, improve future evaluation activities, and demonstrate promising web-based PD practices.
The findings of the present evaluation study with respect to influences of participating in Project ACE on teachers’ knowledge and skills about working with children with ASDs and their families showed that teachers, regardless of whether they received field experience courses or not, perceived themselves to be more knowledgeable and competent in all the areas targeted by the project on completion. These findings are consistent with findings of prior research investigating the effectiveness of web-based courses and programs in helping teachers improve their knowledge and skills about working with children with disabilities and their families (Ajuwon & Craig, 2007; Brown & Woods, 2012; Chen, Klein, & Minor, 2009; Frey, 2009; Huai et al., 2006; Maddox & Marvin, 2013).
With respect to comfort level in using teaching strategies learned in the courses throughout the project, participants reported that they felt comfortable applying these strategies in their practices. Participants reported infrequent use of teaching strategies while working with children with ASDs, with the exception of strategies to support communication development. These findings are also consistent with the findings of previous research reporting increased teacher knowledge and comfort but rare use of evidence-based practices following PD (Leblanc et al., 2009; Suhrheinrich, 2011). One explanation for this finding might be that teachers working with children with disabilities at all levels from preschool to high school need follow-up support as they struggle to adapt new instructional practices to their classroom contexts (Guskey & Yoon, 2009; Kretlow & Bartholomew, 2010; Snyder et al., 2011; Snyder & Wolfe, 2008). Current research evidence suggests that teachers encounter many challenges when beginning to implement new practices, and to help them overcome these challenges, they should be provided with individualized follow-up support focused on implementation of the newly learned practice (Kretlow & Bartholomew, 2010; Scheeler, Bruno, Grubb, & Seavey, 2009; Scheeler, McKinnon, & Stout, 2012; Suhrheinrich, 2011).
Consistent with the findings of prior research on web-based PD programs for pre-service and in-service teachers (Ajuwon & Craig, 2007; Anderson et al., 2010; Brown & Woods, 2012; Chen et al., 2008, 2009), the participants in this study, whether they received two field experience courses or not, were very positive about the web-based format of the courses. It was found that participants were satisfied with different aspects of the web-based courses including technology, relationships, and learning. Across courses, most participants reported that the course websites were easy to use, online discussions supported their learning, course instructors provided them with ample feedback, and their knowledge and skills about working with children with ASDs improved as a result of receiving courses offered throughout the project. Moreover, they reported that it was not difficult to set aside time to go online for web-based courses. In addition, they reported that establishing relationships with other classmates was difficult and noted missing face-to-face interaction with course instructors. Conversely, participants reported that they learned in web-based classes as well as they would learn in face-to-face classes and noted spending more time on web-based courses than they did in most of their face-to-face courses. Taken together, these findings suggest that although participants miss face-to-face interactions with course instructors and have difficulty building relationships with classmates, these factors have little impact on their learning and the time they spend focusing on course content.
An important finding of the present evaluation study was that although participants across the two groups gained knowledge and skills as a result of participating in the project and felt comfortable using teaching strategies at the end of the project, there were no statistically significant differences between participants who received two field experience courses and participants who did not take the field experience courses on these outcomes. Moreover, statistically significant group differences were observed in only one or two items of the satisfaction survey across the courses. One possible reason for no group difference might be that due to the in-service nature of the project, all participants had experience with and access to real cases of children with ASDs and their families. Therefore, participants in both groups may have had similar opportunities to apply the knowledge and skills they learned to their practice.
Limitations
There are limitations associated with the evaluation study reported in this article that should be considered when interpreting the results. First, data reported in this article were based only on self-report and the participants’ perceptions about the changes in their knowledge, skills, and practices. Second, data regarding the use of knowledge in practice were collected only at the end of the program. Therefore, teachers’ initial comfort level in using teaching strategies and frequency of use of these strategies were not available to evaluate the effects of the program on these variables. Third, this evaluation study included a relatively small sample size (N = 33). Fourth, not having a business-as-usual comparison group presented threats to the internal validity of the evaluation study as non-program related factors such as maturation, exposure to other PD activities, and the pre-test could have affected the participants’ performance on outcome measures. Finally, child outcomes were not measured in the present evaluation study.
Implications for Future Research and Practice
The present evaluation study highlighted several areas to consider in developing web-based PD programs for teachers who work with children with ASDs. To promote teachers’ correct and frequent use of evidence-based teaching strategies learned during programs, web-based PD programs should include more systematic delivery of follow-up support. Although course instructors in Project ACE provided teachers with written feedback for projects/assignments focusing on the implementation of the teaching strategies, this does not appear to be as effective as support and feedback provided in the context of the classroom while teachers are working with children with ASDs. Because the participants of Project ACE were working at many locations across the state, it was not possible to visit each classroom multiple times to observe and provide feedback. One possible solution to this problem is the use of video-based coaching and feedback where teachers provide a videotape of their implementation of an instructional strategy while working with children with ASDs and an instructor provides written feedback based on the video clip. Video-based coaching and feedback has been used in several research studies that focused on literacy practices and has proven to be effective in supporting teachers to implement evidence-based instructional strategies (e.g., Kinzie et al., 2006; Pianta, Mashburn, Downer, Hamre, & Justice, 2008). Moreover, video-based coaching can be paired up with audio or bug-in-ear coaching. A growing body of literature has demonstrated the effectiveness of these methods in supporting individuals with ASD in their work environments (e.g., Bennett, 2013; Bennett, Ramasamy, & Honsberger, 2013) and in providing support to pre-service and in-service teachers who work with children with disabilities (e.g., Coninx, Kreijns, & Jochems, 2013; Dieker, Rodriguez, Lignugaris/Kraft, Hynes, & Hughes, 2014; Israel, Carnahan, Snyder, & Williamson, 2013; Scheeler, Congdon, & Stansbery, 2010).
To provide teachers with opportunities to interact face-to-face with course instructors, virtual office hours can be integrated into programs so teachers and course instructors meet online for a pre-determined time period using freely available online software applications that allow voice and video calls (Thompson, Klass, & Fulk, 2012). Moreover, instead of or in addition to asynchronous discussions, synchronous discussions where participants and course instructors are required to be online at the same time can be used to encourage social interactions among participants and between participants and course instructors. Synchronous discussions can be utilized in several different ways such as chat rooms (text only), conference calls, video conferencing, or virtual worlds. Course instructor(s) can integrate a few optional synchronous discussions into web-based PD programs so participants who are interested in meeting and talking with other classmates and instructors can attend for extra credit. Alternatively, instructors can establish several mandatory synchronous discussion sessions (60-90 minutes in length) and grade participants based on their contributions.
The findings and limitations of the present study also inform several recommendations for future research evaluating web-based PD programs designed for teachers of children with ASDs. The program evaluation studies should (a) investigate program outcomes using additional data collection strategies (e.g., live or video-based classroom observations, portfolio evaluations), (b) examine child outcomes as the ultimate goal of these PD programs to improve services children with ASDs receive, and (c) collect more data periodically to identify the influences of specific program components on teachers’ implementation of teaching strategies.
In summary, despite the limitations of the current evaluation study, preliminary findings showed that the web-based PD program described in this article that focused on training teachers to work with children with ASDs was effective in helping teachers develop and improve their competencies, knowledge, and skills. Moreover, teachers felt comfortable using the teaching strategies they learned in the program and reported some application of these strategies in their practices. Also, most teachers reported their satisfaction with the web-based courses offered in the program. The need, however, for more stringent monitoring of program outcomes and data collection processes is warranted. In addition, there are a number of strategies that could be added to the web-based program to increase implementation of practices and enhance the building of relationships. These include systematic delivery of follow-up support and additional virtual opportunities to interact face-to-face with course instructors and peers.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
