Evaluation of a Multicomponent Online Communication Professional Development Program for Early Interventionists

Abstract

Collaborative early intervention (EI) service delivery is enhanced by professional development focused on knowledge and skills in the content and process of delivering services. This article describes a multicomponent online professional development course designed to build infant toddler specialists’ capacity to support children with communication needs and their families. A preliminary program evaluation approach was used to examine the impact, feasibility, and provider satisfaction of the initial implementation of the course. Findings indicated that providers enhanced their knowledge and skills in communication development and intervention and applied those skills within family-centered services in natural environments. Specifically, they demonstrated course competencies in identifying communication acts, developing intervention targets, and planning communication-promoting strategies within the context of coaching caregivers in activities and routines. Implications of professional development designed within the framework of adult learning principles, evidence-based content, and empirically supported instructional methods are presented in relation to the effectiveness of EI.

Keywords

early intervention communication professional development collaborative consultation family centered natural environment

Professional development (PD) is commonly proposed as a way to increase early intervention (EI) providers’ knowledge and skills (Bruder, Mogro-Wilson, Stayton, & Dietrich, 2009). The most widely used PD format, the workshop/conference style of learning, has not been linked to significant changes in providers’ practice (Campbell, Chiarello, Wilcox, & Milbourne, 2009; Garet, Porter, Desimone, Birman, & Suk Yoon, 2001; Snyder & Wolfe, 2008). A focus on defining PD and examining approaches that are linked to intended outcomes has increased in recent years (Buysse, Winton, & Rous, 2009; Snyder, Hemmeter, & McLaughlin, 2011). To affect practice and knowledge, it is recommended that PD be based on a logic model that links learner knowledge with child and family outcomes (Bruder et al., 2009) and that the program’s logic model serves as the framework for evaluating the PD effectiveness (Bellini, Henry, & Pratt, 2011). Based on adult learning principles, a combination of instructor-led and participant-driven activities has been associated with improved learner knowledge and application. Specifically, PD that includes learner reflection has been found to have the largest effect on performance improvement (Dunst & Trivette, 2009).

As the literature demonstrates limitations of workshops and supports comprehensive PD systems, time and resource challenges become paramount. Technology-supported PD is gaining momentum to flexibly meet training needs (Bruder et al., 2009; Chen, Klein, & Minor, 2009). Online PD programs have been linked with improving providers’ skills in supporting children with multiple disabilities (Chen et al., 2009), improving teacher–child interactions (Pianta, Mashburn, Downer, Hamre, & Justice, 2008), promoting literacy skills (Diamond & Powell, 2011), and making language intervention decisions (Buzhardt et al., 2011). Building on the outcomes of previous online early childhood PD programs, the Communication Coach course was developed to be a multicomponent online PD course based on a logic model of intended impacts with a focus on theoretically and empirically supported adult learning principles, targeted early communication intervention practices, and instructional methods.

Course Development

Considering that communication is the most addressed developmental domain on Individualized Family Service Plans (IFSPs) and that the consultative role of teams continues to expand, providers from various backgrounds need to be equipped to serve children with communication delays (Bruder, 2005; Hebbeler et al., 2007). In particular, special instructors support children and families in broad areas of child development in accordance with the child’s IFSP (Early Intervention Program for Infants and Toddlers With Disabilities, 2011); therefore, they serve an increasing number of children whose needs include communication. Most special instructors have a background related to general child development; therefore, their instruction and skills in communication development and intervention vary based on educational and experience backgrounds.

Logic Model

The Communication Coach course was developed from a logic model of intended inputs, activities, outputs, outcomes, and impacts specific to a northeastern state’s needs (Bellini et al., 2011). Specifically, PD for special instructors focused on delivering effective interventions for children with communication needs as a part of a collaborative team was identified as a priority. The conceptual basis for the online course included three key learner outcomes delivered through an approach that engages participants as adult learners: (a) communication development and intervention, (b) collaborative service delivery for children with communication needs, and (c) use of explicit and evidence-based methods for providing and teaching caregivers to embed intervention (Buysse et al., 2009).

Based on theoretically and empirically supported EI practices, adult learning principles, and instructional methods, we developed five 6-hr online asynchronous multicomponent content units comprising the Communication Coach course: Setting the Stage for Communication Development, Early Communication, Expanding Communication, Communication for Conversations, and Caregiver Coaching. Within each unit, there were specific activities that provided opportunities for special instructor and speech-language pathologist (SLP) collaboration.

Adult Learning Principles

Adult learners’ life experiences and previous knowledge influence current learning; therefore, adult learning principles guided the Communication Coach course development. Content delivery and activities were developed recognizing that adults tend to be autonomous, self-directed, and goal oriented and that they prefer to be partners in the process of relevancy-oriented learning (Knowles, Holton, & Swanson, 2005). The course content and activities were situated within the context of EI service delivery to support and maximize the learners’ contextual relevancy with the aim of increasing gains in knowledge and skills (Lave & Wenger, 1991). Authentic examples and observations paired with sustained and active engagement were provided to align with effective adult learning characteristics (Bransford, Brown, & Cocking, 2000).

Targeted Early Communication Intervention Practices

The curricular content of the Communication Coach course reflected the following core principles of child development and family-centered services in natural environments: (a) communication skills develop within the framework of overall development (Allen & Marotz, 2010), (b) young children learn best while actively engaged in their everyday activities (Dunst, Bruder, Trivette, & Hamby, 2006; Wetherby & Woods, 2006), (c) team members collaborate in sharing information and in the identification and provision of service delivery (Horn & Jones, 2005), and (d) EI providers support families to embed intervention strategies into their daily routines (Kashinath, Woods, & Goldstein, 2006; Woods, Kashinath, & Goldstein, 2004).

Communication development and intervention supports were framed within these key principles of family-centered services in natural environments. Information on communication development progressed from an overview of communication to development and intervention at the prelinguistic, early word, and early conversational stages (Pence & Justice, 2008) to integrating the information in coaching caregivers. Evidence-based communication intervention strategies were presented emphasizing enhanced milieu teaching (EMT; Kaiser & Trent, 2007). EMT is a communication intervention that can be implemented by caregivers in play and in a variety of natural, everyday activities and has been shown to be effective in increasing communication skills in children with language delays of various etiologies (e.g., Kaiser, Hancock, & Nietfeld, 2000; Kaiser & Roberts, 2012; Roberts & Kaiser, 2012). Specifically, environmental arrangement, responsive interactions, descriptive talking, and expanding and building strategies were presented for each developmental stage.

After engaging in content focused on communication development and intervention, the learners participated in a content unit on caregiver coaching to build triadic intervention with the parent(s), provider, and child working together. Coaching approaches, which are based on theories of adult learning, are used to support parents in a context-driven, reciprocal, and reflective partnership (Rush & Sheldon, 2011; Woods, Wilcox, Friedman, & Murch, 2011). Strategies to support the caregiver through the use of specific behaviors were presented: conversation and information sharing, direct teaching, demonstration, observation, guided practice with feedback, caregiver practice with feedback, problem solving, and joint interaction (Friedman, Woods, & Salisbury, 2012).

Instructional Method

Each unit was structured using the R.O.P.E. (Read, Observe, Practice, Exhibit) instructional method (Brown & Woods, 2010) to provide multicomponent situated learning opportunities to increase knowledge and skills. R.O.P.E. is congruent with Johnson and Aragon’s (2003) recommended principles for effective online learning: address individual differences, create a real-life context, motivate the learner, provide hands-on activities, avoid information overload, encourage social interaction, and encourage student reflection. The R2D2 (Read, Reflect, Display, Do) framework is a commonly used online instructional method designed specifically for higher education courses, which are often focused on knowledge acquisition (Bonk & Zhang, 2006).

Although there is theoretical support for the R2D2 framework, there is limited research examining its impact, effectiveness, or learner satisfaction. R.O.P.E. expands opportunities that are provided in R2D2 to align with the needs of instructional methods training for practicing EI providers. The expanded opportunities to observe, practice, apply, and reflect on skills in the context in which they will be used in actual practice are important when the intended outcomes include changed skills and increased knowledge. The situated learning opportunities used in the R.O.P.E. method have been supported in other educational settings (Honigsfeld & Schiering, 2004; Landry, Anthony, Swank, & Monseque-Bailey, 2009; Putnam & Borko, 2000).

Purpose

The purpose of the present program evaluation was to examine how the goals and objectives of the Communication Coach course were met to provide information on online EI professional development. Specifically, this evaluation explored the extent to which the program had an impact on the following outcomes for the targeted group of EI providers: (a) enhanced knowledge and skills application of communication development and intervention, (b) enhanced application of coaching caregivers in varied family-identified routines, and (c) understanding of the feasibility, satisfaction, and perceived impact of the course.

Method

Participants

The study included 24 EI special instruction providers in a northeastern state who participated in the duration of the Communication Coach course. Each of the providers voluntarily gave consent to participate in the evaluation of the program. The providers represented 63% of the EI service counties across the state, including urban, rural, and suburban locations. Demographic information is provided in Table 1.

Table 1

Participant Demographic Information

Provider characteristics (n = 24)	n	%
Discipline
Early childhood special education	11	45.8
Early childhood education	9	37.5
Special education	2	8.3
Occupational therapy	1	4.2
Elementary education	1	4.2
Education
Bachelor’s degree	17	71
Educational specialist’s or master’s degree	7	29
Gender
Female	24	100
Race/ethnicity
Caucasian	24	100
Online learning experience
First online course	13	54.2
One to three previous online courses	4	16.6
More than five previous online courses	6	25

	M (SD)	Range

Professional experience
Years in discipline	6.92 (10.29)	1.5-30
Years in EI	6.49 (6.71)	1-28

Note: EI = early intervention. Some items do not total (n = 24) due to participants not providing an answer to all of the demographic questions.

Sampling Procedures

EI administrators selected 30 providers, who gave informed consent, representative of the state’s county service areas to participate; 24 providers completed all five units. The reasons for attrition included moving, no longer working in EI, and family/personal concerns that influenced time available for the course. The level of sustained participation in this course (80% for providers who completed precourse activities and 89% for providers who completed the first unit) was comparable or higher than reported levels of participation in similar self-directed (79%; Nefdt, Koegel, Singer, & Gerber, 2010) and online (65%; Pianta et al., 2008) PD activities.

Measures

Measures completed at three different scheduled intervals, pre–post, within course, and post-only, were of relevance to this study. Providers completed two pre–post measures to examine changes from before the course to after course completion: (a) case study application test and (b) communication development and intervention knowledge and skills survey. Several competency-based learning activities were integral to the course; however, only those that were provided in a standard case study format with consistent questions across units were used as evaluation measures. Post-only measures included video-recorded home visit intervention session segments to demonstrate content application and a course evaluation to measure satisfaction. Several of the measures were created specifically for this study to evaluate the Communication Coach course in relation to its intended outcomes.

Case Study Application Test

Providers answered applied questions based on an EI case study. The case study was piloted with a group of 15 EI providers in two states who were not participants in the current evaluation. A scoring guide of potential appropriate answers was developed based on the input from these providers and the authors’ listing of targeted attributes that needed to be included. The written case study described a toddler and his mother during their typical bath routine. Providers were assessed in eight areas: identifying examples of the child’s expressive language, receptive language, true words produced, form, content, and use; and writing intervention targets and intervention strategies to support the child’s communication.

The instructor and a trained research assistant scored the open-ended responses for each of the eight questions according to the scoring guide. Answers to each question yielded a score of 0, 0.5, or 1. Tabulated question scores were converted to percentage correct in the following areas: (a) knowledge of developmental communication acts, (b) writing appropriate communication targets, and (c) development of embedded intervention strategies. The instructor and research assistant reached 80% interrater agreement across 10 case application tests that were completed by providers who were not participants in this study before scoring the case studies for this group of providers. The instructor scored all of the tests and the research assistant independently scored 25% (n = 6) of the case studies yielding an interrater agreement of 93%.

Communication Development and Intervention Knowledge and Skills Survey

Providers evaluated their knowledge and skills at the beginning and end of the course through a 25-item researcher-developed measure administered through the online course site. Although appropriate caution is warranted with self-report measures (Ehrlinger & Dunning, 2003), using study-specific measures to evaluate learners’ knowledge and skills is consistent with other EI PD research (Chen et al., 2009; Dunst & Trivette, 2009). Providers rated their current knowledge in specific areas of communication development, application of those areas of communication development, and their integration of communication development and intervention in providing EI services for children with communication delays.

Within-Course Measures

Providers practiced applying their skills on standard video cases in preparation for the individualized application competency demonstration for each unit. After watching short (1-3 min) videos of a child and caregiver interacting in everyday routines, providers identified the child’s communication acts, developed an embedded communication intervention target, and wrote intervention strategies. A scoring rubric guide was developed with descriptions of attributes needed to earn a score of 0, 0.5, or 1 for each question. Individual question scores were converted into percentage correct scores. Similar to the reliability process for the case study application test, the instructor and a research assistant met initial reliability standards of a minimum of 80% interrater agreement across 10 measures before scoring the participants’ responses. The instructor scored all video measures, and to ensure ongoing interrater reliability, 25% of the video cases were also scored by the research assistant, yielding an interrater agreement of 96%.

Video Measures

At the completion of the course, providers submitted videos from two different home visit sessions to demonstrate their application of integrated course content within a family-identified routine. Providers were instructed to submit 10 to 15 min of video; however, the length of videos submitted ranged from 7 to 24 min. The entirety of each submitted video was used because the video measures were not dependent on total length of time. In addition to the video, providers completed accompanying routine-based intervention plans and session summary notes. The first video was submitted at the end of the fifth unit and the second video was submitted 6 to 8 weeks after completion of the units to measure maintenance of skills. The videos and written materials were analyzed for types and frequency of caregiver coaching strategies, routines used as the context for intervention, and session note routine, communication intervention strategies, and caregiver coaching strategies identification.

Video Coding

Undergraduate and graduate student research assistants at the Childhood Research and Practice Center at Florida State University (http://cec-rap.fsu.edu) coded the videos based on the family-guided routines-based intervention (FGRBI) coding system. Specifically, two sets of modified whole interval coding definitions were used: caregiver coaching and routines. The research assistants were trained to 80% reliability on the coding systems before beginning coding for this study. Two research assistants coded at least 30% of the videos to establish interobserver reliability. Throughout the study, interobserver agreement was at 84% for caregiver coaching and 95% for routines coding. The research assistants assigned one code for caregiver coaching and one code for routines for each 30-s interval based on operational definitions. If two codes were present in an interval, the code that occurred for the majority of time was assigned. Additional information on the FGRBI coding system is available from the second author upon request and has been described in other studies (Woods et al., 2004; Woods & Kashinath, 2007).

Caregiver coaching strategies coding

Each 30-s interval was assigned one code based on the predominate behavior that the provider exhibited to interact with the caregiver from a potential of 11 behavior codes. The individual caregiver coaching codes were summarized and categorized as follows: conversation and information sharing, specific coaching strategies, observation, joint interaction, and child focused. Conversation and information sharing consisted of providers’ sharing information with the caregivers and conversing about general information. When providers specifically coached the caregivers to implement strategies through direct teaching, demonstrations, guided practice with feedback, caregiver practice with feedback, and problem solving, it was categorized as specific coaching strategies. Observation consisted of the provider actively observing the caregiver and child participate in a routine. Intervals coded as joint interaction involved the provider and caregiver jointly interacting with the child without providing specific coaching to the caregiver. The providers’ behavior was coded as child focused when the provider directly interacted with the child without involving the caregiver through coaching and/or joint interactions.

Routines coding

An activity was considered a routine when the child and adult shared a minimum of two turns and the activity had the potential for having the key components of a routine: a clear beginning and ending, predictable or logical steps, embedded outcomes, repetition, and opportunities for turns between child and adult (Woods et al., 2004). Coders assigned 1 of 16 potential routine context codes to each 30-s interval. Individual routine codes were summarized and categorized into the following routine classes: play (including play with objects, outside play, social play, pretend play), pre-academic (including book sharing, drawing, singing), caregiving (including meals, dressing, bathing), community and family (including family chores and community outings), and no routine (including time spent outside of specified routine definitions).

Session notes analysis

Providers completed session summary notes to measure their identification and resulting documentation of routines, communication intervention strategies, and caregiver coaching strategies used during sessions. The session summary note was adapted from the one used in the state of the study to include a category of caregiver coaching strategies. A 3-min sample of each provider’s second visit video was randomly selected and clipped from the original submitted video. Providers received the video sample electronically on a secured shared filing system (www.dropbox.com). Based on the 3-min sample, the individual providers and researcher independently completed the session note.

Reflective analysis

In addition to the session summary note, providers answered reflective questions about the 3-min sample of their second video-recorded session. They were provided with a worksheet that included the following questions: “What worked well in this sample?” “What would you do differently?” and “What do you propose as the next steps for this routine with this family? What is your rationale for choosing these next steps?” Providing a framework for self-directed reflective practice as a means to understanding targeted content and practices is recognized in the EI literature (Campbell & Sawyer, 2009; Dunst, Trivette, & Deal, 2011) and has been shown to be an effective component of PD programs (Dunst & Trivette, 2009).

Satisfaction

After finishing the course, providers completed an anonymous final course evaluation for the purpose of characterizing their experience, satisfaction, and usefulness of the course. Providers answered 43 five-point Likert-type rating scale questions about their experience in the course across five categories (personal learning, course technology, course content: process, course content: activities, and course content: assignments) and 5 open-response narrative questions about their impressions of the course The course evaluation was designed for participants to complete in less than 30 min. Responses to rating scale items were summarized for each of the six categories using means and standard deviations.

Program Evaluation

A program evaluation was completed to explore the impact, feasibility, and participant satisfaction of the pilot implementation of the online Communication Coach course. This evaluation was designed to match the program’s objectives and outcomes by collecting data on the targeted group of providers who participated in the course. Stakeholders can use the findings to assist in making decisions on refinement of the course before developing a more controlled comparison evaluation to make definitive causal statements (Fitzpatrick, Sanders, & Worthen, 2004).

Procedure

The providers selected to participate in the course were mailed acceptance letters, written materials about the course, and informed consent forms. The researchers and state EI administrators held an introductory 1-hr conference call to discuss course objectives, procedures, and the corresponding evaluation of the pilot implementation. Following the conference call, researchers sent an email detailing the course process and requirements and provided links to the online course within the state EI and education supported learning management system. Tutorials for using the course site were provided. The majority of course activities were presented and completed through this course site.

After giving informed consent, providers completed the case study application test and the communication development and intervention knowledge and skills survey. The EI administrators mailed providers the two books for the class (Apel & Masterson, 2001; Pence & Justice, 2008). Providers progressed through the content units sequentially in a self-directed manner within an 8-month time frame. Within each unit, providers engaged with the content through a multicomponent instructional method, R.O.P.E, designed to scaffold learning from foundational knowledge to applied skills. The R.O.P.E. framework was displayed on the course landing page and throughout each activity as a graphic organizer to provide structure and continuity across units.

The Read section consisted of assigned readings from the course books, relevant articles and handouts, and recorded slide presentations. The Pence and Justice (2008) readings provided instructional content at the level of a graduate course in language development, and the Apel and Masterson (2001) readings provided family-friendly explanations of highlighted portions of language development. Recorded presentations linked the text information to the applied practice of EI services.

The Observe section consisted of situated video examples with text and audio annotations to guide the provider’s viewing. The videos demonstrated course content within a family’s everyday routines. For example, the words “time delay” displayed on the screen when a mother paused while holding a hand towel to give the child an opportunity to communicate a request for the towel. In a different video, audio narration introduced the video by saying, “Notice how the provider guides Mom to expand on the child’s single word by saying, “You could respond with ‘more lotion.’”

The Practice section consisted of a video of a family participating in a typical routine followed by questions about the video. Describing the child’s communication, identifying communication targets, recognizing demonstrated communication intervention strategies, and developing additional embedded communication intervention strategies were included as questions in each of the units’ practice section. Providers were able to modify their answers as many times as they needed. They were provided with immediate, automatic feedback with exemplar answers after submitting their responses.

The Exhibit section consisted of an applied competency-based assignment for each unit. Exhibit assignments included developing parent handouts on communication development, compiling charts of responsive strategies embedded in family-identified routines, and writing an FGRBI plan. The exhibit assignments were reviewed by the course instructors and individualized feedback was provided. Providers were given an opportunity to redo the assignment if it did not meet the competency standards of 80% correct compared with the specific activity rubric. No provider required more than two attempts for successful assignment completion.

The final exhibit assignment included a written and video demonstration of the providers’ integration of the knowledge and skills from the entire course. The EI administrators mailed the providers video cameras and tripods to complete the video demonstrations and we provided written and online screen-shot tutorials for the process of using the video camera and uploading videos to a secure video sharing site. The providers were instructed to complete intervention planning worksheets and to include a 10- to 15-min video demonstration of their interaction with the caregiver and child within the targeted routines. Providers submitted a second video and written demonstration 6 to 8 weeks after the first video to measure their maintenance of applying course content in providing EI services. Providers completed a session summary note and video reflection on a random 3-min sample selected by the researchers. We excluded portions of the video in which the video or audio quality was not adequate to view the interaction. A simple random sampling procedure was used to assign each video to represent the beginning, middle, or end of the viewable submitted video portions. After completing the five units and the two videos, providers completed the posttest measures through the course site: (a) satisfaction evaluation, (b) case study application test, and (c) communication development and intervention knowledge and skills survey.

Results

The intended outcomes of the Communication Coach course included the following: (a) enhanced knowledge and skills application of communication development and intervention, (b) enhanced application of coaching caregivers in varied family-identified routines, and (c) understanding of the feasibility, satisfaction, and perceived impact of the course. Therefore, the findings of this preliminary evaluation are focused on providing information in those three areas.

Communication Development and Intervention

Providers’ knowledge of developmental communication acts, identification of appropriate communication targets, and development of embedded intervention strategies were measured through the providers’ responses to the written case study application test and the within-course video case study. All providers who completed the course reached the competency standard of 80% in this area. To further understand the growth from the beginning to the end of the course, we used paired-samples t tests. Providers increased their skills from the beginning (M = 59%, SD = 19%) to the end of the course (M = 89%, SD = 14%) as measured by the case study application test, t(23) = 8.39, p < .001, d = 2.51. Descriptive and inferential results for each individual test item are reported in Table 2. To decrease the measurement threat to validity of only relying on the same test at two points and to further describe the participants’ change over time, a subset of the skills measured in the case study application test was examined in relation to within-course measures. Differences across time points (pre-course, end of Unit 2, and post-course) in the providers’ communication descriptions (i.e., descriptions of child communication acts in terms of form, content, and use), writing of developmentally appropriate communication intervention targets, and developing routine-based intervention strategies are represented in Figure 1.

Table 2

Time Differences of Pre–Post Case Study Application Tests

	M	M
	Pre-course (%)	Post-course (%)	df	t	p	d
General expressive examples	97	100	23	1.45	.162	—
General receptive examples	83	91	23	2.15*	.043	1.08
True words	50	85	23	4.04**	.001	1.21
Communication form	54	92	23	4.24***	<.001	1.37
Communication content	33	81	23	6.8***	<.001	1.99
Communication use	60	92	23	3.98**	.001	1.26
Intervention targets	47	81	23	4.5***	<.001	1.35
Intervention strategies	54	94	23	6.99***	<.001	2.32
Total score	59	89	23	8.36***	<.001	2.51

Note: Effect size is based on Cohen’s d ≥ 0.20 is small, 0.50 is medium, and 0.80 is large.

p < .05. **p < .01. ***p < .001.

Figure 1

Providers’ change across time in the following skills: describing child communication, writing intervention targets, and developing intervention strategies.

Coaching Caregivers in Family-Identified Routines

Caregiver Coaching Strategies

Providers used varied caregiver coaching strategies at the completion of the course (Video 1) and maintained the use of varied caregiver coaching strategies 6 to 8 weeks after the course (Video 2). The frequency of caregiver coaching strategies was not significantly different from Videos 1 and 2 as measured with paired-samples t tests, indicating that providers successfully maintained those skills after the course was completed: conversation and information sharing, t(23) = 0.17, p = .87; specific coaching, t(23) = 0.09, p = .93; observation, t(23) = 0.03, p = .98; joint interaction, t(23) = 1.17, p = .26; and child focused, t(23) = 0.69, p = .50.

The aggregate data from Videos 1 and 2 demonstrated that providers spent less than 7% of their time providing direct child-focused interactions. The rest of the time was spent sharing information and conversing with the caregivers (8.93%), specifically coaching the caregivers to implement strategies through direct teaching, demonstrations, guided practice with feedback, caregiver practice with feedback, and problem solving (27.68%), observing the family in their natural routines (17.86%), or jointly interacting with the child and caregiver (32.86%). The percentage of time providers spent using each caregiver coaching strategy category in Videos 1 and 2 is presented in Figure 2.

Figure 2

Caregiver coaching strategies

Routines

Aggregate data from both videos were used to characterize the diversity of routines across routine classes. Providers and the families they served used a combination of play (43.48%), caregiving (23.84%), pre-academic (12.14%), and community and family (11.79%) routines as the intervention context. More than 90% of the time was spent engaged in a routine. Figure 3 depicts visually the percentage of time spent in each routine category.

Figure 3

Routines

Feasibility, Satisfaction, and Perceived Impact

Feasibility and Satisfaction

Providers’ anonymous responses on the final course evaluation provided data characterizing their experience, satisfaction, and usefulness of the course on a 5-point Likert-type scale (5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, and 1 = strongly disagree). Overall, the course was rated highly, with four of the five categories of questions rated with M > 4: personal learning (M = 4.5, SD = 0.48), course technology (M = 3.99, SD = 0.63), course content: process (M = 4.4, SD = 0.35), course content: activities (M = 4.42, SD = 0.35), and course content: assignments (M = 4.41, SD = 0.48).

Perceived Impact

Providers rated their knowledge and skills in specific components of providing early communication intervention at the start and end of the course. The differences in these ratings were used to understand the impact that the providers perceived from their course participation. Providers reported an increase in knowledge of communication development and intervention from the start of the course (M = 2.76, SD = 0.74) to course completion (M = 3.62, SD = 0.58), t(23) = 5.59, d = 1.63. Similarly, providers rated their ability to apply their knowledge in specific areas of communication development and intervention as higher after the course (M = 2.75, SD = 0.74) than at the start (M = 3.62, SD = 0.55), t(23) = 5.63, d = 1.66. In addition, providers reported higher abilities in performing EI activities related to serving children with communication delays and their families from the start of the course (M = 2.89, SD = 0.72) to course completion (M = 3.64, SD = 0.63), t(23) = 4.97, d = 1.42.

Reflective Identification in Session Notes

To expand on the applicability of identifying strategies in provided examples and developing intervention plans for children on their caseloads, providers completed a session summary note based on a video of their own session. Their video self-reflective identification of routines, communication intervention strategies, and caregiver coaching strategies was compared with the researcher’s analysis of the same video sample. We used kappa statistics to examine the level of agreement between the provider and researcher. Providers and researchers had agreement in the almost perfect category for identification of routines, κ = 0.84, p < .001, 95% confidence interval (CI) = [0.74, 0.94]; moderate agreement over identification of intervention strategies, κ = 0.48, p < .001, 95% CI = [0.37, 0.59]; and substantial agreement over identification of caregiver coaching strategies, κ = 0.65, p < .001, 95% CI = [0.49, 0.81].

Discussion

The purpose of this study was to evaluate the impact, feasibility, and provider satisfaction of the Communication Coach course for the targeted group of providers. A multicomponent online PD approach was designed to enhance providers’ knowledge and skills in communication development and intervention. Those knowledge and skills were integrated into family-centered EI by focusing on building capacity to coach caregivers in family-guided routines.

Evaluating carefully designed PD activities for EI providers has important implications for meeting the needs of children and families served in EI programs (Snyder et al., 2011). Recognizing the number of children receiving EI services for communication, the paucity of data on PD outcomes, and a need for understanding the processes used to influence providers’ knowledge and skills, findings from PD evaluations such as this one are particularly informative. It is important to interpret these findings within the framework in which the Communication Coach course was examined. The evaluation was designed to explore the program’s logic model for the targeted group of providers (Fitzpatrick et al., 2004); therefore, definitive causal claims are not appropriate. These preliminary results are most appropriately applied in making decisions of refinement of the course, establishing a foundation for a more controlled evaluation, and demonstrating promising PD practices.

Communication Coach Methods

The findings of the present program evaluation further support the promising impact of online multicomponent PD programs designed with opportunities to build on content through observation, practice, reflection, and contextual application (Buysse et al., 2009; Dunst & Trivette, 2009). The R.O.P.E. instructional method expanded on current online learning frameworks (Bonk & Zhang, 2006), principles for adult learning (Bransford et al., 2000) and online instruction (Johnson & Aragon, 2003), and effective early childhood PD components (Dunst & Trivette, 2009).

The high ratings of annotated video examples, narrated presentations, overall course satisfaction, video camera access, the R.O.P.E. organization, and the practice video examples support providing situated learning opportunities (Lave & Wenger, 1991) within a multicomponent instructional structure based on adult learning principles (Bransford et al., 2000). A provider described the multicomponent nature of the course by commenting,

I liked the time schedule flexibility. I liked the different teaching modalities. The readings were insightful. The recorded PowerPoint presentations were a fantastic way to make sure the key important points were understood. The assignments and video observations were excellent for applying the knowledge. Overall, I found this course to be very valuable.

Providers collaborated with SLPs within their teams to develop parent support materials and intervention plans throughout each course unit. The collaborative component of the course not only supported teaming but also provided a means and support for such collaboration to occur. Providers initially expressed concerns over scheduling time for teaming activities; however, by the end of the course, SLP collaboration was rated highly and described as an integral part of their practice in supporting children and families with communication needs. Following the course, several providers commented that the reciprocal nature of interactions with team SLPs had become smoother.

Communication Coach Course Content

Communication Development and Intervention

Aligning with the situated learning opportunities of the course, the case study application test was based on an everyday routine, as opposed to assessment information, to examine performance within an environment similar to what is encountered while providing EI services. Considering that all providers in the course had at least 1 year of experience providing EI services, it was expected that they would show at least minimal competence in answering some of the questions. Although identifying general communication examples was a strength for the providers, their skills were lower in identifying which communication acts constituted true words and describing the child’s communication form, content, and use. In addition, providers initially scored low on writing intervention targets and intervention strategies. The providers’ increase in skills in each of the areas from the beginning of the course to the end is interesting for two reasons: (a) the relative low initial scores on six of the eight assessed areas support the need for PD activities aimed at increasing communication development and intervention knowledge and skills, and (b) the significant increases with large effect sizes for the seven areas with initial scores below 85% support the impact of the Communication Coach course on their learning.

Providers’ perception of their knowledge, skills, and EI application increased from the beginning to the end of the course. Interestingly, the two areas that were rated the highest on the post-measure, knowledge of intervention strategies and skills in intervention strategies, also were two of the highest rating increases. The providers’ perceptions of improving in the area of intervention strategies matched their performance on the case study application test, and matched the qualitative growth that was apparent within their unit activities. For example, one provider wrote “use toys, games, finger plays, books” as an intervention strategy at the beginning of the course, and on the post-test provided the following intervention strategy, “When he makes the ‘baba’ sound in the bath, respond to his sound by modeling ‘bubble.’” Another provider identified “home ideas for families to work on between sessions” as an initial intervention strategy, and at the end of the course included the following strategy, “Place the cup out of reach, but in view, so that he has the opportunity to request it.” Each of these examples demonstrates the providers’ ability to develop specific evidence-based communication-promoting strategies (responsiveness and environmental arrangement, respectively) embedded within the family’s natural routine.

Family-Centered Communication Intervention in Natural Environments

Providing intervention within family-identified routines and supporting a child’s caregivers to promote the child’s development are key components of family-centered communication intervention in natural environments (Swanson, Raab, & Dunst, 2011). Therefore, in addition to measuring providers’ specific communication intervention skills, it was important to understand the context in which they applied those skills.

Home visit videos

This evaluation was strengthened by the providers submitting two video-recorded sessions. Evaluating the impact of PD in the applied setting in which the targeted skills are intended to be used provides relevant information about participants’ implementation of the practices. The diversity of routines represented in the videos supports the notion that providers are providing intervention within the natural environment (Dunst et al., 2006; Woods et al., 2004). One provider commented,

I feel I am much more, if not almost entirely, routines-based to my approach to treatment and coaching families. I am a true convert. It totally works and parents understand and feel it is easy to apply the knowledge and models we discuss and practice together. I also believe I am more confident and competent in writing and implementing routines-based outcomes and strategies that are family-friendly and “usable in everyday life.”

Within routines, providers supported families by spending more than 90% of the time sharing information with the caregiver(s), observing the child and caregiver, interacting with the child and caregiver jointly, or specifically coaching the caregiver. One provider reported, “Before this course, I struggled with the coaching aspect of early intervention. I now feel more confident in becoming more of a coach and provide strategies.” Providers maintained similar distributions of caregiver coaching strategies from the first to the second video, indicating short-term maintenance.

Self-reflection of intervention videos

The levels of agreement between the providers and instructors on session summary notes indicate that the providers not only demonstrated the use of the strategies taught in the course but they also recognized and documented their strategy use. It is interesting that, although the providers demonstrated sizable gains in the area of intervention strategies on other study measures, agreement between providers and instructors in identifying intervention strategies used in the session yielded more variance than agreement over caregiver coaching strategies and routines. This variance may be attributed to the newness of the skills and/or the providers conceptualizing the closed response choice in a different manner than the researchers. Providers identified appropriate future intervention plans and current strengths and needs when answering open-ended reflective questions. For example, one provider reflected, “Both Mom and I need to pause more instead of narrating every step. Next time, we should offer her the choice of two shirts without saying as much descriptive talking and have her respond with a gesture.” Responses such as these indicate that providers may understand and identify the strategies at a higher level than what was displayed in the closed response options.

The evaluation outcomes show promise of the course’s impact beyond the required course activities. The subjective comments from the participating providers support the notion of enhanced knowledge and skills. A provider commented, “I am better equipped for answering communication questions from families and more confident in assisting them appropriately based on their child’s needs.” Another provider said,

I find myself now rarely bringing toys into the home—instead, I utilize everything in that child’s environment to work on communication and other targeted developmental needs within what the family is doing. I hope to refine writing and developing “true” routines-based outcomes and strategies and intend to share this knowledge with other EI professionals. I will encourage other providers to use videotaping as an effective professional growth tool as coaches to families.

Limitations of the Evaluation

There are limitations to the program evaluation research presented that should be considered when interpreting the results. The largest limitation is the absence of baseline home visit session videos. Although there were two pre–post measures that provided information on change from the beginning to the end of the course, comparing video measures would have a stronger foundation to make statements about actual practice. In addition, not having a control or comparison group presented significant threats to the internal validity of study. The single group design does not allow for the changes in the providers’ skills to be primarily attributed to the course. Factors other than the course, including maturation, exposure to the pretest, and other PD activities, could have influenced the providers’ performance. The relatively small sample size is also a limitation.

Implications

There is an urgent need for high-quality EI PD that is designed to influence providers’ knowledge and skills. This evaluation demonstrated several promising practices and highlighted areas to consider in developing and evaluating EI PD activities. Targeting evidence-based developmental content within the context of family-centered services in natural environments shows promise for supporting both the intervention of EI and the process of delivering EI. Providers demonstrated specific communication development and intervention skills and integrated those skills into recommended EI practices (Sandall, Hemmeter, Smith, & McLean, 2005). The positive ratings indicate that an intensive self-paced asynchronous online PD course is feasible. The comprehensive course design focused on adult learning principles, evidence-based content, and relevant and empirically supported instructional methods. The R.O.P.E. instructional method was rated as meaningful by the providers. It provided situated learning opportunities progressing from authentic guided video examples to video practice case studies to application with EI families. Using video-recorded interactions as a tool for evaluating PD shows promise as a feasible way to examine skills in applied settings.

As a preliminary evaluation, the findings highlight aspects of course delivery and course content. Building on this evaluation, it would be useful to design a study comparing this course, or a similar online PD course using the R.O.P.E. method, to workshop-based PD and/or to a “business-as-usual” condition. This would provide information on the effects of the course on providers’ knowledge and skills. After the multicomponent course is examined in a controlled study, examining individual components of the approach could lead to determining the active ingredients associated with change (Snyder et al., 2011). In addition, extending PD evaluations to also examine effect on measures of child development will add to our understanding of effective and meaningful ways of increasing capacity to influence child growth.

In summary, despite the limitations of the current program evaluation design, the findings provide preliminary support for both the R.O.P.E. multicomponent online instructional method and targeting communication development and intervention PD for EI providers. To influence EI practices, there is a need for PD providers/consultants and researchers to focus systematically on knowledge and skills in the content and methods of PD.

Footnotes

Authors’ Note:

Jennifer A. Brown and Juliann J. Woods, School of Communication Science and Disorders, Florida State University. Jennifer A. Brown is now at the University of Georgia in the Department of Communication Sciences and Special Education. Partial support for this research was provided by grants from the U.S. Department of Education, Office of Special Education Programs—Project LIFE: Leadership in Family-Centered Early Intervention Personnel Preparation Grant (H325D070023) and Kidtalk-Tactics Model Demonstration Center on Early Childhood Language Intervention (H326M070004), and a contract from Pennsylvania Early Intervention Services. The opinions expressed are not those of the funding agencies, and no official endorsement should be inferred

References

Allen

E. K.

Marotz

L. R.

(2010). Developmental profiles: Pre-birth through twelve. Belmont, CA: Wadsworth.

Apel

Masterson

(2001). Beyond baby talk. New York, NY: Three Rivers.

Bellini

Henry

Pratt

(2011). From intuition to data: Using logic models to measure professional development outcomes for educators working with students on the autism spectrum. Teacher Education and Special Education, 24, 37-51.

Bonk

C. J.

Zhang

(2006). Introducing the R2D2 model: Online learning for the diverse learners of this world. Distance Education, 27, 249-264.

Bransford

J. D.

Brown

A. L.

Cocking

(2000). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

Brown

J. A.

Woods

J. J.

(2010, October). Learning the ROPEs: Impact of a multi-component method for early intervention coursework. Presented at The Division for Early Childhood Conference, Kansas City, MO.

Bruder

M. B.

(2005). Service coordination and integration in a developmental systems approach to early intervention. In Guralnick

M. J.

(Ed.), A developmental systems approach to early intervention: National and international perspectives (pp. 29-58). Baltimore, MD: Paul H. Brookes.

Bruder

M. B.

Mogro-Wilson

Stayton

V. D.

Dietrich

S. L.

(2009). The national status of in-service professional development systems for early intervention and early childhood special education practitioners. Infants & Young Children, 22, 13-20.

Buysse

Winton

P. J.

Rous

(2009). Reaching consensus on a definition of professional development for the early childhood field. Topics in Early Childhood Special Education, 28, 235-243.

10.

Buzhardt

Greenwood

C. R.

Walker

Anderson

Howard

Carta

J. J.

(2011). Effects of web-based support on early head start home visitors’ use of evidence-based intervention decision making and growth in children’s expressive communication. NHSA Dialog, 14, 121-146.

11.

Campbell

P. H.

Chiarello

Wilcox

M. J.

Milbourne

(2009). Preparing therapists as effective practitioners in early intervention. Infants & Young Children, 22, 21-31.

12.

Campbell

P. H.

Sawyer

L. B.

(2009). Changing early intervention providers’ home visiting skills through participation in professional development. Topics in Early Childhood Special Education, 28, 219-234.

13.

Chen

Klein

M. D.

Minor

(2009). Interdisciplinary perspectives in early intervention: Professional development in multiple disabilities through distance education. Infants & Young Children, 22, 146-158.

14.

Diamond

K. E.

Powell

D. R.

(2011). An iterative approach to the development of professional development intervention for head start teachers. Journal of Early Intervention, 33, 75-93.

15.

Dunst

C. J.

Bruder

M. B.

Trivette

C. M.

Hamby

D. W.

(2006). Everyday activity settings, natural learning environments, and early intervention practices. Journal of Policy and Practice in Intellectual Disabilities, 3, 3-10.

16.

Dunst

C. J.

Trivette

C. M.

(2009). Let’s be PALS: An evidence-based approach to professional development. Infants & Young Children, 22, 164-176.

17.

Dunst

C. J.

Trivette

C. M.

Deal

A. G.

(2011). Effects of in-service training on early intervention practitioners’ use of family-systems intervention practices in the USA. Professional Development in Education, 37, 181-196.

18.

Early Intervention Program for Infants and Toddlers With Disabilities, 34 C.F.R. part 303 (2011).

19.

Ehrlinger

Dunning

(2003). How chronic self-views influence (and potentially mislead) estimates of performance. Journal of Personality and Social Psychology, 84, 5-17.

20.

Fitzpatrick

J. L.

Sanders

J. R.

Worthen

B. R.

(2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Allyn & Bacon.

21.

Friedman

Woods

Salisbury

(2012). Caregiver coaching strategies for early intervention providers: Moving toward operational definitions. Infants & Young Children, 25, 62-82.

22.

Garet

M. S.

Porter

A. C.

Desimone

Birman

B. F.

Suk Yoon

(2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 28, 915-945.

23.

Hebbeler

Spiker

Bailey

Scarborough

Mallik

Simesonsson

. . . Nelson

(2007). Early intervention for infants and toddlers with disabilities and their families: Participants, services, and outcomes. Final Report of the National Early Intervention Longitudinal Study (NEILS). Menlo Park, CA: SRI International.

24.

Honigsfeld

Schiering

(2004). Diverse approaches to the diversity of learning styles in teacher education. Educational Psychology, 24, 487-507.

25.

Horn

Jones

(2005). Collaboration and teaming in early intervention. In Horn

Jones

(Eds.), Young Exceptional Children Monograph Series No. 6: Interdisciplinary teams (pp. 11-20). Missoula, MT: Division for Early Childhood of the Council for Exceptional Children.

26.

Johnson

S. D.

Aragon

S. R.

(2003). An instructional strategy framework for online learning environments. New Directions for Adult & Continuing Education, 100, 31-43.

27.

Kaiser

A. P.

Hancock

T. B.

Nietfeld

J. P.

(2000). The effects of parent-implemented enhanced milieu teaching on the social communication of children who have autism. Journal of Early Education & Development, 4, 423-446.

28.

Kaiser

A. P.

Roberts

M. Y.

(2012). Parent-implemented enhanced milieu teaching with preschool children with intellectual disabilities. Journal of Speech, Language, and Hearing Research. Advance online publication. doi:10.1044/1092-4388(2012/11-0231)

29.

Kaiser

A. P.

Trent

J. A.

(2007). Communication intervention for young children with disabilities: Naturalistic approaches to promoting development. In Odom

Horner

Snell

Blacher

(Eds.), Handbook for developmental disabilities (pp. 224-246). New York, NY: Guilford.

30.

Kashinath

Woods

Goldstein

(2006). Enhancing generalized teaching strategy use in daily routines by caregivers of children with autism. Journal of Speech, Language, and Hearing Research, 49, 466-485.

31.

Knowles

M. S.

Holton

E. F.

Swanson

R. A.

(2005). The adult learner: The definitive classic in adult education and human resource development (6th ed.). London, England: Elsevier.

32.

Landry

S. H.

Anthony

J. L.

Swank

P. R.

Monseque-Bailey

(2009). Effectiveness of comprehensive professional development for teachers of at-risk preschoolers. Journal of Educational Psychology, 101, 448-465.

33.

Lave

Wenger

(1991). Situated learning: Legitimate peripheral participation. New York, NY: Cambridge University Press.

34.

Nefdt

Koegel

Singer

Gerber

(2010). The use of a self-directed learning program to provide introductory training in pivotal response treatment to parents of children with autism. Journal of Positive Behavior Interventions, 12, 23-32.

35.

Pence

Justice

(2008). Language development from theory to practice. Upper Saddle River, NJ: Pearson.

36.

Pianta

R. C.

Mashburn

A. J.

Downer

J. T.

Hamre

B. K.

Justice

(2008). Effects of web-mediated professional development resources on teacher–child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23, 431-451.

37.

Putnam

R. T.

Borko

(2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29, 4-15.

38.

Roberts

M. Y.

Kaiser

A. P.

(2012). Assessing the effects of a parent-implemented language intervention for children with language impairments using empirical benchmarks: A pilot study. Journal of Speech, Language, and Hearing Research. Advance online publication. doi:10.1044/1092-4388(2012/11-0236)

39.

Rush

D. D.

Sheldon

M. L.

(2011). The early childhood coaching handbook. Baltimore, MD: Paul H. Brookes.

40.

Sandall

Hemmeter

M. L.

Smith

B. J.

McLean

M. E.

(2005). DEC recommended practices: A comprehensive guide for practical application in early intervention/early childhood special education. Longmont, CO: Sopris West Education Services.

41.

Snyder

Hemmeter

M. L.

McLaughlin

(2011). Professional development in early childhood intervention: Where we stand on the silver anniversary of PL 99-457. Journal of Early Intervention, 33, 357-370.

42.

Snyder

Wolfe

(2008). The big three process components of effective professional development: Needs assessment, evaluation, and follow-up. In Winton

P. J.

McCollum

J. A.

Catlett

(Eds.), Practical approaches to early childhood professional development: Evidence, strategies and resources (pp. 13-52).Washington, DC: Zero to Three.

43.

Swanson

Raab

Dunst

C. J.

(2011). Strengthening family capacity to provide young children everyday natural learning opportunities. Journal of Early Childhood Research, 9, 66-80.

44.

Wetherby

Woods

(2006). Effectiveness of early intervention for children with autism spectrum disorders beginning in the second year of life. Topics in Early Childhood Special Education, 26, 67-82.

45.

Woods

Kashinath

Goldstein

(2004). Effects of embedding caregiver implemented teaching strategies in daily routines on children’s communication outcomes. Journal of Early Intervention, 26, 175-193.

46.

Woods

J. J.

Kashinath

(2007). Expanding opportunities for social communication into daily routines. Early Childhood Services, 1, 137-154.

47.

Woods

J. J.

Wilcox

Friedman

Murch

(2011). Collaborative consultation in natural environments: Strategies to enhance family-centered supports and services. Language, Speech and Hearing Services in Schools, 42, 379-392.