Abstract
This study was conducted to evaluate the effects of a Distance Mentoring Model (DMM), including performance-based feedback and technology support, on expanding the use of recommended home visiting practices by early interventionists. Data from 18 early interventionists indicated that participation in the DMM approach was associated with increased use of various caregiver coaching strategies and routine contexts. Specifically, participants spent significantly less time in child-focused intervention and more time using specific coaching interactions with the caregiver and child as a dyad after participating in technology-supported, performance-based feedback. All participants reported that they would participate in technology-supported, professional development opportunities again in the future.
Research on the services and supports provided to families enrolled in early intervention (EI) highlights the critical need for high-quality professional development (Bruder, 2010). Survey data gathered by the Center to Inform Personnel Preparation Policy and Practice in Early Intervention and Preschool Education (2007) indicated that most states report that EI professionals across disciplines are not adequately trained to provide family-centered services and supports. When interviewed about participation-based services, most EI providers did not describe two key components of recommended practice in EI: (a) use of the family’s routines and activities as the intervention context and (b) EI providers’ role to coach caregivers to embed intervention in these routines (Fleming, Sawyer, & Campbell, 2011). Related research substantiates that the adoption and use of coaching is limited and challenging for EI providers to implement (Campbell & Sawyer, 2007; Peterson, Luze, Eshbaugh, Jeon, & Kantz, 2007; Salisbury, Woods, & Copeland, 2010). Collectively, these studies indicate a clear need to support EI professionals so that they can move from awareness to implementation of recommended practice (Sheridan, Edwards, Marvin, & Knoche, 2009). Yet, effective strategies to promote the implementation of caregiver coaching among EI providers are unknown. This article focuses on this issue and reports results of a multicomponent, technology-supported professional development initiative that contributes preliminary evidence on strategies that impact provider performance.
Although workshops are the prevailing format of professional development in EI (Bruder, Mogro-Wilson, Stayton, & Dietrich, 2009), current evidence suggests that this type of delivery is not associated with sustained change in practice (Andrews, Bobo, & Spurlock, 2010; Snyder & Wolfe, 2008). Attention to evidence-based adult learning strategies is needed to facilitate meaningful changes in practice (Bruder et al., 2009; Trivette, Dunst, Hamby, & O’Herin, 2009). Components of effective professional development identified in the EI literature include active learner participation, problem solving, reflection, and ongoing learning opportunities (Dunst & Trivette, 2009; Polly & Hannafin, 2010), as well as multiple learning episodes and learner self-assessment using established performance criteria (Trivette et al., 2009). Unfortunately, there is limited evidence regarding when or how to incorporate specific components (e.g., individualized coaching, online learning, peer mentoring, group workshops, etc.) to support changes in EI providers’ use of recommended practices (Buysse, Winton, & Rous, 2009) and enhance ongoing use (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). Below, we summarize information about current approaches to professional development and their relationship to provider performance.
Dynamic Professional Development Supports
Dynamic professional development supports such as coaching, consultation, and mentoring incorporate evidence-based adult learning strategies to actively engage participants. While variously defined, coaching, consultation, and mentoring describe relationships that provide individualized support to facilitate professional growth or improve performance. Definitions vary with regard to the nature of the relationships between partners. Coaching relationships focus on evaluative reflection, practice, and action planning with more “hands-on” support from the coach, whereas consultation is typically less direct (Rush & Shelden, 2011). Growing evidence, primarily in early care and education settings (e.g., Campbell & Milbourne, 2005; Neuman & Cunningham, 2009), demonstrates that coursework supplemented with on-site (in classroom) coaching and consultation is more effective in creating change than coursework alone. Although this evidence base is emerging for coaching and consultation in classroom settings, the potential value of these strategies to support EI professionals working with families in home and community settings is not known.
Mentoring
Mentoring describes a supportive relationship in which an individual with knowledge and experience in a given area facilitates a colleague’s professional growth through feedback, reflection, goal setting (Neuman & Cunningham, 2009), and regular evaluation of the mentoring relationship. Although the specific interactions that take place between mentor and mentee are seldom investigated in the research literature, concise, individualized feedback for the mentee is generally cited as an important component of effective mentoring relationships (e.g., Diamond & Powell, 2011). In the present study, mentoring included two different relationships. First, two peer mentees identified at the onset of the professional development initiative developed individual plans, collaborated on their goals, discussed strategies, developed action steps, and scheduled regular meetings to review video recordings of their own home visits and to meet with an expert mentor (university researcher). The benefits of peer mentoring include shared knowledge of proposed program-level outcomes and familiarity with specific contexts of EI service delivery in their program. The second mentoring relationship was between the two peer mentees and the expert mentor. During regularly scheduled expert mentoring sessions, the expert mentor shared performance-based feedback with mentee pairs via technology to facilitate problem solving and reflection in support of their individual goals. In the present study, technology-supported expert mentoring was used not only to discuss the mentee’s performance but also to offer a model for and practice in mentoring including sharing performance-based feedback, for peer mentees.
Performance-based feedback
Elements of effective feedback include information related to learning goals, information based on performance and progress toward goals, and feedback to encourage next steps of the learning process through additional practice and deeper understanding (Hattie & Timperley, 2007). Performance-based feedback refers to information and observations that relate to the mentee’s specific actions and behaviors in an activity. Performance-based feedback using a visual component (e.g., bar graph or video clip) may increase the impact of feedback (Noell et al., 2005; Reinke, Lewis-Palmer, & Martin, 2007), whereas graphed feedback paired with verbal feedback may be needed for sustained changes (Casey & McWilliam, 2008). Despite these emerging insights, little is known about the application of feedback for EI providers in home and community settings.
Technology-based support
Recent research related to the use of technology to increase the quality and effectiveness of professional development activities reports promising findings (Hemmeter, Snyder, Kinder, & Artman, 2011; Pianta, Mashburn, Downer, Hamre, & Justice, 2008). Online activities, including face-to-face interaction, opportunities for reflection (Chen, Klein, & Minor, 2009), and partnering together with another EI provider from the same agency (Chen, Klein, & Minor, 2008) were identified as beneficial by participating professionals. The Internet, digital video, and other readily available technology tools may serve not only as a means of information access, interaction, and communication with a coach or mentor but also as a context for dynamic performance-based feedback and supported reflection.
Technology can offer a convenient format for timely feedback related to individualized support. For example, recent studies used email to provide feedback to preservice professionals (Barton & Wolery, 2007; Brown & Woods, 2011) and preschool teachers (Hemmeter et al., 2011) to effectively increase their use of targeted behaviors. The use of video can also provide a means for supervisors, instructors, and mentors to offer performance-based feedback to those learning a new skill and to highlight approximations toward desired behaviors as learners watch and reflect on similarities and differences between desired and observed performance. This feedback can be provided in person or via technology. For example, a recent study of Head Start teachers compared the effects of participation in ongoing professional development in two formats: web-only or consultative (Pianta et al., 2008). Web-only participants independently accessed web-based training materials including lesson plans and video clip examples, and the consultative group received written and visual feedback using online video chat and web-based training materials about their own video-recorded teaching sessions from a consultant. Teachers who received feedback on their own performance related to instructional interactions with students in their classrooms via video sharing with consultants showed significantly greater gains on characteristics of teacher–child interaction, including emotional support, classroom management, and instructional support than those who viewed general video examples. Performance-based video feedback delivered via technology is an efficient and effective professional development tool, yet more research is needed to determine how these promising classroom-based findings can be expanded to EI home and community settings.
Distance Mentoring Model (DMM)
The DMM (http://cec-rap.fsu.edu/frontpage/) was designed to affect change in practice through systematic, ongoing professional development activities for EI professionals using evidence-based strategies described in the research literature on teaching and learning (Donovan, Bransford, & Pellegrino, 1999). The DMM addresses the challenges of adoption and use of recommended practices by incorporating active participation in goal setting, concrete examples from work settings, ongoing support, and performance-based video feedback using technology. Five professional development strategies were integrated into this model: (a) face-to-face training workshops; (b) peer and expert mentoring with scheduled interactions; (c) video review and performance-based feedback; (d) follow-up to feedback that included video files, email summaries of mentoring and feedback, and plans for the next video; and (e) monthly newsletters. This multifaceted approach was evaluated over a period of 8 months with a sample of EI providers involved in conducting home visits. The impact of this approach on provider performance was evaluated and is reported below.
Research Questions
The purpose of the present investigation was to examine how DMM technology-supported performance-based feedback affects the home visiting practices of EI providers. The specific questions guiding this work were the following:
Research Question 1: Does DMM technology-supported performance-based video feedback affect the diversity of caregiver coaching strategies EI providers use during home visits?
Research Question 2: Does DMM technology-supported performance-based video feedback affect the diversity of routines used by EI providers during home visits?
Research Question 3: How satisfied were participants with using technology to support professional development activities?
Method
Professional Development Context and Content
The researchers contracted with a state Part C lead agency in a midwestern state to provide professional development and evaluate the effectiveness to increase the state’s EI providers’ implementation of family-guided routines-based intervention (FGRBI). The methods described within this article target an ongoing initiative using DMM. To measure change in practice, we focused on EI providers’ use of measurable caregiver coaching strategies and routine contexts that would be visible on video recordings and could occur during EI home visits. Participants learned to use specific caregiver coaching strategies and to embed intervention in everyday activities in FGRBI workshops, and participated in ongoing DMM activities (i.e., mentoring, performance-based feedback, email summary, and newsletters) to deepen their knowledge and skills.
Caregiver coaching strategies represent ways in which EI providers teach caregivers how to use intervention strategies with their child, and routine settings are the contexts in which intervention occurs. Specific caregiver coaching strategies include direct teaching of intervention strategies, demonstration, guided or caregiver practice with feedback, problem solving, and reflection. Other more general caregiver coaching interactions include conversation and information sharing (CIS) in which the caregiver and EI provider exchange information related to the child and family, observation of caregiver–child interaction, and joint interaction (JI) in which the EI provider and caregiver work as partners with the child, practicing intervention strategies embedded in routines (Friedman, Woods, & Salisbury, 2012). Routine settings include play (e.g., toy play, constructive play, pretend play, physical play, social games), caregiving (dressing, hygiene, feeding, comfort/disability-related), pre-academic and literacy (book sharing, songs and rhymes, computer/television, writing/drawing), and family/community (errands, chores, socialization, recreation) routines.
Design of Professional Development and Participants
To recruit participants, program coordinators in each of the state’s eight regions selected four to six EI providers to attend three face-to-face training workshops and participate in ongoing distance mentoring for a year. Coordinators agreed to support providers’ participation with 1 hr of paid work time each week dedicated to mentoring activities. A total of 34 EI providers (17 teams of two peer mentors) attended an initial FGRBI training workshop and established their initial plans for video collection, review, and feedback. Peer mentor teams then participated in distance mentoring in two feedback formats—Skype and conference call. Participants were expected to submit videos monthly, resulting in eight total videos for each during the study period. Inclusion criteria for data analysis were attendance at the two face-to-face training workshops held in the 8-month study period, submission of at least four home visit videos, participation in at least four feedback sessions (two in each format), and ongoing contact with one or two families for video recording. In all, 18 participants met the inclusion criteria. Of these participants, most were women (94%) and Caucasian (94%) with one provider of Hispanic background. A total of 12 (67%) had master’s or specialist’s degrees, and the rest (33%) had bachelor’s degrees. Participants had an average of 7 years of experience working in their discipline and, on average, 1 year of experience working in EI. In all, 39% of participants were early childhood special educators, and the rest were early childhood developmental specialists (11%), speech–language pathologists (16%), occupational therapists (16%), physical therapists (6%), elementary educators (6%), or social workers (6%). We provide detailed demographic information in Table 1. All participants had access to the technology required for the professional development initiative including video cameras, telephones, computers with audio and video capability, and Internet access in their region.
Participant Demographic Information
Note: EI = early intervention.
Procedure
Each participant selected a family on his or her caseload and obtained informed consent from the family to be video recorded and participate in educational research. The researchers also obtained informed consent from all participating EI providers. The Institutional Review Boards at the university and the state agency approved all consent forms for participating caregivers and EI providers. Providers then video recorded a “typical” home visit in its entirety with their selected family and either uploaded it to a password-protected website (www.dropbox.com) or mailed a portable storage device to the researchers. These videos provided an initial data point for each participant. Participants submitted subsequent home visit videos to the researchers at a rate of one per month (or less often) for 8 months. These videos each provided an additional data point to track changes in use of caregiver coaching strategies and routines as the context for embedded intervention.
On receipt of each home visit video, the expert mentor contacted the peer mentor team within 24 hr to confirm receipt of the video and schedule a feedback session. The expert mentor participated in a feedback session with each peer mentor team for each video submitted (e.g., If a provider submitted four videos, he or she participated in four feedback sessions with the expert mentor and his or her peer mentor). The expert mentor scheduled feedback sessions with peer mentees as soon as possible after video receipt, but the length of time varied according to the availability of the mentor and mentees. Feedback sessions were approximately 1 hr in length, used a feedback fidelity checklist, addressed each provider’s video-recorded home visit session, and discussed peer feedback with each other.
Video coding
Undergraduate and graduate student research assistants (RAs) coded all submitted home visit videos, which ranged in length from 30 to 75 min each. RAs coded videos in 30-s intervals using the FGRBI Coding System, developed by the researchers. Using this coding schema, RAs assigned a code to describe the coaching strategy and the routine observed in each interval. The coding schema includes operational definitions of 11 caregiver coaching strategies and 16 routines or activity contexts in which intervention can occur. Project personnel trained RAs to apply the coaching strategy and routine definitions using video recordings drawn from other sites. To establish reliability between coders, two coders rated videos independently and attained a minimum of 80% agreement prior to scoring the DMM videos. Once we collected DMM videos, we established and maintained interrater agreement by having two independent coders code a random selection of 30% of all videos. We calculated percentage of agreement for each video as the total number of coding agreements/agreements plus disagreements. Results indicated an overall rate of intercoder agreement of 80% or greater for routine and caregiver coaching coding. Information on the FGRBI coding system is available from the second author on request and is published in other studies (Salisbury, Cambray-Engstrom, & Woods, in press; Woods & Kashinath, 2007; Woods, Kashinath, & Goldstein, 2004).
Video clips for performance-based feedback
After review of each home visit video, the expert mentor edited two video clips for each participant on a team. Video clips were approximately 1-min long and selected as a context for discussion of two specific goals established by the participant during the previous feedback session (e.g., increase use of caregiver practice with feedback, participate in caregiving routines with the family, etc.). For each video, the expert mentor selected one clip as an example of the participant using a caregiver coaching strategy or embedding intervention into a family routine with the child and family. The other clip illustrated a missed opportunity for caregiver coaching or intervention embedded in routines and served as a context for problem solving and reflection.
Performance-based feedback
A feedback fidelity checklist served as a guide for delivery of performance-based feedback based on participants’ video-recorded home visit sessions. After an opening exchange between the peer mentors and the expert mentor to review goals, the expert mentor shared performance-based feedback, engaged in problem solving and action planning, and facilitated self-refection with participants for approximately 30 min each. Following each feedback session, the expert mentor sent an email to each participant containing a typed summary of the feedback session, including the clips of the videos watched and the jointly developed goals and plans for the next session.
Use of technology for feedback
We used two technology-supported formats, Skype and conference call, to support performance-based feedback via distance mentoring. Skype is a free Internet-based application that enables live video chat and screen sharing (www.Skype.com). If providers were unfamiliar with Skype, the first author helped them set up an account over the phone and practiced until they were comfortable using the technology. After setting the stage for the feedback session by reviewing individual goals and sharing data on routines and coaching strategy use, the expert mentor shared her screen via Skype and viewed the prepared video clips for performance-based feedback together with the provider team. Live video chat and video clip viewing were supported with one to two PowerPoint slides with text highlighting key points (e.g., what to watch for, reflective questions) before and after each video clip. The expert mentor prepared slides and video clips ahead of time using a free online application (www.voicethread.com) and accessed it from a password-protected site online during the Skype session. The expert mentor emailed a video file with the edited slides and video clips to each provider following the Skype session along with the written summary of feedback from the Skype session.
Conference call feedback sessions differed from Skype sessions in that providers did not watch clips from their home visits during feedback sessions but, instead, the expert mentor verbally prompted them to recall an activity or interaction that occurred during their submitted home visit video. Rather than watching video clips of the provider interacting with the child and caregiver and viewing slides, the expert mentor shared verbatim feedback (i.e., the exact words the provider said during the session) to illustrate examples or facilitate reflection and problem solving.
Measures
Caregiver coaching in routines
RAs coded each home visit video according to the FGRBI coding system as previously described and assigned a code for the caregiver coaching strategy used by the provider for the majority of each 30-s interval. RAs coded guided or caregiver practice with feedback and problem solving by frequency of occurrence as these strategies did not typically meet the 30-s coding requirement. A code of “Other” was assigned when interactions between the provider and caregiver did not relate to the child or EI. RAs then entered the caregiver coaching codes for each provider into a database and collapsed codes into five categories (see Table 2) for the purpose of analysis.
Caregiver Coaching Data: Paired Samples Results—Differences Between First and Second, and First and Fourth Videos
p < .01.
Routine coding
RAs also coded all videos according to the FGRBI coding system for routine type. Routines were categorized by interactions that a child, provider, and/or caregiver participated in that had the potential for identifiable outcomes, predictable sequence, repetition, multiple turns, and a clear beginning and end. RAs assigned routine codes according to activities that lasted for the majority of each 30-s interval, and in which the child took two or more turns in interacting with an adult. For example, an interval might be coded as hygiene-related as the child washes her hands before snack, with the next interval coded as food-related as the child begins to eat.
Fidelity of feedback sessions
Accountability and consistency of mentors is an essential component of the mentoring relationship (Rikard & Banville, 2010). The researchers used a fidelity checklist for feedback sessions to ensure that the expert mentor provided video and email feedback in a consistent manner to all participants. The development of this fidelity checklist was necessary to maintain congruence between workshop content and mentoring. The second author completed the feedback fidelity checklist via live observation for 30% of all feedback sessions. Feedback fidelity was 100% for all checked sessions.
Participant satisfaction and social validity
At the completion of the study, the first author emailed all of the 34 original participants asking them to complete an anonymous online survey to share their thoughts and experiences on technology use and participating in feedback sessions. The survey consisted of 11 questions total, the majority of which used Likert-type (e.g., Please rate the following components of the project in terms of how helpful they were in supporting your learning: very helpful, somewhat helpful, slightly helpful, not at all helpful) or ranking scales (e.g., Please rank the following components of the project in order of importance: first, second, third, etc.), with two questions being open ended.
Analyses
Analyses of the video data occurred sequentially. First, we evaluated the effectiveness of technology-supported, performance-based feedback using paired sample t-test analyses. We performed initial analyses to determine if there were differential effects on provider change due to feedback format type (participating in feedback via Skype or conference call). There were no significant differences in provider performance changes related to feedback format, indicating that participants made similar changes using conference call and Skype. Second, we collapsed the data from technology-specific groups (Skype or conference call) to a single group. We then performed t tests to determine the effects of participating in technology-supported performance-based feedback (in either format) using pretest and subsequent video samples. Specifically, we compared providers’ first and second videos to detect changes following one feedback session, and insofar as video recordings of four home visits were available for all participants, we examined the first and fourth videos to ascertain the trajectory of change (i.e., improvement vs. maintenance relative to Video 2).
Results
Comparisons of caregiver coaching data are displayed in Table 2. There was a significant decrease in child-focused intervention between participants’ first and second videos (t = 4.15, p < .001) and first and fourth videos (t = 3.27, p < .001). A significant increase was observed in the use of specific coaching between participants’ first and second videos (t = −3.92, p < .001) and first and fourth videos (t = −2.80, p = .01). Significant differences were not observed between participants’ first and second, or first and fourth videos for CIS, JI, or interactions categorized as “Other.” We calculated effect sizes to provide further information about the changes in coaching strategy use between participants first and second, and first and fourth videos. Large effects (Cohen, 1992) were observed in the areas of specific coaching strategies (first and second video d = −1.28; first and fourth video d = 0.97) and child-focused interactions (first and second video d = 0.92; first and fourth video d = 1.02). Percentages of 30-s video intervals coded by caregiver coaching strategy are displayed in Figure 1.
Percentages of coaching strategies used in home visit videos.
Comparisons of routine-category data are displayed in Table 3. There was a significant decrease in the amount of play from the first to the second home visit video (t = 2.06, p = .05) and a significant increase in family/community routines (t = −2.03, p = .05) in the first to the fourth video. Medium effects were observed in play (first and second video d = 0.63), caregiving (first and second video d = 0.54), and family/community (first and second video d = 0.5; first and fourth video d = 0.62) routines indicating less play and increased use of caregiving and family/community routines after the first video. Percentages of 30-s video intervals coded by routine category are displayed in Figure 2.
Routine Data: Paired Samples Results—Differences Between First and Second, and First and Fourth Videos
p ≤ .05.
Percentages of routines used in home visit videos.
Participant satisfaction and social validity
A total of 24 of the 34 original participants completed the online survey. It was not possible to identify surveys completed by the 18 participants who met criteria for data analysis for this study due to the anonymous nature of the survey. When asked about the most satisfying aspect of distance mentoring, the majority of providers mentioned the interactive nature of the feedback (n = 18, 78%). Providers also noted watching video clips of their own home visit sessions (n = 7, 30%), convenience (n = 3, 13%), and the ongoing (n = 3, 13%) and positive nature (n = 4, 17%) of mentoring. One participant wrote,
I enjoyed the Skype sessions as it was interactive and fun to watch the video clips together and then discuss them. I was able to learn a lot by watching myself interact with the family as well as by watching my partner’s video clips.
Most participants noted time and scheduling difficulties as challenges to distance mentoring (n = 15, 65%), whereas only a few listed technology issues (n = 2, 9%). Other challenges included the lack of face-to-face interaction (n = 2, 9%) or motivation (n = 2, 9%). When asked to rank order the components of DMM in the order of importance (with 6 being the “most important” and 1 being the “least important”), participants rated workshops first (M = 5.3), Skype feedback second (M = 4.2), watching video clips from their home visit sessions third (M = 3.4), and conference call feedback, feedback notes via email, and monthly newsletters rated fourth, fifth, and sixth, respectively. The majority of respondents agreed or strongly agreed (n = 20, 87%) that they could easily access the technology needed for distance mentoring and that their participation in distance mentoring helped them to meet professional goals (n = 20, 87%). All participants reported that they would participate in a distance mentoring opportunity in the future (n = 23, 100%) for professional development if available.
Discussion
In this study, we examined the effects of a multicomponent professional development approach on EI providers’ use of caregiver coaching strategies and a variety of routines for embedded intervention. In addition, we looked at participants’ satisfaction with the various components of the DMM approach. We found that providers demonstrated a significant increase in use of specific coaching strategies and decrease in time spent in child-focused intervention after initial workshop training and participation in one feedback session. Providers maintained changes after four feedback sessions. Considering recent findings that suggest EI providers typically spend the majority of time in child-focused intervention during home visits (Campbell & Sawyer, 2007; Peterson et al., 2007), these results offer an encouraging demonstration of the utility of this multicomponent approach that includes technology-supported mentoring. We also observed changes in the types of routines used with families during home visits after participating in workshops and performance-based feedback sessions with the greatest expansion in use of family and community activities.
Consistent with recent investigations (e.g., Basu, Salisbury, & Thorkildsen, 2010; Colyvas, Sawyer, & Campbell, 2010), the present study identified several themes regarding home visiting practices of EI providers. CIS and JI were the most frequently observed coaching strategies, with specific caregiver coaching strategies (direct teaching, demonstration, guided practice with feedback, caregiver practice with feedback, problem solving, and reflection) seldom used in the pretraining videos. Both CIS and JI are important to a family-centered, consultative approach because they indicate a collaborative interaction with caregivers rather than direct intervention with the child. However, CIS and JI do not illustrate the EI provider’s role in teaching specific intervention strategies to caregivers to implement in everyday activities between visits. The use of specific caregiver coaching, such as providing feedback as the caregiver interacts with the child, can increase the caregiver’s use of intervention strategies (Friedman et al., 2012).
We found that providers used multiple routine types at the initiation of the study; intervention occurred in play, caregiving, and pre-academic routines and activities, with play as the predominate context. The increased use of family and community activities for embedded intervention may be interpreted as a logical extension of the child and family’s day and environments beyond the home and support generalization opportunities for the child and family.
One of the many challenges in studying the practices of EI providers in home-based settings is the individualized nature of the process. As home visits are individualized to address child and family priorities, it is generally accepted that the amount of time spent in various coaching conditions will vary widely. However, a shift from the minimal use of specific caregiver coaching interactions to more and varied types within each home visit could be an indirect index of increased interaction with the caregiver as an intervention partner with the child.
This study extends evidence on mentoring to EI providers in home-based settings. It supports recent research indicating that performance-based feedback can be effective when delivered from a distance via technology for classroom teaching personnel (e.g., Pianta et al., 2008; Powell, Diamond, Burchinal, & Koehler, 2010). Additional insights about the potential value of the DMM approach can be gathered from the participants’ survey responses about components of the model. Interestingly, a majority of participants rated training workshops as the most important component of the overall initiative. Although this was somewhat surprising because of the personal relationships forged during the time spent in the conference call and Skype feedback sessions, it may be related to familiarity with the workshop format for professional development (Bruder et al., 2009), or may be related to providers’ perceptions that workshops offer opportunities for networking and collaboration with peers. Most providers selected Skype as their preferred feedback format over conference call and selected Skype feedback and watching video clips from their own sessions as being more important than conference call feedback. This may indicate that participants preferred the face-to-face interaction but that it was not necessary to make programmatic change as indicated by negligible difference between Skype and conference call feedback formats on participants’ use of caregiver coaching strategies and diverse routines for intervention.
Limitations
One limitation of this study was that most participants video recorded interactions with only one family on their caseload. The use of a consistent caregiver–child dyad interacting with the provider over a series of home visits was intended to mediate the variability (e.g., family emergencies, child illness) that may occur within home visits. While we observed gains in specific coaching strategies with the individual family identified for this study, the impact of participation with additional families or of generalization of the practice changes to other families is unknown. We also did not control for the length of time providers had worked with the family they selected and thus do not know how familiarity may affect the relationship between the caregiver and the provider.
A further limitation of this study is that we did not measure child outcomes. Although providers who participated in at least four feedback sessions showed evidence of change in practice, we cannot determine if these changes resulted in any positive impact on the child or family. To strengthen the impact of future research in this area, in particular in relation to caregiver coaching and routines-based intervention, it will be important to consider the effect on child and family outcomes. Participation is also an important consideration and limiting factor in this study. High rates of participant exclusion from data analyses are not unusual in field-based studies. For example, in a classroom-based professional development study, 65% (113 of 173) of originally participating teachers were included in data analysis (Pianta et al., 2008). Although it is disappointing that 16 of the original 34 participants (47%) were not included in the present study due to inconsistent and infrequent video recording, it was not unanticipated as each provider scheduled with a single target family in contrast to classroom teachers who interact with students daily. Maintaining schedules and consistency of home visits for both families and participants is challenging.
A final limitation is that the large-scale multicomponent design of this initiative (e.g., workshops, Skype/conference call feedback, newsletters, email feedback, etc.) does not allow for analysis of the individual components without influence from the other pieces. While this study measured the impact of workshops and technology-supported performance-based feedback with mentors, the effect of these activities without the other components cannot be determined.
Future Directions
The challenges of research in the natural environment examining the relationships between families, EI providers, and the intervention process should not preclude further efforts. Data from this study were useful in identifying changes that could be expected within a short time, but do not allow conclusions on sustained changes in practice, or additional changes that may result from continued mentoring. Additional data points could provide more information about continued changes and maintenance of caregiver coaching and routines as the context of embedded intervention. Addressing barriers to participation in DMM may support increased participation, allowing for a larger sample and more videos included in analyses.
Future research related to individual components of DMM may also be useful in “unpacking” this particular approach to determine which components promote what types of practice change. For example, it would be interesting to investigate how peer mentors can support practice changes such as the use of problem solving and planning with caregivers or how web conferencing with the caregiver and the provider could increase the fidelity of specific interventions. Moving from child-focused intervention to increasing collaboration with caregivers may be accomplished with more resource efficient mentoring. As technology becomes increasingly accessible, future research may include tools that allow for immediate performance feedback such as handheld digital devices to connect the EI provider and mentor in real time, allowing for immediate feedback and increased opportunities to practice during home visits.
Footnotes
Acknowledgements
The authors wish to thank the service providers and administrators who participated in this study for their collaboration, as well as the research assistants who helped with data collection and analysis.
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a contractual agreement between the authors, Florida State University, and the State of North Dakota Department of Human Services, and by a grant from the U.S. Department of Education, Office of Special Education Programs—Project LIFE: Leadership in Family-Centered Early Intervention Personnel Preparation.