Adapting an Evidence-Based Early Childhood Tier 2 Social-Emotional Learning Intervention for Web-Based Delivery

Initial Development.

Initial drafts of the BEST in CLASS-Web training materials included the eight teacher training modules. These draft modules were incorporated into a website and included video scripts and interactive knowledge checks for teachers. Initial drafts were also created for the BEST in CLASS-Web teacher manual and procedures for web-based implementation of the practice-based coaching components. To develop these initial drafts, the research team consulted other existing web-based and in-service multi-media training materials and websites as examples of web-based training programs, and consulted with experts in the design of web-based professional development materials.

Focus Groups and Interviews

Focus groups and semi-structured interviews were conducted with teachers, administrators, and coaches to examine the professional development needs of early childhood practitioners, gauge “buy in” for BEST in CLASS-Web prototypes, and gather input about technology barriers and resources teachers and schools needed for implementing a web-based program (e.g., availability of Wi-Fi and technology support).

Teachers who had prior experience with BEST in CLASS-PK, delivered in an on-site format, were recruited in-person and online from existing community partnerships with federal- and state-funded early childhood providers to gather information about how materials could be adapted for web-based delivery. During the focus group sessions, teachers were asked about their insights into the potential format of modules, technology barriers, and necessary implementation adjustments. Across the two research sites, a total of 11 teachers participated in focus groups. Teachers were compensated $50 for participation.

Teachers who were naïve to BEST in CLASS-PK were recruited separately to participate in additional focus groups. Teachers were recruited in-person and online from federal- and state-funded early childhood programs (e.g., Head Start). Teachers were asked for their input on the web-based design and prototypes of the training and coaching components of BEST in CLASS-Web. Across the two research sites, a total of nine teachers naïve to BEST in CLASS-PK participated. Teachers were compensated $50 for participation.

Early childhood program administrators and teaching staff, who collaborated previously on BEST in CLASS-PK research studies, were recruited online from the same federal- and state-funded early childhood programs. At the first research site, there were nine participants who engaged in the structured interviews. At the second research site, there were three participants. Participants had a range of expertise and included a Head Start Director, a Head Start Family Administrative Coordinator, education coordinators, education specialists, preschool specialists, and a technology support staff member. Finally, at both research sites coaches who had previous experience with BEST in CLASS-PK (two coaches per site) were interviewed for their input on implementation and design issues. Participants were compensated $50 for participation.

Feedback Analysis and Development Decisions

Feedback from all focus groups and interviews were digitally recorded and transcribed. They were then reviewed to identify emerging patterns and themes by identifying: (1) issues related to the content and design of BEST in CLASS-Web training and coaching materials; (2) website-usage strengths and barriers; and (3) supports needed for implementation. These themes yielded a number of suggestions, including consideration of: (1) teachers' readiness for online training; (2) availability of program's support for online training; (3) accessibility of coaches to provide training and follow-up support; (4) use of a variety of interactive features and learning modalities for professional development that match teacher characteristics and needs; (5) availability of ongoing technical assistance for problem solving; (6) use of real-life examples and video exemplars to illustrate practices, and (7) strategies to ensure a strong coach-teacher relationship and alliance.

Based on these suggestions, the following revisions were made to the BEST in CLASS-Web materials. First, a system was developed to monitor frequently asked questions within the training modules to ensure that teachers received the technology support they needed in a timely manner. This addressed concerns about teacher readiness for online learning, the need to provide ongoing technical assistance, and the availability of technology support for online training. Additionally, study staff were trained to answer questions and solve technology issues that arose. In response to feedback about website content, text on the website was reduced, modules were shortened, a variety of activities and learning modalities were included, and pictures featuring real-life classrooms and teachers were added. To illustrate, knowledge check activities (e.g., multiple choice questions after watching a video exemplar) were included at the end of each module to gauge teacher understanding. Participants also emphasized the importance of a strong coach-teacher relationship and having accessible coaches. A meet and greet session was added to the BEST in CLASS-Web coaching schedule to facilitate teacher and coach rapport and trust building, contributing to the development of a strong coach-teacher relationship. For more accessible coaches, a button for “contacting your coach” was also added to the website to facilitate coach and teacher communication.

Expert Panel Review

Following revisions to the materials based on feedback from the focus groups and structured interviews, an expert panel was consulted. The purpose of this panel was to review and provide feedback on BEST in CLASS-Web materials and website components. Panel members were recruited online and were nationally recognized researchers with expertise in at least one of three areas: EBD, early childhood, or web-based professional development. Expert reviewers were asked to review three modules and provide feedback via an online platform. Research assistants reviewed the expert feedback to identify themes and compared their reviews to reach a consensus. Initial feedback themes were synthesized by the study team. Expert reviewers were compensated $500 for their time.

Feedback from the expert reviewers included the following themes and suggestions: (1) decreasing complex language; (2) operationalizing the teacher-coach model to clarify roles and responsibilities of teachers and coaches; (3) clarifying examples and non-examples of Tier 2 behaviors; (4) improving the explanation of the coaching process; and (5) including clearer descriptions of key concepts at the beginning of each module. In response to expert feedback, the language in the modules was made more user friendly, expectations for teachers and coaches in the coaching process were made explicit, and the descriptions of child behavior and teaching practice examples were changed to better illustrate the Tier 2 level. High-quality exemplar videos from real-life early childhood classrooms were also added to each web-based module. These videos illustrated teachers' use of all of the BEST in CLASS-PK practices and were created through a partnership with a production company.

In addition to the further development of the teacher training modules and coaching procedures, two BEST in CLASS-Web Technology User manuals were developed to facilitate and communicate protocols: one for coaches and one for teachers. Each technology user manual extensively detailed each platform involved in BEST in CLASS-Web participation. Further, each technology user manual described steps and instructions for how to navigate and troubleshoot each task associated with BEST in CLASS-Web (i.e., completing modules, uploading and sharing teaching videos, observing and coding videos, filling out questions and forms, creating and sharing coaching plans, and video conferencing).

Second Round Focus Group Sessions

We conducted a second round of focus group sessions with the same participants who attended the first focus groups (i.e., teachers experienced with BEST in CLASS-PK, teachers naïve to BEST in CLASS-PK, experienced coaches). Across the two sites, this second round of focus group sessions included twelve teachers with previous experience with BEST in CLASS-PK and six teachers naïve to BEST in CLASS-PK. During this second round of focus groups sessions, we asked participants to review the revised BEST in CLASS-Web website design and prototypes to obtain further input for refining the content, screen designs, and functionality for usability and feasibility testing. Teachers and coaches also compared the information available in the technology user manual to the actual steps taken to complete each task to suggest gaps in information, unnecessary information, and information that needed to be added.

The previous technique, used in the first round of these focus group sessions, was used again to review these interviews and identify themes. Feedback from the second round of focus group sessions included suggestions about improving the readability of the teacher technology user manual, the website home page, the content of module 1 (i.e., the introductory module), and general feedback about the content and usability of all the modules. With regard to the technology user manual, experienced BEST in CLASS-PK teachers reported that the manual was long and might be too overwhelming for teachers to use. Both groups of teachers liked the visuals used (e.g., photos and added exemplar videos) and that the manual was broken down into week-by-week practice sections. Suggestions for how to improve the manual included creating a video of someone using the technology (instead of relying only on written text), re-arranging some sections of the manual, adding clarity to specific sections (e.g., adding headers and more explicit language), and including a digital copy of the manual on the website. Based on this feedback, edits were made to the teacher and coach technology user manuals to reflect changes made to modules and procedures.

Teacher reports were mixed regarding the visual appeal of the website homepage. A group of naïve teachers reported that the page was organized, simple and self-explanatory whereas two other groups (one naïve, one experienced) reported that the page was not engaging and lacked visual appeal. Suggestions for how to improve the home page included: (1) adding additional detail about the purpose and functions of each element of the page (e.g., buttons); (2) adding buttons and functions (e.g., password reset option and home button); and (3) re-arranging elements of the page (e.g., module text at the top of the page). From this feedback, we added check marks next to each module to show if the module has been completed, section buttons on the home page so teachers could jump to sections within the modules, and a progress bar for the modules that were in the process of being completed.

For module 1, naïve teachers tended to report that the explanations, narrations, and examples were clear and useful. Experienced teachers, however, reported that the module was a bit overwhelming, some videos were not as high quality as others, and that the knowledge checks were too easy. Suggestions for improving module 1 included: (a) adding printable sheets in the resource section; (b) adding color to visuals; (c) adding photos and video-recordings when possible; (d) including a completed action plan; and (e) including a checklist of next steps. From this feedback, we trimmed the length of the narration and density of the text in each module. We also edited the knowledge checks to increase the level of difficulty and tried to incorporate more variety with the types of knowledge check activities. We created links to documents in the resources section, and inserted photos throughout the modules to increase the visual appeal. Finally, we created an end slide for each module to include a checklist of the next steps.

In-House Alpha and Beta Testing

Alpha and beta testing of each component of the website was conducted, including all teacher training and coaching materials. These tests were designed to assure the website functioned seamlessly as intended. For alpha testing, two participants at each site conducted testing. For beta testing, study staff played the role of two coaches and two teachers at each site and conducted the testing in an interactive way. A teacher tester was paired with a coach tester to complete the testing and the pair engaged in coaching activities while role-playing as a coach and a teacher in each module. Teacher and coach testers used a spreadsheet to check the corresponding content of the website. During these tests, each tester used a written log to guide troubleshooting and refinement.

Usability Testing

Subsequently, usability testing was conducted to evaluate comprehension, user appeal, and system performance. Seven teacher users were recruited in-person from early childhood teacher preparation programs and local early childhood classrooms across each research site and they agreed to participate for two weeks. Pilot users were presented with nearly all online components of the intervention implemented on a denser schedule to minimize time between sessions. Users were given a set of tasks to complete and data were gathered to track their participation. Teachers also interacted with a coach during the two-week trial. Teachers were asked to provide web-based feedback following usability testing. This feedback requests asked about teachers' level of understanding of the BEST in CLASS-PK practices and the appropriate steps for completing the modules. Questions also addressed the ease and difficulty in completing the modules (e.g., “What made this module difficult to understand? And what makes the module easy to understand? Please describe in as much detail as you can.”). Teacher were compensated $100 for their time.

Phase 1 Final Model Integration

Data from the alpha, beta, and usability testing were synthesized (e.g., the written log created during alpha and beta testing; themes identified from web-based feedback) to increase the validity of model revisions and refinement made in the next phase of development.

Coaches reported that they generally had a clear understanding of their tasks before the initial meeting with teachers. However, one coach noted that some of the specific details needed further clarification. For example, this coach was unclear on how a coach should coordinate teacher paperwork and what was needed from the teacher. Overall, coaches described their first coach-teacher meeting as positive and easy; one coach reported that it was easier than expected to develop rapport.

Coaches also reported that teachers found the technology challenging (e.g., viewing resources online, viewing videos, screen sharing during videoconferencing, and gaining access to each website and internet platform used). A problem with viewing videos was identified in other feedback sources (e.g., interviews and teacher feedback) as well. Other common challenges included coordinating a time to meet and using multiple online platforms for document sharing. Recommendations for improving the website included moving toward a shared platform (e.g., both teacher and coach use only websites), adding greater clarity to the coach technology user manual, and improving the coaching dashboard function on the website.

Overall, teachers reported the modules were clear, organized, and concise. Teachers appreciated the overall appearance of the modules, the video examples, the open response questions at the end of the modules, and the knowledge checks. Additionally, teachers thought the content was friendly, informative, and detailed. They reported that the short videos and interactive games were engaging and that the open response questions and interactive activities made the modules engaging and provided meaningful practice.

Teachers also provided constructive feedback. Teachers reported that the overall use of graphics, color, and visuals to support the narration of the modules could be improved. Teachers thought that the narration was repetitive with the text on the screen, and that the pages could be more visually engaging, colorful, and include higher quality graphics. Teachers also reported that the modules would be improved if the teachers could receive more “real time” feedback about their responses.

Based on the coaching difficulties encountered and feedback from coaches and teachers about the storage of video recordings, we transitioned to a coaching platform called TORSH Talent (www.TORSHTalent.com). TORSH Talent is a website designed specifically for facilitating teacher professional development. It includes a layout for organizing coach-teacher shared resources and a mechanism for coaches to provide point-in-time feedback on teachers' self-submitted videos of their use of practices in the classroom. As we transitioned to the TORSH Talent website, we heavily revised the BEST in CLASS-PK coach and teacher technology user manuals to reflect the new platform.

Feasibility Testing

Teachers were recruited in-person at each site to participate in feasibility testing. At the first site, five teachers and ten focal children participated. At the second site, three teachers and five focal children participated. All teachers taught in early childhood education classroom, 87.5% were female, 50% were Black/African American, 37.5% were White, and 12.5% were Latino/a. Participating teachers had taught on average for 15.25 years (SD = 11.5, range 2–33). Teachers ranged in education level, 37.5% had a master's degree, 37.5% had a bachelor's degree, and 25% had a teaching credential. Teachers were compensated $100 for participation. To screen children for eligibility, participating teachers nominated the top five children in their classroom who demonstrated problem behaviors by ranking them using the Early Screening Project (ESP; Feil et al., 1995). Children were eligible to participate if they: (1) were between the ages of 3- to 5-years old and enrolled in a participating teacher's classroom; (2) were one of the two children per classroom with the most elevated risk score as ranked by the ESP (Feil et al., 1995; Walker et al., 2014); and (3) had caregiver consent. Pre-test data were collected with all teachers, children, and coaches prior to teachers beginning modules and before any coaching sessions began. Pre-test measures included measures from the coach, the teacher, and child direct assessments (see Conroy et al., 2021b for a list of measures).

During the feasibility study, teachers worked independently through the modules and participated in practice-based coaching sessions with their coach. Technical issues with the modules or coaching session functioning were continuously monitored by the study staff. When technical issues did arise, the BEST in CLASS-Web staff met with TORSH Talent staff and the organization specializing in technology-assisted professional development to troubleshoot and solve issues. Adaptations needed for the technology protocol or procedures were tracked weekly and revised in a working draft version of technology user manuals. Information about site-specific internet challenges was also collected and tracked to further improve the technology tools and requirements needed to successfully implement the intervention.

Following the feasibility test, post-test measures were collected from teachers and coaches as well as individual surveys about their experiences during the BEST in CLASS-Web feasibility study process. Teacher surveys included questions about overall impressions of each module, what participants liked, what made the module difficult or easy to understand and use, and suggestions for changes. Unfortunately, many of the teachers included in the feasibility study had significant issues accessing the internet in their early childhood program sites. As a result, only two teachers completed these surveys. Coaches were asked about challenges experienced in teacher-coach meetings, challenges their teachers encountered, suggestions for website improvement, and understanding of coach expectations.

Data Analysis and Development Decisions

Feasibility test data were reviewed and included an assessment of pre-/posttest measures (i.e., teacher and child outcomes) and implementation fidelity measures (i.e., coach and teacher). A summary was created and shared with the project team to discuss progress, barriers, and suggested changes for the future pilot test. Additionally, feedback from coaches and teachers on implementation was summarized and shared with the team. Interviews with coaches and teachers yielded several main themes. These themes and the response to the phase 2 data are detailed below.

First, in response to the manuals, coaches reported that the BEST in CLASS-Web teacher technology user manual was helpful for understanding procedures. Teachers reported that they struggled to remember that there were two different manuals (one for using the technology, one for learning the BEST in CLASS practices) and felt that the manuals were too long. Teachers reported that they rarely used the manuals because they were not easily accessible; however, they did find the table of contents and photos helpful in the technology user manual. In response to these concerns, the BEST in CLASS-Web teacher technology user manual was shortened and made more user friendly.

Second, teachers and coaches continued to experience technical difficulties that created barriers to efficiently and feasibly use the website and virtually attend coaching meetings. At one site, many teachers did not have access to reliable wireless internet in their early childhood program. This lack of access impeded their ability to visit the BEST in CLASS-Web website, use TORSH Talent, virtually meet with their coach, or access email. Several participants felt that using videoconferencing was the most difficult aspect of the project. To address these concerns, iPads^® were pre-programmed for teachers with their individual accounts set up prior to starting the project in the pilot phase. Additionally, Wi-Fi was tested at each participating center. When the internet was not sufficient, mobile Wi-Fi devices were provided that enabled high-speed internet access for participating teachers.

Another theme that emerged was about the platform TORSH Talent, which was used to upload video recordings of the teachers' implementation of the BEST in CLASS practices and related coaching documents. Feedback from the coaches and teachers regarding TORSH Talent was that the platform was useful for accessing all of the BEST in CLASS-Web resources in one place (e.g., manuals and forms). Teachers also enjoyed the exemplar video library. Teachers reported that having the program automatically upload their videos (as opposed to teachers manually going through the upload process) and seeing anecdotal comments were helpful. Teachers and coaches did not use TORSH Talent for communication, citing that it was easier to meet through videoconferencing and email. Some teachers and coaches reported that the platform was not easy to use at first and that there were a lot of steps to completing each task (e.g., creating comments and uploading documents). To address these concerns, the number of action steps required of teachers was reduced for tasks such as uploading and naming videos. Procedures for the coaches to use when accessing TORSH Talent were also standardized so coaches followed the same procedures with all teachers.

Finally, regarding the modules on the website, teachers and coaches reported that they were quick, straightforward, and helpful for understanding the practices. However, teachers reported that the knowledge checks were not helpful for increasing their understanding of practices and that they were unaware if they were answering correctly based on automatic feedback provided by the website. In response, the feedback generated for knowledge checks was altered. For example, previous feedback was generic “thank you for your thoughts,” “thank you for your answer,” and revised feedback is more explicit “Yes, making sure your use of rules is enthusiastic will communicate you’re pleased with the focal child's behavior.”

Final version of BEST in CLASS-Web website and prototype

After the feasibility test, themes from teacher and coach surveys, feasibility issues, and teacher interview summaries were used to finalize content for the BEST in CLASS-Web website and manual prototypes. Final website changes included formatting text and making language consistent across module pages. Additionally, protocols and technology user manuals were updated with new technology information (i.e., changes due to technical problems encountered in the feasibility test and updates to the online platforms).

The coach and teacher technology user manuals were finalized to better serve the needs of coaches and teachers. For instance, one significant change after the feasibility test was the use of a “coaching workbook” for the BEST in CLASS-Web coaching process. The workbook was developed to facilitate ease of coaching and included all the forms and manual sections that are needed for a coaching meeting. A need for an administrator manual also arose, which included information about how to set up users for each web-based platform, how to set up technology protocols for the project, and an Appendix with study-related forms.

Teacher, Coach, and Expert Panel Feedback

Throughout the iterative design process, we solicited participant and expert feedback on BEST in CLASS-Web prototypes. This feedback was critically important in informing model revisions and clarifying program procedures. Specifically, this feedback assisted with improving the appeal and readability of the online modules, simplifying coaching procedures, and providing technology support to teachers and coaches. Teacher and coach input also underlined the importance of technology user manuals when creating online intervention platforms.

In planning for participant and expert feedback, the time commitments needed to revise material was an important consideration. For example, in phase 1 an expert panel review was planned to occur earlier in the development process but was postponed. We found it most helpful to receive feedback from the experts after the draft materials were in final draft form. Similarly, a second round of expert review was planned but not conducted. Revisions to the modules and website took longer than expected, which delayed the alpha, beta, and usability testing according to the initial project timeline. Given the adjusted timelines for these tests and the substantial revisions that occurred following these tests, the team decided that conducting the feasibility test prior to obtaining further expert review would be beneficial. Thus, the modules were not sent out for expert panel review a second time prior to feasibility testing. Finally, at the conclusion of feasibility testing individual web-based feedback forms were distributed to teachers instead of previously planned focus groups. Teachers worked through the online training modules at various paces. It was most beneficial to receive detailed feedback at the individual teacher level and for each module. Allowing for this type of flexibility in soliciting participant and expert feedback greatly improved our ability to revise the materials and adapt the program to participant needs.

It is also important to note that due to the rapid nature of the iterative process and our critical need to receive feedback from practitioners in order to quickly translate this feedback into actionable development work, we did not conduct formal coding or thematic analyses of interviews, focus groups, or feedback forms. Instead, we held informal interviews, focus groups, and solicited expert advice to leverage this feedback for the development project. The informal nature of this approach also meant that we did not collect demographic data on interview participants (e.g., years of experience and education level) in a more rigorous manner. Future development projects should look to collect detailed information about the interview participants (e.g., expert reviewers) and use qualitative data analysis methods (e.g., thematic coding analysis) to code feedback received when adapting and testing online programming.

Usability

The usability testing focused on how easy BEST in CLASS-Web was to use for participating teachers and coaches. During alpha, beta, and usability testing we experienced barriers to efficiently and feasibly using specific sections of the website (i.e., video uploading). These barriers are best summarized as ongoing technical issues with the website. Examples include coaches being unable to accurately track user progress on the modules, message their teachers, and view their teaching videos. Teachers were unable to view several videos in the modules, upload their videos that were filmed, or message their coaches. Unfortunately, these issues were not able to be fixed during alpha, beta, and usability testing. Before feasibility testing, we determined (after discussion between research sites and the organization specializing in technology-assisted professional development) that these issues would require more time than was available to resolve before the next phase of testing began.

To adapt to the technical problems encountered, as highlighted above, we moved to the integration of three platforms: the VIVI© website, TORSH Talent, and Zoom© for the feasibility testing phase. The combination of these platforms was sufficient for study needs, and overall the teachers and coaches were satisfied with these program components. However, the services were fragmented and required participants to learn to use three platforms. The field is in need of a single integrated platform where teachers can receive professional development training modules, upload videos, receive coaching through videoconferencing, and organize program forms. This platform will increase the accessibility and ease of web-based delivery.

An important promoter of usability was the teacher-coach relationship. Teachers in the pilot study reported that coaching meetings were generally positive and teachers reported high levels of alliance with their coach. Coaches were described as reliable, understanding, and patient. One teacher reported that the best part of the intervention was their coach. Some teachers reported that holding coaching meetings during the day (typically during naptime) was inconvenient because they often had competing events happening in their classroom that needed their attention. Coaches demonstrated flexibility and cooperation by working together with teachers to use a calendar and send reminder emails to teachers about upcoming meetings. Future web-based programs should consider the value of the teacher-coach relationship when designing web-based programs.

Feasibility

Feasibility testing revealed Wi-Fi problems for several early childhood program sites. The internet access at these sites was not sufficient for the teachers to access the internet and upload video recordings of their teaching practices. To address this issue, mobile Wi-Fi devices were purchased for each teacher. However, after using the mobile Wi-Fi devices for several months, the agency administrators expressed concern about the additional equipment. After meeting with the administration to discuss challenges that occurred with internet access, the instructional technology director was able to provide some of the teachers with additional access to the internet in several of the sites. As a result, the mobile Wi-Fi devices were withdrawn from the majority of the sites and teachers began using the enhanced Wi-Fi. When assessing the feasibility of future web-based delivery programs, it will be critically important to consider how to address early childhood program access to high-quality internet services and ways to provide this resource if necessary. This challenge is an important one for the field to overcome in order to increase the generalizability of and access to web-based interventions.

An additional concern with regard to feasibility is the extent to which teachers have the foundational technology skills needed to engage in web-based programming. In the focus groups, teachers had high levels of technology knowledge. However, teachers in the phase 3 pilot study varied in their technology skills. To illustrate, one participant did not have internet access at home, work, or in the community. Further, 70% of surveyed participants reported they do not ask for help when they encounter difficulties with technology. Only half of participants reported using technology on a daily basis. Technology user manuals and technology support will be an important cornerstone for successful web-based delivery of interventions. The BEST in CLASS-Web teacher technology user manual included a wide range of ‘how to’ information including how to access the website, how to upload a video, how to contact your coach, and how to log into email. It is important to note that technology is constantly evolving and these changes may impact technology user manuals and program materials. To illustrate, between the feasibility and pilot study, changes were made in the TORSH Talent platform that changed the interface. This required new instructions and images of the platform for the teacher, coach, and administrator technology user manuals. Future web-based delivery should incorporate user friendly technology manuals as companions to program manuals and intervention materials and frequently update these manuals to accommodate technology changes.

Finally, there was variability in the extent to which teachers were motivated to independently work through online modules. For example, four out of ten teachers completed the online modules, two teachers completed six modules, and four teachers completed three modules or less. Teachers spent from 13 min to 5 h in total across all online modules, and spent between an average of 17–37 min within a single module. Additionally, the amount of time for which users accessed each module slowly decreased as modules progressed. Future revisions of online modules could include interactive activities and knowledge tests in order to continue through the program and increase program compliance.