Using Text Messaging to Deliver Performance Feedback to Preservice Early Childhood Teachers

Abstract

The purpose of this study was to examine the effectiveness of using text messaging to deliver performance-based feedback (PF) to preservice teachers working in inclusive early childhood classrooms. A multiple baseline across behaviors single-case research design was replicated across four participants to examine the relation between PF delivered via text message and preservice teachers’ use of self-selected target behaviors. Results indicated text messaging was an effective method for delivering PF and subsequently increasing teacher target behaviors when individualized to meet the needs of teachers; however, results were variable across teachers.

Keywords

single-subject designs experimental studies preparation personnel

In the past two decades, the expansive disconnect between research in early intervention/early childhood special education (EI/ECSE) and the practices implemented with young children and their families has been highlighted by several researchers (Metz & Bartley, 2012; Odom, 2009). For example, the field of EI/ECSE has a set of recommended practices that are “based on the best-available empirical evidence as well as the wisdom and experience of the field” (Division for Early Childhood [DEC], 2014, p. 3). As the field of EI/ECSE works to minimize the research-to-practice gap, the factors that promote practitioners’ use of recommended practices must be identified and defined. One way to support practitioners in implementing recommended practices (e.g., descriptive praise, choices, language expansions; DEC, 2014) is through professional development (PD) that is comprehensive, collaborative, linked to instructional goals, and sustained (Sheridan, Edwards, Marvin, & Knoche, 2009; Snyder, Hemmeter, & McLaughlin, 2011). Although traditional forms of PD (e.g., workshops, multiple day trainings, conferences) have been shown to be ineffective (Joyce & Showers, 2002), a lack of consensus exists regarding what constitutes effective PD.

One component of PD with expanding empirical support is performance-based feedback (PF; Fallon, Collier-Meek, Maggin, Sanetti, & Johnson, 2015). Fallon and colleagues conducted a meta-analysis of PF and concluded that it was an evidence-based practice for increasing the intervention fidelity of instructional personnel across a variety of authentic educational settings. In fact, several reviews have determined PF to be an effective practice for improving teacher instruction and fidelity to interventions (Casey & McWilliam, 2011a; Kretlow & Bartholomew, 2010; Solomon, Klein, & Politylo, 2012). For example, PF that is provided as a follow-up to initial training has been shown to be effective in influencing teachers’ behavior (Casey & McWilliam, 2008). Artman-Meeker and Hemmeter (2013) found that teachers increased their use of strategies to prevent challenging behavior after didactic training sessions were followed by ongoing PF delivered via email. Likewise, Pianta, Mashburn, Downer, Hamre, and Justice (2008) found that web-mediated PF increased the quality of teachers’ interactions with students.

In EI/ECSE, PF has been delivered in a variety ways, including in vivo during coaching sessions with parents (Kaiser & Roberts, 2013), face-to-face immediately following teacher observation sessions (Schepis, Reid, Ownbey, & Parsons, 2001), via written feedback forms and checklists used to review behaviors immediately after an observation (Barton, Chen, Pribble, Pomes, & Kim, 2013; Casey & McWilliam, 2011b), and using technologies such as email to deliver written feedback (Barton, Pribble, & Chen, 2013; Barton & Wolery, 2007; Hemmeter, Snyder, Kinder, & Artman, 2011). Casey and McWilliam (2011a) found that verbal PF (alone or in conjunction with other modes of PF delivery) was the most common form of PF used. However, they noted two limitations with verbal PF: (a) the delivery of PF requires practitioners to leave their classroom or shift their attention, which might necessitate additional staff and (b) verbal PF does not create a permanent record that might be useful for reviewing and recording progress.

One solution to the impermanence of and personnel required to provide verbal PF is the use of digital or wireless technologies. A burgeoning literature exists on the use of email as an effective method for delivering PF in ECSE settings (Barton, Fuller, & Schnitz, 2016; Barton, Pribble, & Chen, 2013; Barton & Wolery, 2007; Hemmeter et al., 2011). For instance, Barton and Wolery (2007) and Barton and colleagues (2016) demonstrated that coaching using email PF resulted in an increase in self-selected target behaviors for ECSE preservice teachers. Likewise, Artman-Meeker and Hemmeter (2013) found a functional relation between email PF and teaching teams’ use of strategies to prevent challenging behaviors. Furthermore, Rathel, Drasgow, Brown, and Marshall (2014) demonstrated that first-year teachers provided a greater ratio of positive-to-negative comments with students when provided with specific PF via email.

Other means of delivering PF via wireless technology in ECSE settings include using annotated videos of teachers’ performance (Powell, Diamond, Burchinal, & Koehler, 2010) and using bug-in-ear (BIE) PF to coach educators on their implementation of communication strategies (Coogle, Rahn, & Ottley, 2015). For example, Hemmeter and colleagues (2011) included a link to an exemplar video clip in their PF emails; each clip provided a model of appropriate use of the target behavior in typical preschool settings and was referenced in the corrective feedback portion of emails. Similarly, Artman-Meeker, Hemmeter, and Snyder (2014) used video clips as part of a study comparing distance coaching versus workshop training alone on Head Start teachers’ implementation of practices that promote children’s social–emotional competence. Coogle and colleagues (2015) effectively used BIE technologies to increase teachers’ use of recommended practices. However, BIE, video, and web-based technologies require resources (e.g., specialized devices or software, technological expertise) that might preclude their ubiquitous use in ECSE settings.

Although there is accumulating information regarding ECSE PD (Schachter, 2015; Snyder et al., 2011), research on training preservice teachers to prepare for their roles as classroom teachers has been limited (Barton & Wolery, 2007; Barton et al., 2015). Given young adults aged 18 to 29 years (the majority demographic for preservice teachers) are more likely to own a smartphone and use text messaging to communicate than any other age demographic (Smith, 2015), text messaging might be a particularly useful mechanism for delivering PF to preservice teachers. Similar to email in its omnipresence and ease of use, text messaging is a potentially feasible method for delivering immediate PF to teachers. Researchers have examined the use of text messaging as a response medium to encourage student engagement (Cheung, 2008; Markett, Sanchez, & Weber, 2006), improve attendance and completion of parent training programs (Murray, Woodruff, Moon, & Finney, 2015), and increase communication between parents and teachers (Pakter & Chen, 2013). However, no studies to date have examined the use of text messaging as a mechanism for sending PF in ECSE.

The current study examined the relation between PF provided via text messaging (hereafter: text message performance feedback [TMPF]) and preservice teachers’ use of targeted behaviors in ECSE classrooms. The following research questions were addressed:

Research Question 1: Following brief regularly scheduled observations, does notification of the target behavior result in an increased level of preservice teachers’ use of target behaviors?

Research Question 2: Does TMPF increase levels of preservice teachers’ use of target behaviors?

Research Question 3: Will preservice teachers’ increased levels of target behaviors maintain in the long-term (1–3 months after the end of intervention), during covert observations, and generalize to settings outside the classroom (i.e., playground)?

Research Question 4: Do levels of child engagement change when teachers’ use of target behaviors increases?

Method

Participants

Four preservice teachers (i.e., working toward teaching licensure) were recruited from a university-based inclusive preschool program after obtaining human participant approval from the institutional review board. The teachers were recruited for participation in the current study based on a recommendation from the school director. All four teachers were completing paid fellowships in their assigned classrooms during the course of the study. The inclusion criteria were that the preservice teacher must (a) currently work in a classroom, (b) be in good academic standing, (c) have unlimited text messaging, (d) demonstrate infrequent use of target behaviors (fewer than five instances of each behavior in two 20-min screening observations), and (e) read and respond to texts daily. Preservice teachers were screened for inclusion through questionnaires, direct observations, and responses to three text messages sent over 3 days (to ensure they read their text messages). To be included, preservice teachers were required to respond to each text message within 6 hr from the time it was sent.

The four participants were females, first-year students in a special education graduate program working toward a master’s degree and teacher licensure. Tara was 23 years and White/Non-Hispanic, Michelle was 21 years and White/Non-Hispanic, Chelsea was 23 years and White/Latina, and Heather was 21 years and White/Non-Hispanic. None of the participants had any paid experience in ECSE settings prior to their current placements as classroom teaching fellows. The participants’ coursework had included the use of recommended practices (DEC, 2014), including the target behaviors in this study. However, none had received systematic PF on their use of selected target behaviors.

Two coaches served as the primary data collectors; three graduate students in special education collected reliability data. The coach for Tara, Michelle, and Chelsea was a female, second-year master’s degree student in special education; the coach for Heather was a female, first-year doctoral student in special education. One coach was White/Latina, and the other was White/Native American. Both were trained research assistants working toward behavior analysis certification and had paid experience as teachers in ECSE classrooms. Although no formal coaching training occurred, a faculty member in special education reviewed the target behaviors with each coach prior to starting the study.

Other participants in the study included children in the classrooms. Tara’s classroom had 10 children aged 24 to 41 months, including two with Individualized Educational Programs (IEPs). Michelle’s classroom had eight children aged 12 to 19 months, with one with an Individual Family Service Plan (IFSP). Chelsea’s classroom had seven children aged 12 to 19 months, including two with IFSPs. Heather’s classroom had eight children aged 12 to 22 months, with one with an IFSP. In addition to the teaching fellow (i.e., the study participant), each classroom also had a lead teacher and an assistant teacher.

Settings and Materials

The study took place in four classrooms of a university-based inclusive preschool. Observations occurred three to five times per week in the classroom during regularly scheduled free play. During this time, children engaged in a variety of typical preschool activities (e.g., blocks, books, dramatic play). Observations for Heather occasionally occurred during regularly scheduled art activities. The art activities included different materials across sessions but were similar to free play in that they were open ended, not teacher directed. Participants’ generalization of target behaviors was measured on the outdoor playground or in the indoor gym, dependent on weather conditions. The playground included typical outdoor play equipment (e.g., swing set, slides, tricycles, climbing structure); the indoor gym included typical indoor gross motor play equipment (e.g., mats, balls, small riding toys). Covert observations were conducted from the observation booths (for Chelsea and Heather only), which were adjacent to each classroom; they contained a one-way window and speaker system, allowing data collectors to see and hear classroom activities without their presence being known. Data were collected in vivo using a researcher-developed data collection form and were graphed daily using Microsoft^® Excel^TM. Data were collected for Tara and Michelle concurrently. Data were collected for Chelsea immediately after data collection ended for Michelle, and for Heather, 2 months after data collection ceased for Michelle.

Response Definitions and Measurement

The primary dependent variable was the frequency of participants’ use of self-selected teaching practices. After participants consented to participate in the study, the coaches presented each with a list of six discrete verbal behaviors and instructed them to rank the behaviors according to those they were most interested in using and receiving TMPF related to their use. The behaviors were descriptive praise, choices, language expansions, emotion labeling, play expansions, prompts for social interactions, reminders of behavior expectations, and redirections (see Table 1). The target behaviors were related to recommended practices (DEC, 2014), could be measured objectively, were likely to occur in free-play contexts, and were used infrequently by participants during prestudy observations. The top three ranked behaviors were then identified as the target behaviors for each participant.

Table 1.

Definitions and Examples of Target Teacher Behaviors.

Target behavior	Definition	Example
Choice	A vocal choice between two objects, materials, or activities to one child (contextually relevant and functional)	During art, the teacher asks a child whether she wants to use a “paintbrush” or “hands.”
Play expansions	Imitating child’s play action with a new related play action, and mapping 1–4 words or vocalization on to the expanded action	A child drives a toy car. The teacher puts a figure in a car and says, “doll in car.”
Emotion labeling	A vocal label of emotion relevant to current situation and directed at one child	A child is smiling and jumping, and the teacher says, “You look so excited!”
Redirections	A vocal direction to engage in an alternative response, motor action, or play action that is more appropriate than the words, motor actions, or play actions the child is currently doing	At lunchtime, a child bangs the table with his fork and the teacher says, “You can eat with your fork.”
Language expansions	A vocalization that repeats and extends child’s given communication; the expansion can contain 1–4 words plus child’s vocalization	In blocks, child points and says “block.” The teacher picks up the block and says, “yellow block.”
Prompts for social interactions	Vocal prompt to direct the attention or actions of one or more children to another child to elicit a social interaction	Teacher says, “Joseph, come look at all the new trucks Brian is playing with!”
Reminders of classroom expectations	Verbally restates the classroom rules and expectations directed; must be positively phrased and based on the established classroom rules	On the way outside, Teacher says, “Remember we are safe and walk in the hallways.”
Descriptive praise	Vocal statement of approval given contingent on and specific to the child’s behaviors within 5 s after the behavior	Child walked up the stairs and went down the slide. Teacher says, “Nice job remembering to walk up the stairs before sliding.”

Event sampling was used to record the frequency of the three-target behaviors during 15-min observations. Coders tallied all target behaviors observed and recorded at least five verbatim examples of the target behavior for each tier to provide PF to the participant. Momentary time sampling was used to estimate the percentage of children appropriately engaged. At the 56 s mark of each minute (across 15 min), coders counted and recorded (a) the total number of children within 5 feet of the participant and (b) the total number of those children appropriately engaged. At the 00 s mark, coders resumed recording teacher behaviors.

Experimental Design and Analysis

A multiple-baseline-across-behaviors design, replicated across four participants, was used to examine the functional relation between TMPF and participants’ use of self-selected teaching behaviors (Gast, Lloyd, & Ledford, 2014). Intervention began with the first behavior that demonstrated stability in level and trend during baseline. Intervention with subsequent behaviors commenced once the previous behavior demonstrated an increased and stable level or therapeutic trend. This procedure was replicated across behaviors for all participants. Visual analysis was used to examine the relation between TMPF and participants’ use of target behaviors (Gast & Spriggs, 2014). Specifically, six data characteristics were examined daily: level, trend, and variability were analyzed between and across conditions, and immediacy of effect, overlap, and consistency of data patterns were analyzed across conditions, tiers (i.e., target behaviors), and participants. The design met contemporary design standards with reservations due to having three, albeit stable, data points during initial baseline conditions (Kratochwill et al., 2013).

Procedures

Baseline

During baseline observations, participants were not told which target behaviors were being observed. The data collector recorded the frequency of participants’ use of all three selected target behaviors. Coaches then sent a brief text message to develop rapport between the participant and coach. Text messages included a general positive statement about the classroom activities and a request for a response (e.g., a scheduling question) to increase our confidence that the participants read text messages each day.

Notification only

To examine whether awareness of the target behavior alone would increase participants’ use of target behaviors, a notification-only condition was implemented with Michelle and Chelsea. During this condition, the researcher sent the participant a text message providing the definition of the target behavior and an example of how that behavior could be used (see Figure 1).

Figure 1.

Examples of text messages sent during baseline, notification, no feedback, feedback, and maintenance conditions.

TMPF

Intervention began once the first target behavior demonstrated a stable level of responding during baseline. Following each observation, the coach sent the participant TMPF using her personal cell phone. The TMPF included (a) a positive opening statement, (b) a frequency count of target behavior(s), (c) one verbatim example of her use of the target behavior, (d) feedback related to the target behavior, (e) a positive closing statement, and (f) a response request (see Figure 1). The TMPF for subsequent tiers included frequency counts of previously intervened behaviors as well as the current target behavior; however, verbatim examples were provided for the current target behavior only. TMPF varied in length (approximately 30–200 words per message).

Generalization and maintenance

Generalization observations occurred approximately every five to 10 sessions throughout all conditions. The coach did not send TMPF following the observations, but data collection occurred in the same manner as other conditions. Maintenance sessions occurred one to two times per week for the first month immediately following the intervention condition for Michelle, Chelsea, and Heather, and again 2 months after the intervention condition ended for Michelle. For Tara, maintenance data were collected only for redirections as her resignation from the school precluded the collection of maintenance data on additional behaviors. Text messages sent during this condition were identical to baseline.

Procedural adaptations

Idiosyncratic adaptations for all participants were made utilizing the dynamic nature of single-case research design (Barton, Ledford, et al., 2016). Data-informed adaptations were made to improve outcomes while maintaining experimental control. The specific adaptations were based on effective coaching practices (Artman-Meeker, Fettig, Barton, Penney, & Songtian, 2015).

Reminder text messages

For Tara, a reminder text message was sent on the morning of observations identifying the target behavior(s) that would be observed. Reminder texts included (a) a positive opening statement, (b) a reminder that her use of the target behavior would be observed that morning, and (c) a positive closing statement. For example, “Good morning! Just a reminder: I’ll be observing your use of play expansions this morning. Have a great day!” A frequency count of missed opportunities also was added to the reminder text message.

Fading

Fading of TMPF for Michelle commenced when the intervention began with the second behavior (to observe maintenance of the initial target). The coach sent TMPF on the evening of each observation. TMPF included a (a) positive opening statement, (b) request for a response (a scheduling question), and (c) positive closing statement (see Figure 1 for example). A frequency count of the target behavior was included in every other TMPF, and PF on the behavior focused on the diversity of her use of the target behavior (e.g., “You used a variety of descriptive praise statements today”).

Goal setting

For Chelsea and Heather, the coach set a specific goal in each message during TMPF on play expansions due to lower than expected frequencies.

Covert observations

Observations were conducted from an observation booth to examine the frequencies of target behaviors when the participants were unaware of being observed. No TMPF was sent after these observations.

Interobserver Agreement and Procedural Fidelity

To assess interobserver agreement (IOA), two coders simultaneously and independently recorded data on participants’ use of target behaviors and child engagement during at least 25% of sessions across all conditions and participants. Coders included graduate students and a faculty member in special education. Coders were all females, three White/Non-Hispanic, one White/Latina, and one Filipino/Pacific Islander, and none had an affiliation with the preschool program or were otherwise involved in the study. Furthermore, the faculty member was not an advisor for any of the participants. Before beginning data collection, coders were trained in observation procedures, and practice sessions were conducted observing teachers not otherwise involved in the study. Data collection began once coders reached at least 90% agreement on all measures across three consecutive observations. IOA was calculated using an interval-by-interval gross method for estimating IOA ([smaller/larger] × 100; Ayers & Ledford, 2014). However, agreements and disagreements were identified per 1-min intervals and totaled across the 15 intervals for each observation, which allowed for a more precise measurement of reliability than comparing the totals per observation. IOA averages are shown in Table 2.

Table 2.

Dependent Variable IOA Across Conditions for All Participants.

Participant	Baseline		Intervention		Generalization		Maintenance
Participant	Average (range)	% of sessions	Average (range)	% of sessions	Average (range)	% of sessions	Average (range)	% of sessions
Teacher behaviors
Tara	91 (75–100)	33	92 (74–100)	50	91 (60–100)	100	NA	NA
Michelle	96 (88–100)	33	90 (60–100)	46	93 (66–100)	80	89 (66–100)	60
Chelsea	94 (87–100)	66	93 (64–100)	56	88 (50–100)	43	91 (56–100)	40
Heather	96 (93–100)	25	85 (53–100)	38	87 (76–100)	40	86 (75–100)	33
Child engagement
Tara	98 (96–100)	33	91 (85–98)	50	92 (86–97)	100	NA	NA
Michelle	98 (96–98)	33	92 (80–100)	46	81 (69–89)	80	94 (87–100)	60
Chelsea	95 (87–92)	66	89 (75–100)	56	83 (77–88)	43	99 (97–100)	40
Heather	89 (82–100)	25	95 (81–100)	38	92 (69–100)	40	100	33

Note. IOA = interobserver agreement.

Procedural fidelity was measured for 93% of text messages sent by the coaches across all conditions. The coach took a screenshot of the message and sent it via text to a coder, who recorded yes or no for the following criteria: (a) sent on the evening of the observation, (b) began with a positive opening statement; (c) included a frequency count, verbatim example, and strategy for using the target behavior; (d) included a request for a reply; and (e) concluded with a positive statement. During baseline and maintenance conditions, text messages were to include all components except (c). Average procedural fidelity across conditions was 98% for Tara, 99% for Michelle, 99% for Chelsea, and 96% for Heather. Tara responded to 75% of the texts, Michelle responded to 97% of the texts, Chelsea responded to 95% of the texts, and Heather responded to 94% of the texts.

Social Validity

The social validity of the study was assessed via an online survey completed by 11 raters who were naïve to the study purpose and results. All raters worked in the preschool program where the study occurred, but none were participants. Raters were asked to denote the relevance of each target behavior, as pertained to its use in a preschool classroom, using a 5-point Likert-type scale. Descriptive praise received the highest average rating (M = 5.0), and choices received the lowest average rating (M = 4.45). The social validity was high across target behaviors given all received an average rating of 4.45 out of 5 or higher.

Results

Teacher Behaviors

Tara

Tara demonstrated low, but slightly variable, levels of responding across target behaviors during baseline. TMPF began with redirections; however, Tara’s use of redirections remained at baseline levels with just a slight increase in trend. Her level of redirections remained variable with the addition of a reminder TM on the morning of observations and when the number of missed opportunities was provided in the TMPF. Similar to redirections, Tara’s use of choices remained low throughout the intervention condition. Following the fourth session, Tara disclosed via text to the coach that her classroom team did not use choices in a way that was consistent with the operational definition for the study. Thus, data collection concluded for choices, and a meeting was held between Tara and the coach to identify alternative target behaviors. Data collection began on play expansions and emotional labeling once Tara selected these as alternative target behaviors. During baseline, Tara had low and stable levels of play expansions. Use of play expansions immediately decreased on the first day of intervention, then displayed a sharp increase and remained relatively stable for four sessions. However, Tara resigned from the classroom and was withdrawn from the study, which precluded additional observations. Tara demonstrated minimal increases in levels of target behaviors during generalization sessions.

Michelle

Michelle’s levels of descriptive praise and choices were low and stable during baseline; her use of play expansions was variable (range = 0–5), but low.

Descriptive praise

For descriptive praise, a slight increase in level (range = 3–4) was demonstrated with the onset of the notification-only condition (i.e., no feedback), whereas subsequent tiers remained low and stable. With the introduction of the TMPF, Michelle’s use of descriptive praise immediately increased and maintained a stable level throughout intervention (range = 8–13) with no overlap with the previous conditions. During fading and maintenance, her level of descriptive praise slightly decreased from intervention with more variability (range = 5–12). Her use of descriptive praise increased slightly in the generalization setting with the introduction of TMPF and remained at a stable level through fading and maintenance conditions.

Choices

Her use of choices immediately increased during the notification-only condition (range = 4–8), demonstrating a gradual decreasing trend across the subsequent three sessions and no overlap with baseline. With TMPF, Michelle’s use of choices increased in level and variability (range = 3–14). During maintenance, her use of choices demonstrated a slight decreasing trend. Her use of choices in the generalization setting increased with TMPF but decreased during maintenance.

Play expansions

Michelle’s use of play expansions was low and stable during notification-only condition (range = 2–4). Her level of play expansions increased with the introduction of TMPF (range = 3–11), had one session that overlapped with the preceding condition, some variability during the maintenance condition, and a noteworthy increase during the final maintenance session. Her use of play expansions in the generalization setting did not increase until maintenance.

Chelsea

Chelsea’s use of redirections was low during baseline (range = 1–5) and the notification-only (i.e., no feedback) conditions (range = 2–4). Play expansions and descriptive praise were low during baseline with some variability.

Redirections

When TMPF was introduced, her use of redirections was initially variable and eventually stabilized, with a slight increasing trend across the final six sessions. Her use of redirections was variable during maintenance (range = 0–15) and had considerable overlap with baseline (36%, 10 of 28 data points overlapped) but increased during the final two TMPF sessions. In the generalization setting, Chelsea’s use of redirections was low in the notification-only condition and increased with TMPF. During maintenance, there was an immediate decrease in Chelsea’s use of redirections with some variability; her level eventually reached and exceeded levels observed during maintenance. During covert observations, Chelsea’s levels of redirections remained stable during intervention and maintenance.

Play expansions

Chelsea’s use of play expansions had an immediate increase but a sharp decreasing trend with no overlap with baseline during the notification-only condition (range = 12–22). Her use of play expansions did not increase with the introduction of TMPF. After the coach set a goal for her to use one to two play expansions per min, she had an immediate increase in level and minimal overlap with preceding conditions (range = 22–31). An immediate decrease in level occurred during maintenance, which demonstrated a gradual accelerating trend across the final maintenance sessions (range = 3–12). Her level of play expansions was low and variable during covert observation sessions and increased when a goal was set. She had a decreasing trend during maintenance, but an increase in the final covert observation. In the generalization setting, play expansions increased with TMPF; a sharp decrease occurred during the first maintenance session and gradually increased across the last three sessions.

Descriptive praise

An immediate increase in level of descriptive praise and a gradual increase in trend with some variability occurred during the notification-only condition (range = 12–21). With the introduction of TMPF, Chelsea had an immediate increase in level; however, she had an immediate decrease during maintenance. Her level of descriptive praise was stable during baseline in generalization settings, with a gradual increase during intervention (range = 11–18) and a slight decrease in maintenance. For covert observations, descriptive praise was low and stable during baseline (range = 0–4), had a gradual increase during notification-only and TMPF conditions (range = 3–18), and had a decrease during maintenance.

Heather

Heather demonstrated low but variable rates of all target behaviors during baseline.

Language expansions

With the introduction of TMPF, Heather’s use of language expansions demonstrated an immediate and increasing trend—stabilizing around 10 expansions per session—with minimal overlap with baseline, although use was variable throughout the intervention (range = 2–18). During maintenance, her level of language expansions remained higher than in baseline with minimal overlap. Her use of language expansions increased during generalization with the introduction of TMPF but did not change during covert observations.

Promotions of social interactions

Her promotions of social interactions immediately increased with the introduction of TMPF, and although variable, they had no overlap with baseline (range = 5–34). During maintenance, her promotions of social interactions decreased to baseline levels and then demonstrated an increasing trend. Her promotions of social interactions slightly increased during generalization and covert observations with the introduction of TMPF.

Play expansions

Heather’s use of play expansions stabilized with the introduction of TMPF (range = 3–11), however, did not demonstrate a level change from baseline. Following five intervention sessions with minimal change, a goal of 15 to 20 play expansions was set for Heather via TMPF. Following this adaptation, Heather demonstrated an immediate increase in level. Her use of play expansions was variable during the maintenance condition and ended at levels higher than observed during baseline. Her use of play expansions slightly increased during one generalization session with the introduction of TMPF but did not change during covert observations or maintain.

Overall, TMPF was functionally related to an increased level of participants’ use of target behaviors for Michelle, Chelsea, and Heather, but not for Tara (see Figures 2 –5). Experimental control was demonstrated for Michelle, Chelsea, and Heather; behavior change occurred concurrent with the intervention’s introduction, and no changes in the level of subsequent tiers (behaviors) occurred until the intervention was introduced. Michelle, Chelsea, and Heather had three intraparticipant replications with consistent latency and magnitude of change. Conversely, Tara’s behaviors remained low and variable across conditions. The minimal number of covert and generalization observations precludes identification of a functional relation for generalization data for all participants.

Figure 2.

Tara’s frequency of target behaviors.

Figure 3.

Michelle’s frequency of target behaviors.

Figure 4.

Chelsea’s frequency of target behaviors.

Figure 5.

Heather’s frequency of target behaviors.

Child Engagement

The average percentage of children engaged remained high across participants and conditions. Child engagement was high and stable for Tara (m = 91%, range = 79%–100%), Michelle (m = 98%, range = 82%–100%), Chelsea (m = 97%, range = 81%–100%), and Heather (m = 93%, range = 69%–100%) across all sessions and conditions. Levels of child engagement did not change across conditions. Graphed data are available via email from the authors.

Discussion

The purpose of this study was to examine the relation between TMPF and participants’ use of self-selected target behaviors. A functional relation between TMPF and target behaviors was identified for Michelle, Chelsea, and Heather. Their rates of target behaviors also increased in generalization settings (playground or gym) for most participants and target behaviors, with some behaviors maintaining after TMPF ceased. A functional relation was not demonstrated for Tara, which might have been due to misunderstandings about the target behaviors; she resigned from the classroom in the middle of the study, which precluded additional data collection.

Several studies have demonstrated a functional relation between PF delivered via email (Artman-Meeker & Hemmeter, 2013; Barton, Chen, et al., 2013; Barton & Wolery, 2007; Barton et al., 2015; Hemmeter et al., 2011) or other formats (checklists, Casey & McWilliam, 2011b; BIE, Coogle et al., 2015) and teacher behaviors. The current study extends the research on PF in ECSE settings by showing the efficacy and feasibility of using text messaging to deliver PF to preservice teachers. Furthermore, the current study had strong methodological rigor. For example, we collected and reported IOA for all conditions, participants, and dependent variables, and we collected and reported procedural fidelity for all conditions to measure adherence to the research protocol and differentiation across conditions. Reviews of the PF literature have pointed to rigor and procedural fidelity in particular as significant limitations (Artman-Meeker et al., 2015; Fallon et al., 2015). Furthermore, we had three intraparticipant and interparticipant replications. These positive outcomes highlight the need for individualized adaptations to coaching to ensure participants’ improvements, and continued replications are warranted (Kratochwill et al., 2013).

Feasibility of TMPF

In the current study, data were collected on three-target behaviors and on child engagement during brief 15-min observations. This method was efficient for observing multiple behaviors at a time and might be particularly feasible in ECSE and preservice contexts. Furthermore, TMPF allowed participants to review feedback at a convenient time. For these reasons, TMPF might be more feasible than face-to-face meetings. However, reading text messages could be problematic for ECSE staff who might not have time to review PF during working hours. Furthermore, the use of fading in this study extends research on the use of fading as a component of PF for special education teachers (DiGennaro, Martens, & Kleinmann, 2007). During the fading condition in the current study, the level of descriptive praise decreased slightly from the TMPF condition for Michelle but remained higher than baseline. In contrast, DiGennaro and colleagues (2007) found that levels of teacher treatment integrity remained at intervention levels when PF was systematically faded to once every other day, once per week, and once every 2 weeks, which might be a more feasible frequency in ECSE contexts and should be examined in future research.

Individualized Approaches to Coaching

We included minor adaptations to the TMPF intervention for each participant to facilitate improved behavior change. In each case, the adaptations to the intervention were conducted such that experimental control was maintained and functional relations were identified. These adaptations—which were easily implemented in the ECSE context—strengthen the ecological validity of TMPF by demonstrating its flexibility and versatility when used to improve specific, discrete teacher behaviors. Furthermore, coaching and PF should always be dynamic and responsive to individual teacher needs and allow for data-based decision making.

The current study demonstrates the flexibility of both TMPF and single-case research given teacher improvements occurred without the loss of experimental control. For example, Chelsea’s and Heather’s use of play expansions increased when goals were set. Similarly, Barton and Wolery (2007) found goal setting was effective for increasing a teacher’s use of specific praise. Although setting a goal effectively increased Chelsea’s and Heather’s use of play expansions, levels decreased during maintenance. This suggests goal setting might be an effective way to increase teachers’ use of behaviors, but teachers might not maintain high rates of behaviors. Future studies should examine the effectiveness of goal setting alone and with other coaching practices. Furthermore, levels of play expansions and descriptive praise were low during baseline for Tara and Michelle and slightly higher for Chelsea. Data were collected on Tara and Michelle at the beginning of their first semester in the graduate program; however, for Chelsea, baseline data were collected at the end of her first semester and during her second semester. Artman-Meeker and Hemmeter (2013) found linking PF to didactic instruction was effective for increasing teachers’ use of target behaviors; levels of target behaviors during baseline for Chelsea and Heather may have been higher than Michelle and Tara due to instruction on these behaviors potentially occurring prior to the start of TMPF.

Maintenance and Generalization

During the notification-only condition, some target behaviors increased above baseline levels, but all were lower than with TMPF. Preservice teachers might slightly or temporarily increase their use of target behaviors when they know they are being observed, but TMPF might result in higher rates of responding. Cossairt, Hall, and Hopkins (1973) demonstrated brief improvements in teachers’ use of praise following didactic instructions on its use, but a more drastic and long-term increase in teachers’ use of praise when they were provided PF and PF plus social praise. In addition, they found that providing PF with social praise directly after instruction led to an increase in the use of praise, which maintained. Conversely, in our current study, participants demonstrated decreased levels of target behaviors during maintenance. Providing training plus PF might be more effective for maintaining teachers’ use of target behaviors than TMPF without training, as shown by Cossairt and colleagues (1973).

Target behaviors increased in initial observation sessions of the notification-only condition for Michelle and Chelsea, but these rates of behavior did not maintain in the classroom or generalize to other settings. This suggests that notifying preservice teachers that their behavior will be observed (without TMPF) is not enough to meaningfully increase their use of behaviors. However, it is likely that teachers’ awareness that they are being observed has some impact on their use of target behaviors. For example, Chelsea’s and Heather’s levels of target behaviors were lower during covert observations than when they were aware of being observed. During covert observation, Heather was often completing other tasks such as preparing materials, communicating with families, and cleaning, whereas other teachers in the room were interacting with children. Our presence during classroom observations might have occasioned more teacher–child interactions. Future research might examine the impact of TMPF related to teachers’ knowledge of their actions being observed. Our use of covert observations is an important contribution to the PD research, but our findings suggest more robust PD interventions might be needed to produce lasting change.

Target Behaviors Amenable to TMPF

Participants in this study were provided TMPF on behaviors that have not been examined or have been minimally examined in the PF literature in EC. Although descriptive praise, choices, and language expansions are target behaviors often included in this literature, no PF studies previously included redirections, and few have examined emotion labeling (Barton, Chen, et al., 2013; Barton & Wolery, 2007; Barton et al., 2015; Hemmeter et al., 2011) and prompting social interactions (Barton et al., 2015). Although research exists on coaching preservice teachers to implement a complex play intervention (i.e., imitating a child’s play action, modeling a new play action, and verbally labeling the play action; Barton, Pribble, & Chen, 2013), no prior PF study has specifically examined play expansions as a target behavior.

Child Engagement

The rates of child engagement across baseline and intervention conditions were at or near ceiling levels (range = 69%–100%) for all participants. Also, the measurement system used to measure engagement (momentary time sampling) might not accurately capture the true engagement level of young children. Although momentary time sampling, when compared with other interval systems, has been shown to produce estimations of behavior most consistent with actual behavioral duration (Ayers & Ledford, 2014), it displays variable accuracy when large intervals are used, particularly, with behaviors that are of shorter duration than the interval (Lane & Ledford, 2014). Because the children who participated in this study were 22 to 36 months, an age at which the attention span in a highly stimulating environment is likely to be short, the interval chosen (1 min) might have been longer than the average engagement of children, and thus inappropriate for accurately capturing the engagement level of children. Future research should examine more precise measures of child engagement.

Study Limitations

Although TMPF was effective for increasing preservice teacher behaviors, the ideal rates of these behaviors in these particular settings are unknown. For instance, Chelsea tended to use fewer redirections compared with descriptive praise and play expansions. This might be because there were more opportunities to use praise and expand play. However, redirections occurred with greater frequency during generalization sessions, potentially because outdoor environments have less structure and more opportunities for redirections. For Michelle and Heather, the time of observations occasionally occurred when they were with just one child. Using high rates of some of the target behaviors (e.g., choices, promotions of social interactions) with one child might be more difficult and less authentic than with multiple children. Compared with Michelle, who sometimes interacted with only one child per interval and used between 3 and 11 play expansions during intervention sessions, Chelsea often interacted with up to five to six children per interval and used 13 to 19 play expansions during intervention sessions. Future studies should examine ideal rates of teacher behaviors in various ECSE classroom settings and activities by concurrently measuring the impact on child behaviors. For example, Hemmeter and colleagues (2011) concurrently measured teachers’ use of descriptive praise and child challenging behaviors. Although variable across teachers, the rates of challenging behaviors decreased as teachers’ use of descriptive praise increased. Future studies should continue to examine changes in child behaviors relative to teachers’ receipt of PF.

Although the participants in this study were relatively homogeneous in regard to their experiences and education, Tara’s results indicated TMPF alone was not a sufficient method for increasing target behaviors. Some teachers might need more individualized PF (e.g., live coaching, video models; Barton, Chen, et al., 2013; Hemmeter et al., 2011). Furthermore, texting might not be convenient for all teachers—especially, those who are accustomed to communicating in-person, over the phone, or via email—nor it allow for multiple inboxes or accounts as email does. Some teachers may prefer to keep work-related communications separate from their personal accounts and might not respond to text messages related to their professional life outside of working hours. Furthermore, although one advantage to TMPF is that communication can occur anywhere and anytime, these very features might lead to teachers skimming the TMPF rather than purposefully reviewing it, as they might with email feedback. Although requiring a response from teachers was intended to increase our confidence that the teachers reviewed each message, the response required was the same type of question (e.g., a scheduling question) in every TMPF, which might have been viewed as inauthentic and have increased the likelihood of automatic responding. Also, sending images via text messaging might be more cumbersome than with email; however, as text messaging technologies are rapidly improving, this is unlikely to be a limitation in the future. Future studies should identify critical components of TMPF, and strategies for ensuring the PF is reviewed.

Conclusion

This study demonstrated that PF delivered via text message produced changes in preservice teachers’ behavior. Furthermore, it extends the research on PD and PF and proposes text messaging as a new method of delivering PF to preservice teachers. The prevalence of smart phones has made text messaging a widely used form of communication, and text messaging provides a practical and relevant method for delivering PF (Smith, 2017). Likewise, text messages provide written documentation of PF and can be delivered quickly without interrupting preservice teachers’ workday. Nevertheless, continued research is needed on text messaging as a PF method before it can be broadly recommended. Future replications of the current study should include examining ideal rates of teacher behaviors, the influence of specific teacher behaviors on behaviors of children in the classroom, immediate versus delayed PF, and the components of text messages that are critical to for improving teacher behaviors.

Footnotes

Acknowledgements

The authors would like to acknowledge their deep gratitude to the children, families, teachers, and staff of the Susan Gray School and the Special Education graduate students at Peabody College, whose participation and support made these studies possible.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Artman-Meeker

Fettig

Barton

E. E.

Penney

Songtian

(2015). Applying an evidence-based coaching framework to the early childhood professional development literature. Topics in Early Childhood Special Education, 35, 183–196.

Artman-Meeker

K. M.

Hemmeter

M. L.

(2013). Effects of training and feedback on teachers’ use of classroom preventive practices. Topics in Early Childhood Special Education, 33, 112–123.

Artman-Meeker

K. M.

Hemmeter

M. L.

Snyder

(2014). Effects of distance coaching on teachers’ use of Pyramid Model practices: A pilot study. Infants & Young Children, 27, 325–344.

Ayers

Ledford

J. R.

(2014). Dependent measures and measurement systems. In Gast

D. L.

Ledford

J. R.

(Eds.), Single case research methodology: Applications in special education and behavioral sciences (2nd ed., pp. 124–153). New York, NY: Routledge.

Barton

E. E.

Chen

Pribble

Pomes

Kim

(2013). Coaching preservice teachers to teach play skills to children with disabilities. Teacher Education and Special Education, 36, 330–349.

Barton

E. E.

Fuller

E. A.

Schnitz

(2016). The use of email to coach preservice early childhood teachers. Topics in Early Childhood Special Education, 36, 78–90.

Barton

E. E.

Ledford

J. R.

Lane

J. D.

Decker

Germansky

S. E.

Hemmeter

M. L.

Kaiser

(2016). The iterative use of single case research designs to advance the science of EI/ECSE. Topics in Early Childhood Special Education, 36, 4–14.

Barton

E. E.

Pribble

Chen

(2013). The use of e-mail to deliver performance-based feedback to early childhood practitioners. Journal of Early Intervention, 35, 270–297.

Barton

E. E.

Wolery

(2007). Evaluation of e-mail feedback on the verbal behaviors of pre-service teachers. Journal of Early Intervention, 30, 55–72.

10.

Casey

A. M.

McWilliam

R. A.

(2008). Graphical feedback to increase teachers’ use of incidental teaching. Journal of Early Intervention, 30, 251–268.

11.

Casey

A. M.

McWilliam

R. A.

(2011a). The characteristics and effectiveness of feedback interventions applied in early childhood settings. Topics in Early Childhood Special Education, 31, 68–77.

12.

Casey

A. M.

McWilliam

R. A.

(2011b). The impact of checklist-based training on teachers’ use of the zone defense schedule. Journal of Applied Behavior Analysis, 44, 397–401.

13.

Cheung

S. L.

(2008). Using mobile phone messaging as a response medium in classroom experiments. The Journal of Economic Education, 39, 51–67.

14.

Coogle

C. G.

Rahn

N. L.

Ottley

J. R.

(2015). Pre-service teacher use of communication strategies upon receiving immediate feedback. Early Childhood Research Quarterly, 32, 105–115.

15.

Cossairt

Hall

R. V.

Hopkins

B. L.

(1973). The effects of experimenter’s instructions, feedback, and praise on teacher praise and student attending behavior. Journal of Applied Behavior Analysis, 6, 89–100.

16.

DiGennaro

F. D.

Martens

B. K.

Kleinmann

A. E.

(2007). A comparison of performance feedback procedures on teachers’ treatment implementation integrity and students’ inappropriate behavior in special education classrooms. Journal of Applied Behavior Analysis, 40, 447–461.

17.

Division for Early Childhood. (2014). DEC recommended practices in early intervention/early childhood special education 2014. Retrieved from http://www.dec-sped.org/recommendedpractices

18.

Fallon

L. M.

Collier-Meek

M. A.

Maggin

D. M.

Sanetti

L. M.

Johnson

A. H.

(2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case research. Exceptional Children, 81, 227–246.

19.

Gast

D. L.

Lloyd

B. P.

Ledford

J. R.

(2014). Multiple baseline and multiple probe designs. In Gast

D. L.

Ledford

J. R.

(Eds.), Single case research methodology: Applications in special education and behavioral sciences (2nd ed., pp. 251–296). New York, NY: Routledge.

20.

Gast

D. L.

Spriggs

A. D.

(2014). Visual analysis of graphic data. In Gast

D. L.

Ledford

J. R.

(Eds.), Single case research methodology: Applications in special education and behavioral sciences (2nd ed., pp. 176–210). New York, NY: Routledge.

21.

Hemmeter

M. L.

Snyder

Kinder

Artman

(2011). Impact of performance feedback delivered via electronic mail on preschool teachers’ use of descriptive praise. Early Childhood Research Quarterly, 26, 96–109.

22.

Joyce

B. R.

Showers

(2002). Student achievement through staff development. Alexandria, VA: Association for Supervision & Curriculum Development.

23.

Kaiser

A. P.

Roberts

M. Y.

(2013). Parent-implemented enhanced milieu teaching with preschool children who have intellectual disabilities. Journal of Speech, Language, and Hearing Research, 56, 295–309.

24.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26–38.

25.

Kretlow

A. G.

Bartholomew

C. C.

(2010). Using coaching to improve the fidelity of evidence-based practices: A review of studies. Teacher Education and Special Education, 33, 279–299.

26.

Lane

J. D.

Ledford

J. R.

(2014). Using interval-based systems to measure behavior in early childhood special education and early intervention. Topics in Early Childhood Special Education, 34, 83–93.

27.

Markett

Sanchez

I. A.

Weber

(2006). Using short message service to encourage interactivity in the classroom. Computers & Education, 46, 280–293.

28.

Metz

Bartley

(2012). Active implementation frameworks for program success: how to use implementation science to improve outcomes for children. Zero to Three, 32(4), 11–18.

29.

Murray

K. W.

Woodruff

Moon

Finney

(2015). Using text messaging to improve attendance and completion in a parent training program. Journal of Child and Family Studies, 24, 3107–3116.

30.

Odom

S. L.

(2009). The tie that binds: Evidence-based practice, implementation science, and outcomes for children. Topics in Early Childhood Special Education, 29, 53–61.

31.

Pakter

Chen

(2013). The daily text: Increasing parental involvement in education with mobile text messaging. Journal of Educational Technology Systems, 41, 353–367.

32.

Pianta

Mashburn

Downer

Hamre

Justice

(2008). Effects of web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23, 431–451.

33.

Powell

D. R.

Diamond

K. E.

Burchinal

M. R.

Koehler

M. J.

(2010). Effects of an early literacy professional development intervention on head start teachers and children. Journal of Educational Psychology, 102, 299–312.

34.

Rathel

J. M.

Drasgow

Brown

W. H.

Marshall

K. J.

(2014). Increasing induction-level teachers’ positive-to-negative communication ratio and use of behavior-specific praise through e-mailed performance feedback and its effect on students’ task engagement. Journal of Positive Behavior Interventions, 16, 219–233.

35.

Schachter

R. E.

(2015). An analytic study of the professional development research in early childhood education. Early Education and Development, 26, 1057–1085.

36.

Schepis

M. M.

Reid

D. H.

Ownbey

Parsons

M. B.

(2001). Training support staff to embed teaching within natural routines of young children with disabilities in an inclusive preschool. Journal of Applied Behavior Analysis, 34, 313–327.

37.

Sheridan

S. M.

Edwards

C. P.

Marvin

C. A.

Knoche

L. L.

(2009). Professional development in early childhood programs: Process issues and research needs. Early Education and Development, 20, 377–401.

38.

Smith

(2015, April 1). Smartphone ownership in America. Retrieved from http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

39.

Smith

(2017, January 12). Record shares of Americans now own smartphones, have home broadband. Fact Tank. Retrieved from http://pewrsr.ch/2jbjymk.

40.

Snyder

Hemmeter

M. L.

McLaughlin

(2011). Professional development in early childhood intervention: Where we stand on the silver anniversary of PL 99-457. Journal of Early Intervention, 33, 357–370.

41.

Solomon

B. G.

Klein

S. A.

Politylo

B. C.

(2012). The effect of performance feedback on teachers’ treatment integrity: A meta-analysis of the single-case literature. School Psychology Review, 41, 160–175.