Abstract
Equipping paraprofessionals to implement evidence-based instructional practices with fidelity can enhance the education of students with visual impairments. We used a multiple probe across participants design to evaluate the ability of paraprofessionals to follow constant time delay procedures with high fidelity when teaching braille words. We delivered performance feedback by e-mail and examined student learning outcomes. All three paraprofessionals achieved 100% correct implementation of constant time delay and maintained this high level of implementation over time. All three students successfully learned the targeted words. The combination of the training and support with e-mail feedback was effective at equipping them to deliver an evidence-based practice. Paraprofessionals affirmed the social validity of the intervention and considered e-mail performance feedback as an easy-to-understand way of receiving much-needed guidance. E-mail feedback is a cost-effective and simple tool for providing constructive feedback, while also ensuring paraprofessionals deliver instruction with high fidelity.
One quarter (25.2%) of kindergarten through grade 12 students who are legally blind are considered nonreaders (American Printing House for the Blind, 2017). Students with visual impairments (VI) read at a slower rate than their sighted peers, and this disparity only increases as students advance in school (Corn et al., 2002; Wall Emerson et al., 2009). Such findings suggest that students with VI have literacy needs that are currently not being met. According to the Individuals with Disabilities Education Improvement Act of 2004, the Individualized Education Plan (IEP) for a child who is blind or visually impaired must provide for instruction in braille unless, after an evaluation of the child’s reading and writing skills, needs, and appropriate reading and writing media, the IEP team decides that such instruction is not appropriate for the child. [Sec. 614(d)(3)(B)(iii)]
One promising way to teach students with VI to potentially read with braille is through constant time delay (CTD). CTD is an instructional strategy wherein systematic and repeated prompts are used to teach a target stimulus (Head et al., 2011). In each trial, a target stimulus (cue card or picture) is presented to the participant and paired with an instructional cue (e.g., “Read the word”; Appelman et al., 2014). The instructional cue is followed by a controlling prompt (e.g., “This word is ____”) (Ledford et al., 2008); for children who are verbally imitative, a model prompt is often used. The amount of time between the instructional cue and model prompt is systematically manipulated over the course of multiple sessions until a predetermined criterion is reached (Head et al., 2011).
Indeed, five studies have demonstrated the effectiveness of using CTD to teach braille words and contractions, including teaching novel English Braille American Edition (EBAE) words, Nemeth Code symbols, and/or dot-five contractions to braille readers (Hooper et al., 2014; Ivy et al., 2017; Ivy & Hooper, 2015; Moss, 2016; Wilcox, 2014). These prior studies all involved researchers as the ones who delivered the instruction to students with VI. To promote broader use in typical schools, it is important to also determine whether school staff can learn to implement CTD with high levels of fidelity on their own. Given the prominent roles of paraprofessionals in the education of students with VI, our study focused on training these critical staff members on delivery of this promising instructional approach. For example, one survey indicated that more than 35% of the paraprofessionals were providing direct instruction (McKenzie & Lewis, 2008), and it has been found that many students with VI receive the majority of their literacy instruction from paraprofessionals (Forster & Holbrook, 2005).
However, despite the simplicity and ease of using CTD, particularly with using braille words, it should be strongly mentioned that this study implemented CTD in a controlled context. For best practice, we strongly recommend that CTD not be implemented in isolation. It should be incorporated into a comprehensive literacy program and should not be relied upon as the sole method of introducing or reinforcing braille words or contractions. Furthermore, this study does not posit that only paraprofessionals should be responsible for students’ literacy programs. Paraprofessionals should work collaboratively with the students’ teacher(s). This study specifically examines how paraprofessionals can support students through a CTD intervention when given appropriate guidance and support.
Numerous studies have highlighted the limitations of “one-shot” trainings and advocated for incorporating coaching or other forms of ongoing support into professional development (Buczynski & Hansen, 2010). Therefore, we evaluated the use of performance feedback to support paraprofessionals in implementing CTD. One intriguing way of delivering performance feedback is by using e-mail. E-mail may be more cost-effective, efficient, and feasible than frequent face-to-face meetings that typically characterize in-person coaching (Barton et al., 2013). Prior studies in other areas of special education have found performance feedback to be an effective way of providing professional development (e.g., Artman-Meeker et al., 2014; Barton et al., 2016). We sought to extend this approach to the area of braille reading instruction for students with VI.
The purpose of our study was to examine the effects of a novel form of performance feedback on the fidelity with which paraprofessionals could use CTD to teach braille words to students with VI. Our research questions were as follows:
Is there a functional relation between e-mail feedback and accurate implementation of CTD?
Does the use of CTD increase recognition of braille words for students with VI?
After e-mail feedback is withdrawn, do paraprofessionals maintain high levels of fidelity?
How do paraprofessionals view the feasibility and acceptability of e-mail feedback?
Method
Participants and recruitment
Three dyads participated in this study – each comprising a paraprofessional and a qualifying student from the local residential school for students with VI. Paraprofessionals had to: (a) work with a student who qualifies for the study, (b) be interested in learning CTD to teach braille words, (c) report no knowledge of CTD, and (d) indicate they checked and responded to e-mails daily. Students must have: (a) used English as the primary language; (b) been between 4 and 21 years of age; (c) had a documented VI, as shown in an eye report; (d) used braille as their primary literacy medium (to meet this criteria, the student must demonstrate the ability to understand that a braille cell represents a letter); (e) been able to wait 5 s and attend to a task; (f) been able to track braille line and identify a symbol that was different; (g) been able to find braille lead-in and lead-out lines; (h) been verbally imitative; (i) been able to scan and find differences in a line of braille; and (j) had normal hearing ability. The dyads were Monica and Nina, Janet and Peter, and Kathy and Keith. See Tables 1 and 2 for participant demographics.
Paraprofessional participant demographics.
Student demographics.
ADHD: attention-deficit hyperactivity disorder.
Recruitment
After we received Institutional Review Board approval, the school principal recruited three paraprofessionals and three students with whom the paraprofessionals would work. The paraprofessionals knew the students who were in the study, but did not otherwise work with them regularly. However, the study was conducted during regular literacy instruction time in the students’ regular classrooms. We focused on independent work time so the students did not miss critical instruction time. Each paraprofessional received a $200 honorarium for the extra training time and duties they took on.
The principal selected paraprofessionals based on the study inclusion criteria and who could fit intervention sessions into their current schedule; they were not selected based on their merit and perceived abilities. The principal shared study information with the paraprofessionals, who then agreed to attend the initial CTD training. On the day of the training, they reviewed and signed consent forms, and filled out a demographic questionnaire.
Setting
All experimental procedures took place at a residential school for visually impaired students located in a southeastern state. The paraprofessional implemented the instruction within each student’s regular classroom.
Materials
Braille words
Twelve target words (4 sets of 3) were chosen for each student participant. The braille teacher identified 12 words that each student did not currently know that could be used within the study. The same 12 words were used for each student; word set A (about, better, carry), B (bring, clean, done), C (cut, draw, fall), and D (far, got, hurt). Contracted braille was used. These words were chosen from the third-grade level of the American Printing House for the Blind’s sight word assessment list. The braille teacher reported that all three students were at similar braille-reading levels (i.e., around grades 2 and 3 for literacy). The selected words were of varying lengths as well as tactually distinct, meaning that braille letters that are very similar to each other were avoided across words.
Experimental design and procedures
We used a multiple probe across participants design (Gast & Ledford, 2014). During the baseline phase, observations were staggered and took place at least three times per week for each dyad. We graphed procedural fidelity data for the paraprofessionals (see Figure 1) and used visual analysis to examine changes in level, trend, and variability. Once the first paraprofessional (i.e., the first tier) had at least five data points and stable data during the baseline phase, the e-mail feedback component was introduced. After high levels of fidelity were demonstrated, we withdrew the e-mail feedback to determine maintenance of accurate use of CTD.

Graph of the adult participants and their percentages of procedural fidelity when implementing constant time delay.
Procedures of the study
The study had four phases: initial CTD training, baseline, intervention, and maintenance.
Initial CTD training
Paraprofessionals participated in a 60-min initial training addressing how to implement CTD. We developed a voiceover PowerPoint as the training format (available by request), as this ensured the training could be used by others in the school in the absence of a coach. We developed a training script, recorded the PowerPoint, and solicited feedback from two faculty members. The training was delivered prior to beginning the baseline phase because the paraprofessionals lacked knowledge of CTD and our focus was on evaluating the impact of the addition of performance feedback.
Baseline
The baseline phase began immediately after the initial training. We asked paraprofessionals to implement CTD independently with no assistance; each had a form on which to collect data. Aside from being instructed to start with 0-s sessions, they received no feedback and were asked to implement CTD with their target student at least three times per week. Each session typically lasted between 3 and 5 min.
Intervention
Intervention sessions were identical to the baseline phase with one exception. After each observation was complete, the lead author sent an e-mail to the paraprofessional containing feedback about what was done correctly and incorrectly in the implementation of CTD. Each e-mail included the same components: greeting with positive comment, data with supportive feedback, corrective feedback, response request, and closing encouraging statement. The e-mail feedback was sent within 1 hr after each session in the intervention phase. We asked paraprofessionals to respond to the e-mail to confirm it was read. The intervention continued until at least three stable data points – and a minimum of five data points – were collected.
Maintenance
The e-mail feedback component was withdrawn and we asked paraprofessionals to continue implementing CTD with their target student. At least three maintenance data points were collected for each dyad.
Data collection
We measured the procedural fidelity of the paraprofessionals using live observations with pencil-and-paper data collection forms. The length of the observations was roughly 3–5 min, depending on whether the participants were in 0- or 5-s sessions. The lead author was present during all intervention sessions to collect primary data. The lead author, at the time of the study, was a doctoral student with several years’ experience of both research, supervising and conducting, and instructing students in braille. A secondary observer – a graduate student in special education – was also present to collect reliability data for 33.4% of sessions. This second observer had 5 years of experience as a general education classroom teacher, had successfully completed the braille course, and had 1 year of practicum experience working with students with VI. She also collected data on the training sessions and on 100% of the e-mail feedback.
During each observation, observers entered the setting prior to the start time and observed throughout its duration. Observers sat or stood quietly to the side where the focus student and responses could be clearly seen and heard, but where they were not obtrusive.
Response definitions: procedural fidelity
The primary dependent variable was the fidelity with which the paraprofessionals implemented CTD with their target student. Procedural fidelity data was gathered via a form and was taken on the following steps:
Word card centered on rubber mat;
Attending cue (paraprofessional states: “find end of lead-in line”);
Instructional cue (paraprofessional states: “read the word”);
Physical controlling prompt (paraprofessional places his or her hand over the word, but allowing student to find lead-in line);
Appropriate time delay and mental count (paraprofessional chooses the appropriate time delay for the session; for example, if it is a 5-s session, the student is given up to exactly 5 s to respond);
Verbal controlling prompt (paraprofessional states: “This word is . . .”);
Appropriate instructional feedback (paraprofessional gives the correct response based on whether student gave a response that was an unprompted corrected, unprompted incorrect, prompted correct, prompted incorrect, or no response).
The following one-time per session behaviors were also recorded for fidelity purposes.
Procedure initial
Randomized Contraction Card Order (the word cards have been shuffled prior to the start of the session, so the student is not presented with the same order of words each time);
Word Set Correct and Complete (the set of word cards are correct and complete);
Proper Placement of Rubber Mat (the rubber mat is placed centered in front of the student);
Appropriate Desk Height (the desk is the correct height to allow the student to comfortably read the braille cards with proper braille-reading technique);
Braille Intact (the braille on the word cards is intact and has not been rubbed off or damaged);
Correct phase (0 s, 5 s) has been selected for the session.
Procedure intervention
Student Greeted;
Appropriate Scripted Directions (e.g., the paraprofessional tells the student that they will be asked to read the word and then given time to read the word if they are currently in a 5-s session);
Student’s Feet Flat on the Floor;
Student Dismissal.
Because two paraprofessionals were not familiar with the braille code and had no previous braille experience, certain aspects of proper braille technique were included with the procedural fidelity steps. These steps included such items “student’s feet flat on floor” to maintain proper posture. The braille cards were also cut on one corner so the paraprofessionals could always correctly place the cards for the students to read and ensured braille was never presented upside down. The percentage of procedural fidelity was calculated by adding all of the successfully demonstrated behaviors in a session and dividing by the complete total of behaviors possible in the session times 100%. For each session, there were a total of 10 one-time occurring steps and a total of seven multiple-occurring steps that could occur up to 12 times each, for a total of 84 steps per session. These steps, when added together, meant that a total of 95 steps could occur per each session.
Interobserver agreement
We collected interobserver agreement (IOA) data during observations across all dyads and study phases for a minimum of one third of the observations. We scheduled IOA observations randomly while balancing across participants and phases. IOA was calculated for procedural fidelity using point-by-point agreement by dividing the total agreements by the total agreements plus disagreements and multiplying by 100%. For baseline, IOA was collected during 11 sessions (40.7% of all baseline sessions). IOA ranged from 93.7% to 100% (M = 97.1%). For intervention, IOA data were collected for nine sessions (60% of all intervention sessions). IOA ranged from 92.6% to 100% (M = 99.1%). For maintenance, the secondary observer collected IOA data on five sessions (42.8% of all maintenance sessions). IOA ranged from 96.8% to 100% (M = 99.4%).
Social validity
At the end of the study, each paraprofessional completed a social validity questionnaire asking whether they felt that the e-mail feedback was effective, if the training was adequate, if they felt the students successfully learned braille words, if the e-mail feedback was necessary, if they would recommend e-mail feedback to others, and if e-mail feedback was more effective than previous strategies. These six questions could be answered on a 4-point scale consisting of strongly agree, agree, disagree, and strongly disagree. There were additionally two open-ended response questions, where paraprofessionals addressed what “the best part of getting this coaching was” and “something that could improve the training and coaching.”
Results
Is there a functional relation between e-mail feedback and accurate implementation of CTD?
A functional relation was demonstrated between the introduction of e-mail feedback and increased percentage of procedural fidelity in which the adults implemented CTD (see Figure 1). All three paraprofessionals had a steady decline in percentage of procedural fidelity after the initial training during the baseline phase. Moreover, Monica and Kathy implemented with less fidelity over time. During the baseline phase, Monica’s fidelity ranged between 42.1% and 58.9% (M = 52.5%), Janet’s fidelity ranged between 45.3% and 53.7% (M = 50.7%), and Kathy’s fidelity was much more variable, starting at a high percentage of 86.3%, ranging between 52.6% and 86.3% (M = 68.4%).
During the baseline phase, the errors paraprofessionals made were also variable. Generally, all three paraprofessionals were able to consistently maintain 100% fidelity with the one-time occurring behaviors, except once for “Appropriate Directions are Given for the Session” with Kathy. All three were also able to maintain 100% fidelity with step 1 of the multiple-occurring steps. From there, paraprofessionals had some variability with the steps they typically missed during baseline, but there were some steps they shared in common for the bulk of their errors. For example, within the multiple-occurring steps, Monica tended to miss steps 2 (Attending Cue: find end of lead-in line) at 57 errors (95%), 3 (Instructional Cue: read the word/letter) at 60 errors (100%), and 4 (Physical Controlling Prompt: placing hand over the word, but allowing student to find lead-in line) at 51 errors (85%). Janet also tended to miss steps 2 (126 errors; 95.4%), 3 (131 errors; 99.2%), and 4 (132 errors; 100%) while Kathy mostly missed steps 3 (132 errors; 100%) and 4 (120 errors; 90.9%). The varying number of sessions in the baseline phase account for the opportunities given to each paraprofessional to make errors. The common pattern for errors involved missing steps 2 (total of 183 errors), 3 (total of 323 errors), and 4 (total of 303 errors).
The introduction of the e-mail feedback corresponded with clear and substantial increases in procedural fidelity. All three paraprofessionals had an immediate increase in fidelity with no overlapping data points with the baseline phase. Janet and Monica showed an immediate and substantial level change, increasing to 100% immediately upon introduction of the e-mail feedback and maintaining this level throughout the rest of the study. After a steady decline and ending baseline at 52.6%, Kathy increased her fidelity to 74.7% to 96.8% to 100%, which was then maintained until the conclusion of the study.
Does the use of CTD increase recognition of braille words?
All students were able to learn the braille words during the intervention, as shown by mastery with three data points at 100% in a 5-s session. However, a functional relation could not be established between the introduction of e-mail performance feedback and accurate braille word identification. The student data are reported via unprompted correct responses (see Figure 2), which could only occur during 5-s sessions, meaning that all data reported at 0% were during 0-s sessions and anything above that was during a 5-s session. The students were able to demonstrate three 100% unprompted correct data points, which allowed them to move onto the next word set. Students were able to quickly go through the 0-s sessions and then move onto the 5-s sessions with only one small dip for each student during a 5-s session. For Nina, this decrease occurred during maintenance. For Peter, this decrease occurred during intervention. For Keith, this decrease occurred during baseline. After each decrease, in which all of the students incorrectly read one word, they all increased to 100% unprompted correct responses. This learning occurred without requiring paraprofessionals to implement the intervention with 100% fidelity.

Graph of the student participants (Nina, Peter, Keith) and their percentages of unprompted correct responses. Open squares represent data collected during 0 s. Solid circles represent data collected during 5 s.
Social validity
The paraprofessionals agreed or strongly agreed with all six statements, affirming the social validity of this form of performance feedback. When asked to write about the best part of receiving the coaching, one paraprofessional reported that the e-mail feedback was very helpful and it “helps me to be reminded of what I normally forget when watching training videos.” The second paraprofessional said the e-mails were helpful as a reference tool. The third stated that the e-mails helped reinforce proper technique which increased the student’s success rate at learning braille words. When asked for recommendations, two paraprofessionals said that having handouts directly from the training session would be helpful. One paraprofessional felt the e-mail feedback was perfect for the CTD intervention and said no changes were necessary.
Discussion
The purpose of this study was to extend prior research addressing the effectiveness of research-implemented CTD with students with VI (e.g., Ivy et al., 2017; Ivy & Hooper, 2015) and the effectiveness of performance e-mail (e.g., Artman-Meeker et al., 2014; Barton et al., 2016). Our focus was on improving the extent to which paraprofessionals could independently and accurately implement an evidence-based instructional intervention with students with VI. Specifically, we addressed (a) the effectiveness of e-mail feedback in increasing procedural fidelity in a CTD intervention, (b) the use of CTD for teaching braille word recognition for students with VI, and (c) the feasibility and acceptability of this intervention.
Our findings have important implications for practice. First, paraprofessionals must be given additional and ongoing supports alongside initial training on instructional practices. As evident in the baseline phase of our study, a one-shot training was not enough to ensure that paraprofessionals could successfully implement an intervention like CTD. For example, Kathy was able to initially achieve fairly high-fidelity percentages initially, but it decreased over time. So even if the training is somewhat successful, paraprofessionals still support. As a result, this study makes a very compelling case for making sure educators are given both proper training and ongoing supports.
Second, the versatility of the CTD intervention makes it a viable option for both teachers and paraprofessionals. Given the way the braille index cards were created – with braille for the students and a printed word for the adults – almost anyone in the school would be able to administer this intervention. Such individuals could include paraprofessionals, who typically do not know braille, general education teachers, and even classroom peers. All of these individuals could be working on the students’ braille skills without having to actually know braille themselves. This versatility ensures that the students with VI are working on and learning their braille, even if a braille expert is not there to administer the intervention.
Third, the effectiveness of e-mail feedback could be attractive to school districts as a support tool because it can be a simple approach to use. It is a cost-effective and simple way to convey feedback to adults on how to improve their practices. In addition, e-mail offers a way to convey feedback that is simple and available for later reference in the future. Two of the paraprofessionals anecdotally reported referring back to the e-mail feedback prior to CTD sessions during the course of the study. The approach is also cost effective. For example, we specifically created a voiceover PowerPoint so that the training could be held in future without a member of the research team needing to be there to conduct the training. In addition, since e-mail feedback could be offered via distance, school districts could enlist assistance from coaches in other locales, thus not being restricted to personnel within their own districts. Observations could occur live via Skype or the paraprofessionals could record and upload sessions for their coaches to observe. It negates the need to have schedule and have an in-person observation, which can be both time and cost consuming. During this study, observers specifically did not interact with paraprofessionals or students; this was done to stimulate what could be accomplished via video or distance, which would allow for potentially cost-effective measures to take place by taking out the need for a live person to be present during the observations.
Fifth, using CTD as an intervention for teaching braille intervention should certainly be used with caution and only in certain contexts. In the present study, the students seemed to excel with identifying the braille words with each student experiencing only one small dip during a 5-s session that he or she was then able to recover and prove mastery with three 100% sessions during consecutive 5-s sessions. Although the students themselves did not have any mistakes in common, the mistakes were all very similar – they would read the first letter of the word and if it was the same beginning letter as a previously learned word from a previous word set, the student would identify that word as a previously learned word. It is also worth noting that the students did appear to be learning words even in baseline, despite the fact that paraprofessionals were not implementing CTD with 100% fidelity (see Figure 2). Although paraprofessionals were not implementing all of the steps from the training, they seemed to understand the basic structure enough to present the words to the students and have them read the words. Perhaps exposure alone was enough for the students to learn the words within the context of the intervention. Future research is needed to examine what level of fidelity is needed with CTD or even other interventions to improve student outcomes. It is often automatically assumed that 100% fidelity is needed for an intervention to have an impact on students, but that may not always be the case.
Limitations
Several limitations to this study should be noted. First, the student participants selected for the study differed from the original design. When the study was initially proposed, we had hoped to focus on students with VI who also had an intellectual disability. However, only Nina had an intellectual disability. Second, the paraprofessionals only worked with their target students and used CTD when observed by the research team. Ideally, it would be beneficial to see adults were able to maintain high fidelity after a more extended period of time has passed, but these data could not be collected. Third, this study was conducted at a specialized school and not within any other educational settings, so generalizability of the findings may be limited without further research.
Future research
Future research should build upon our study. This study examined words used within the context of language arts. Future studies should draw upon other subjects from which to identify words (e.g., mathematics, science). Second, this study took place at a residential school for students with VI. Future studies should look at other settings, such as in resource rooms in typical schools or with the varying settings of itinerant teachers. Third, studies should examine paraprofessionals who have an established relationship with the student.
Conclusion
Findings from this study add to the growing literature on the effectiveness and feasibility of the use of performance feedback. Adults showed high levels of fidelity when receiving e-mail feedback and reported high levels of satisfaction with the intervention. This intervention holds promise in helping adult professionals in education to successfully implement CTD with high fidelity, which will hopefully translate into success for the students with whom they work.
