Mixed-Reality Simulation With Preservice Teacher Candidates: A Conceptual Replication

Abstract

Given the critical importance of discrete instructional practices in special education, teacher candidates must be prepared to implement them upon entering the classroom. In preservice teacher education programs, field placements and clinical experiences rarely provide enough opportunities for preservice teachers to gain the proficiency needed to provide effective instruction. In this study, a randomized control research design was used to investigate the effects of a mixed-reality simulation experience compared with traditional classroom practice in the implementation of a system of least prompts. Results suggest that mixed-reality simulation with additional coaching supports significantly improved preservice teachers’ implementation of the prompting sequence. Social validity data collected offer insights into the use of mixed-reality simulation in practice with preservice teachers. Limitations and suggestions for future work are discussed.

Keywords

evidence-based practices mixed reality preservice teachers simulation system of least prompts

Special education teachers need to implement a wide variety of academic and behavioral strategies to support students with disabilities (Cook et al., 2019). In addition, many special educators supervise and train other staff to use evidence-based strategies (Vince Garland et al., 2016). However, knowledge of an academic or behavioral strategy alone does not automatically result in fluent implementation (Brownell et al., 2009). As such, there is a need to support preservice teacher candidates (i.e., individuals enrolled in approved preparation programs hereafter referred to as candidates) in learning discrete instructional skills (McLeskey et al., 2017). To be prepared to run their classrooms, candidates need opportunities to practice instructional techniques and receive feedback (Peeples et al., 2019).

Although preservice programs provide structured field placements, candidates have limited time to practice interventions (Leko et al., 2015). Frequently candidates spend most of their early placements observing and only work closely with students during the final placements of their teacher preparation program (Sayeski et al., 2017). In essence, the majority of early practicum experiences are observational with a limited amount of time to provide instruction and guided practice paired with feedback by a supervisor (Darling-Hammond, 2010). Leko et al. (2015) called for teacher preparation to address this issue by “constructing more opportunities for candidates to practice teaching in structured, carefully sequenced, and closely monitored practical experiences” (p. 28). In addition, schools across the country report difficulty recruiting and retaining qualified special education teachers, particularly in schools that have high levels of poverty, minority student enrollment, and in rural or remote areas (Brown et al., 2015; Cross, 2017; Mason-Williams, 2015; Sutcher et al., 2016), which by extension limits the quality special education field placements available to candidates in teacher preparation programs.

Many teacher preparation programs are exploring various technology options to augment high-quality field placements (Rock et al., 2016). In this article, we explore the use of mixed-reality simulation as a virtual learning environment to support candidates during their field placements. This article reports the results of a conceptual replication of mixed-reality simulation to support candidates’ acquisition of the system of least prompts (SLP) sequence.

Mixed-Reality Simulation

Learning in virtual environments provides unique opportunities for students to develop and practice applied skills. These simulated interactions offer candidates opportunities to refine their skills with virtual avatars in simulated classrooms while soliciting feedback from instructors and peers. Virtual learning environments are a useful tool in preservice teacher education as they allow candidates to practice techniques when they are unable to enter classrooms, such as before practicum placements or during school closures (i.e., COVID-19 pandemic, school breaks). Federal agencies also have required projects to include cost analyses to identify cost-effective and portable interventions (Institute of Education Sciences, 2020). More recently, calls from the American Council on Education ask for federal support to expand technology infrastructure as schools transition to distance learning (American Council on Education, 2020).

One form of virtual learning environments is mixed-reality simulation. Beginning in 2008 and continuing to date, there have been more than a dozen studies of mixed-reality simulation within preservice teacher education and in-service teacher development (e.g., Dawson & Lignugaris /Kraft, 2017; Dieker et al., 2008). Pioneered by Lisa Dieker and colleagues, the mixed-reality, avatar-based simulation environment provides preservice teacher candidates with a classroom-like environment, including student avatars, furniture, and whiteboards. A trained actor or interactor utilizes the student avatars to speak, move, and engage in a simulation. The interactor integrates context-specific knowledge as well as current events (i.e., weather, pop-culture, news) into the simulation.

Mixed-reality simulation has a host of benefits leading it to be described as a 21st century change driver to improve the outcomes of students with disabilities (Rock et al., 2016). Mixed-reality simulation is used within teacher education to support candidates acquiring new skills by practicing with coaching and assessing learning competencies (Dieker, Straub, et al., 2014; Ludlow, 2015). Simulation provides personalized instruction which can be tailored to candidates’ individual needs (Dieker, Rodriguez, et al., 2014; Ludlow, 2015). For example, candidates can personalize their learning goals and receive individualized coaching during and after simulations (Larsen et al., 2020). Additional benefits include repeated practice in a system that does not tire as well as constructive coaching feedback (Hixon & So, 2009). Mixed-reality simulation also allows for instructors to measure candidates’ performance of instructional techniques and identify candidates’ strengths (Dieker et al., 2008). At the same time, candidates can practice intervention techniques without potential harm to an actual student (Bradley & Kendall, 2014; Dieker, Rodriguez, et al., 2014). Candidates are also able to receive practice on target techniques used with low-incidence disabilities or diverse students who may not be present in their practicum placements but may be on their caseload when hired after graduation (Dieker et al., 2015).

Previous Mixed-Reality Studies

Given the aforementioned flexibility of the platform, mixed-reality simulation topics vary. Previous topics have included implementing classroom management strategies (Larsen et al., 2020); preventing, detecting, and responding to bullying (Pas et al., 2019); and conducting informal assessments of learning and delivering instruction (Ludlow, 2015). For example, Driver et al. (2018) investigated the impact of mixed-reality simulation on seven candidates’ perceptions of their readiness to work in collaborative settings as well as communication skills. Four simulations occurred across the 15-week semester. A paired t test indicated significant differences between pretest and posttest scores (d = 0.88), meaning participants significantly improved their perceived readiness to collaborate and communicate.

In a separate study, Larsen et al. (2020) analyzed feasibility and acceptability data from 62 candidates who practiced discrete classroom management skills (e.g., precorrections, opportunities to respond, behavior-specific praise) in the simulator. Candidates completed one simulation during which they were accompanied by a coach (e.g., professor, graduate student). In this mixed-method study, the authors reported most candidates indicated their desire to practice behavior management skills in the simulator. However, a few participants reported avoidance-related goals such “getting it over with.” Participants reported feeling anxious prior to the simulation. Larsen et al. (2020) also reported nonsignificant changes in anxiety, relevance, and utility before and after the simulator. Larsen et al. (2020) called for future mixed-reality studies to address sources of student anxiety prior to simulations. They suggest easing anxiety by (a) providing introductory demonstrations with the instructor modeling teaching in the simulator, (b) explicitly making connections to one or two discrete teaching practices, (c) simplifying materials and shorter lesson plans, and (d) providing teacher candidates with multiple opportunities to practice the same skill. The authors of this study call for participants to interact in the simulator in small groups rather than independently. Larsen et al. (2020) prompted future research in mixed-reality simulators to track fidelity of implementation of coaching sessions.

Researchers have started pursuing whether mixed-reality simulation increases teacher candidate knowledge for both active participants and observers. For example, Ely et al. (2018) randomly assigned 22 teacher candidates to teach in the simulator or observe peers teaching in the simulator one time. The dependent measure included items related to candidate knowledge of Collaborative Strategic Reading practices. Both groups significantly increased their knowledge (d = 1.49); however, there was not a significant difference between the groups. Ely et al. (2018) discussed the role of observation of a candidate (peer) teaching in the simulator. However, few empirical studies have included peer observation in the simulator.

Mixed-Reality Simulation and SLP

Vince Garland et al. (2016) examined the fidelity of implementation of the SLP when candidates were provided individualized clinical coaching in a simulated classroom environment. The authors conducted a multiple probe across participants single case design with six graduate-level candidates enrolled in a special education course. First, candidates received instruction on the use of the SLP when working with a student with autism spectrum disorders. The SLP provides support to a learner using a hierarchy of prompts from the least amount of support (independent level; no prompts) to the most amount of support (controlling level; most intrusive prompt). The types of prompts in an SLP sequence may include verbal, gestural, visual, or physical prompts and may vary depending on the learner and the task. Because physical prompting is not possible within a simulator environment, the candidates implemented visual and gestural prompts with the student avatar. At the independent level, candidates delivered the stimulus without a prompt. For the intermediate prompt level, candidates delivered the stimulus with a visual prompt. Candidates provided visual prompts in the simulator by holding up two pictures (i.e., the picture of the correct answer and a picture of an incorrect answer). The final level was the controlling prompt level in which the candidates held up and gestured to the picture of the correct answer.

The SLP sequence in Vince Garland et al. consisted of five components: (a) establishing the learner’s attention, (b) delivering the stimulus, (c) providing a 5-second response interval, (d) responding to the learner’s attempts, and (e) collecting trial data. The participant’s response to the learner’s attempts was contingent on the learner’s response to the stimulus. If the response was correct, the participant provided specific praise consisting of a praise statement and specific feedback specifying what the learner did that was correct. If the learner’s response was incorrect, the participant delivered the next prompt in the prompting hierarchy. If the learner did not respond within the 5-second response interval, the participant proceeded to the next prompt in the prompting hierarchy.

Visual analysis of the candidates’ performance suggested a functional relationship between individualized clinical coaching within the mixed-reality simulator and increased participant fidelity of the SLP. Mastery criteria were defined as SLP implementation fidelity of 80% over three consecutive sessions. During the intervention, candidates met mastery criteria in three to five sessions, increasing their SLP implementation fidelity from 22% during baseline to 88%. Candidates underwent four baseline sessions, three to five individualized clinical coaching sessions, and two to three maintenance sessions in the simulator. In total, candidates were in the virtual environment at least 10 times. The high frequency of sessions is a weakness of this line of research in terms of efficiency and logistics of university instruction. In addition, candidates completed the simulation on their own with an interventionist to provide feedback. Vince Garland et al. (2016) served as a replication pilot for the current research.

Purpose

To support candidates, feasible, effective, and low-cost instructional methods are needed in special education (Institute of Education Sciences, 2020). Therefore, this study is a conceptual replication of the procedures outlined by Vince Garland et al. (2016) with the hope of creating a parsimonious intervention to instruct students on the SLP. In addition, Ely et al. (2018) was used as a model to add peer observation in the simulator, and the recommendations outlined by Larsen et al. (2020) were employed to improve candidates’ experiences. Thus, we investigated the following research questions:

Research Question 1: What are the effects of simulation and live practice on candidates’ implementation of the SLP?

Research Question 2: To what extent are there differences in candidates’ social validity responses regarding the use of simulation or live practice?

Method

In this study, the authors employed a randomized pretest-posttest control group design with two groups of candidates (Campbell & Stanley, 1963). The live practice group received traditional classroom instruction in the SLP. The intervention group practiced the SLP within the mixed-reality simulator. Pretest and posttest assessments measured accuracy of SLP implementation. Candidates completed social validity questions prior to and following the practice sessions. Procedural integrity of the practice sessions was measured.

Participants

A convenience sample of 30 undergraduate students in a special education cohort at a large southeastern university participated in the study. Participants were enrolled in the Characteristics of Individuals with Intellectual Disabilities and Autism course during the spring semester of their junior year. After obtaining institutional review board (IRB) approval, the third author of the study introduced the procedures for this exempt study to the participants. During this time, the course instructor left the room. Although the study was part of the course, the instructor did not participate in any of the study activities. All students agreed to participate in the study and received course points for completing study activities.

Participants were an average of 20.5 years old (SD = 0.81). Ninety-three percent of the participants were female, and 7% were male. Ninety percent of participants self-identified their race as White, 3% as African American or Black, 3% as Latinx, and 3% as Multiracial. Prior to completing the activities, five participants (17%) self-reported some training or experience in applied behavioral analysis practices. Participants who reported previous training or experience in applied behavior analysis were randomly stratified across the two conditions (Campbell & Stanley, 1963; Kendall, 2003).

Three doctoral students and a university faculty member who is also a doctoral-level, board-certified behavior analyst led the study. The intervention coaches each had at least 15 years of experience and training in applied behavior analysis. To aid the conceptual replication, the research team consulted with Dr. Vince Garland. Dr. Vince Garland shared materials and answered questions.

Intervention Settings

The study took place in two different locations on campus. Each of the settings is described in the following paragraphs.

Mixed-reality simulator condition

Mixed-reality simulations took place in the Digital Media Learning Lab. A high-definition flat-screen television was located on one wall of the room. Participants sat at a table approximately 6 feet in front of the television screen when interacting with the avatar during the simulations. Speakers allowed participants to hear the avatar’s responses. A webcam on the computer below the television and a microphone placed on the table in front of the participant allowed the interactor to see and hear the participant during the simulation sessions. The interactor (i.e., trained actor playing the part of the virtual student) participated remotely. Real-time communications occurred via Zoom web conferencing allowing the participant and the interactor to respond immediately to one another. Small group meetings before and after the mixed-reality sessions took place in separate meeting rooms nearby.

Three participants missed the two mixed-reality sessions due to personal circumstances. The mixed-reality simulator sessions were rescheduled. However due to COVID-19, campus buildings were closed. Therefore, the participants completed their missed mixed-reality simulator sessions and posttest assessments from their homes on Zoom.

Live practice condition

Participants in the live practice condition met in the Media Center. The Media Center classroom contained eight tables, each with four stools in the middle of the room and a row of computers on one side. An interactive whiteboard hung on one wall. The space facilitated classroom-based small group instruction and was similar to a traditional university classroom.

Procedure and Materials

The SLP sequence and prompting hierarchy

All participants’ pretests and posttests occurred in the Digital Media Lab. Participants implemented the SLP during pretesting, intervention, and posttesting with a stand-in “learner.” During pretest and posttest sessions, a member of the research team roleplayed as the learner. In the mixed-reality condition, the learner was the student avatar. In the live practice condition, peers acted as the learner in small groups. A visual of the study procedures is provided in Figure 1.

Figure 1.

Visual of the study procedures.

Pretest and posttest

To measure participants’ implementation of the SLP, the procedures outlined by Vince Garland et al. (2016) were followed. Prior to pretesting, participants received a copy of the story Alexander and the Terrible, Horrible, No Good, Very Bad Day by Judith Viorst. During the pretest and posttest sessions, participants met individually with two members of the research team. One researcher served as the learner, while the other team member recorded data. Participants were provided a list of comprehension questions, picture cards illustrating the answers to the questions, blank data collection form, and pen. As the participant implemented the SLP, the learner followed a script providing one correct response at the independent level, two correct responses at the intermediate prompt level (i.e., visual prompt), and two correct responses at the controlling prompt level (i.e., gestural prompt). The learner did not respond to the stimulus one time requiring the participant to move to the next prompt in the prompting hierarchy after 5 seconds. The participant had to gain the learner’s attention before delivering the stimulus twice during the session.

Researchers videotaped sessions to aid scoring, check fidelity of the learner, and compute interrater agreements. Each participant’s implementation of the SLP was measured with a 30-item rubric for a maximum of 49 points. The four components of the SLP sequence were each worth two points for a total of 38 points. Data collection was worth one point per item for a total of 11 points.

Instruction

Before the intervention sessions, all participants received instruction on the SLP as a group during a regular class session. Researchers created an instructional video describing prompting and the SLP. The 30-minute video included 61 PowerPoint slides and integrated Mayer’s (2008, 2009) Instructional Design Principles (e.g., coherence principle, signaling principle, redundancy principle). Instructional content was adapted from the “Prompting” module produced by the National Professional Development Center on Autism Spectrum Disorder (Sam & AFIRM Team, 2015). One week after receiving instruction, participants completed their respective interventions. Both conditions completed two intervention sessions.

Intervention

For both conditions, the two intervention sessions lasted approximately 30 to 40 minutes and consisted of three parts (i.e., preconference, practice, debrief). Sessions began by the researcher describing the SLP (e.g., prompt levels, prompting technique) and providing a handout explaining the SLP procedure. The first session also included a 2-minute video of the researcher modeling the SLP sequence with a 6-year old student. Next, participants were divided into groups of three to practice implementing the SLP sequence and record response data. Each group received a set of SLP materials, which included the list of comprehension questions, picture cards illustrating the answers to the questions, and three blank data collection forms. A debriefing session followed each intervention session. During the debrief, the researcher prompted the participants to reflect on their experience, identifying a strength and area of improvement.

Live practice sessions

The live practice session modeled typical in-class or group practice. One researcher circulated around the groups providing instructional support and coaching (similar to an instructor during in-class practice). During the live session’s group practice, the researcher provided behavior-specific feedback (e.g., corrections, praise). A group debrief was held at the end of each practice session.

Simulation sessions

To address the concerns outlined by Larsen et al. (2020), the participants in the simulator group met the student avatar before the simulation intervention sessions. The purpose of the session was to familiarize the participants with the mixed-reality environment and provide a brief interaction with the avatar, Nate. Nate is a middle school student diagnosed with high-functioning autism. He has an interest in math and science, especially the solar system. During the session, participants received a description of Nate and his heart map (i.e., a visual representation of things he loves and cares about). Participants entered the simulator, observed the researcher model an interaction (e.g., greeting, pleasantries), and took turns speaking with Nate. The session lasted approximately 30 minutes and did not include instruction or coaching. During this session, participants did not receive any SLP content or discuss implementing the SLP.

Prior to the intervention, the interactor received a written scenario with the learner objectives and an avatar response guide. Then, the researchers and interactor met in the virtual setting for a 1-hour training session to review the SLP sequence and intervention procedures. This ensured procedural fidelity on the part of the interactor. During the intervention, the interactor randomly applied the same response ratio used in the pretest assessment (i.e., one correct response at the independent level, two correct responses at the intermediate prompt level, and two correct responses at the controlling prompt level) with each participant. Participants were also required to gain Nate’s attention and use a 5-second response interval during the simulator sessions. Nate immediately reestablished joint attention when prompted by the participant. He displayed mild behaviors (e.g., mild rocking, avoidance of eye contact, distractibility, low mumbling) during the first session and intensified those behaviors during the second session.

Similar to the live practice condition, simulator participants met in groups of three. Placing students in groups of three allowed them to watch two peers work with the avatar (Ely et al., 2018). Following the explanation of the SLP, the researcher described the simulator scenario and procedures to follow in the event of technical difficulties. After moving to the virtual classroom, the participants observed the researcher model the SLP strategy with the avatar, Nate. Each participant practiced the SLP strategy and collected data on Nate’s responses, while the other participants observed. Similar to previous simulator studies, the researcher or participant could pause the simulation at any time to ask questions, provide feedback, or redo a portion of instruction. Researchers collected participant performance data using a structured coaching form. The structured coaching form contained the critical steps of the SLP (e.g., establish attention, independent prompt, visual prompt, response interval). Beside each step contained a space for the coach to record notes. At the bottom of the form, it prompted the coach to note participant’s goals for the next session. Please contact the first author for a copy of the structured coaching form. An additional researcher provided technical support (e.g., troubleshooting volume issues).

In addition to the general debriefing discussion, as detailed in Larsen et al. (2020), simulator participants completed the After-Action Review Journal. The After-Action Review Journal was a short three-question form used to collect participant reflections of what went well, what could be improved, and actions to improve.

Procedural fidelity

Individual procedural fidelity forms were created for the live practice sessions and the mixed-reality sessions. Each form contained a task analysis of the tasks covered during the respective sessions. Specifically, these were the steps implemented by the researchers. The simulator procedural form included eight steps (e.g., investigator orients participant to the lab, reviews the SLP, gives instructions for beginning and ending simulations, explains protocols should technical difficulties arise, provides coaching during the simulation on the SLP). The live session procedural fidelity form included four steps (e.g., investigator orients participant to the practice session, reviews the SLP, provides coaching during the practice on the SLP). An independent researcher completed a procedural fidelity checklist for 100% of the live practice and 75% of the mixed-reality intervention sessions. Prior to recording fidelity, the researcher met one-on-one with the first author to review the content. Following training, the independent researcher attended the sessions in person and completed the procedural fidelity forms. For a copy of the forms, please contact the first author. To calculate fidelity, the number of steps implemented correctly was divided by the total number of procedural steps and multiplied the quotient by 100 (Ledford & Gast, 2014). Fidelity ratings for the live sessions were 100% and the mixed-reality sessions were 95% (range: 89%–100%) of steps implemented.

In addition to the intervention sessions, fidelity of the learner’s (researcher) responses were recorded for 100% of the testing sessions. The pretest sessions averaged 99.6% accuracy (range: 97%–100%), and the posttest sessions averaged 100% accuracy.

Interobserver agreement (IOA)

Researchers scored each pretest and posttest video using a rubric with accompanying scoring guidelines. The scoring guidelines provided operational definitions of correct and incorrect responses. A second research assistant scored 50% of pretest sessions and 33% of posttest sessions. Researchers calculated point-by-point reliability for each target skill by dividing the number of agreements by the number of agreements plus disagreements times 100 (Kazdin, 2011). The overall IOA of the pretest was 96% (range: 90%–100%) and the posttest was 97% (range: 93%–100%).

Social validity

Participants completed an online social validity survey before the first session and after the last session. The survey included five questions related to the participant’s perceived nervousness and benefit of using the intervention assigned to them. This instrument is based on past research (Larsen et al., 2020). The measure contained a Likert-type scale ranging from one to five (1 = strongly disagree to 5 = strongly agree).

Data Analysis

Scores for pretest and posttest assessments were entered into Microsoft Excel 365 and converted to an SPSS (v.26.0) data set. Descriptive statistics (M and SD) were calculated for each measure. To answer the first research question about the effect of the simulator and live practice on participant implementation of SLP, paired t test analyses were conducted to examine changes between the pretest and posttest scores for each participant. Then, a one-way analysis of variance (ANOVA) was run to evaluate differences between the simulator and live practice groups at posttest. Effect sizes (Hedges’ g) were calculated to evaluate the magnitude of any detected group differences. To answer the second research question regarding the participants’ perceptions, a series of paired and independent samples t tests were conducted on social validity survey responses.

Results

Research Question 1: Implementation of the SLP

After randomization, there was not a significant difference in SLP implementation at pretest between the simulator group (M = 9.6, SD = 5.6) and live practice group (M = 9.5, SD = 4.7); F(1, 28) = 0.001, p = .972, thus indicating randomization was successful. From pretest to posttest, the participants’ average score increased from 9.6 (SD = 5.1) to 38.8 (SD = 7.0) on the SLP rubric. The increase was significant, t(29) = 17.02, p < .001. At posttest, the simulator group on average outscored the live practice group (simulator M = 41.6, SD = 5.4; live practice M = 36.0, SD = 7.4).

A one-way ANOVA (see Table 1) was conducted to determine whether the participants’ posttest SLP scores were significantly different between the two groups. Significant between-group differences and a large effect size were detected (F = 5.59, p < .025, g = 0.84), indicating the simulator had a large, significant impact on participant learning.

Table 1.

Performance on SLP During Pretest and Posttest by Condition.

Measure	Simulator		Live practice
Measure	M	SD	M	SD	F	p value
Pretest	9.6	5.6	9.5	4.7	0.001	.972
Posttest	41.6	5.4	36.0	7.4	5.59	.025

Note. SLP = system of least prompts.

Research Question 2: Social Validity

Participants completed a social validity questionnaire to rate their experiences as a participant in either the simulator or the live practice group (see Table 2).

Table 2.

Social Validity Survey Results.

	Within groups								Between groups (posttest)
	Simulator				Live practice
Statement	PreM (SD)	PostM (SD)	t	p value	PreM (SD)	PostM (SD)	t	p value	SimM (SD)	LiveM (SD)	F	p value
1. I feel nervous or anxious in the live practice/simulator.	3.8 (0.8)	3.8 (1.3)	0.00	1.0	3.6 (1.2)	2.0 (1.0)	4.71	.001	3.5 (1.3)	2.2 (1.2)	8.46	.007
2. I think that the live practice/simulator will be beneficial for me.	4.2 (0.8)	4.8 (0.4)	−2.71	.024	3.7 (0.8)	3.7 (0.9)	0.00	1.000	4.8 (0.4)	3.9 (0.9)	12.94	.001
3. I am worried about how I will perform in the live practice/simulator.	4.4 (0.5)	3.4 (0.8)	3.35	.008	3.7 (0.9)	2.3 (1.4)	3.14	.009	3.4 (0.9)	2.4 (1.5)	4.86	.036
4. The live practice/simulator will be a useful learning tool.	4.3 (0.8)	4.7 (0.5)	−1.81	.104	3.8 (0.9)	3.6 (1.2)	0.48	.638	4.7 (0.5)	3.9 (1.2)	6.40	.017
5. The live practice/simulator is relevant to my program of study.	4.2 (0.6)	4.8 (0.4)	−2.71	.024	4.1 (0.7)	4.2 (0.8)	−0.29	.777	4.8 (0.4)	4.3 (0.8)	3.90	.058

Note. Scale: 1 = strongly disagree, 5 = strongly agree. Pre = pretest; post = posttest; sim = simulator; live = live practice.

Paired samples t tests were conducted to compare the pretest and posttest social validity survey results within the simulator and live practice groups. The participant responses to “I feel nervous or anxious in the live practice/simulator” indicated that the simulator group generally felt more nervous before teaching than the live practice group. For the simulator group, results of the paired samples t test reveal no significant difference from pretest to posttest on this survey item, suggesting simulator participants maintained their general level of nervousness before (M = 3.8, SD = 0.8) and after using the simulator (M = 3.8, SD = 1.3); t(9) = 0.0, p = 1.0. In contrast, the live practice group significantly decreased their level of nervousness from pretest (M = 3.6, SD = 1.2) to posttest (M = 2.0, SD = 1.0); t(11) = 4.71, p = .001. On the item, “I am worried about how I will perform in the live practice/simulator,” both groups agreed to feeling worried about their teaching performance before practicing. However, the live practice group reported feeling less worried on average than the simulator group. Both groups significantly decreased their worry about how they would perform from pretest to posttest. The simulator group mean response—pretest: M = 4.4, SD = 0.5; posttest: M = 3.4, SD = 0.8; t(9) = 3.35, p = .008—and the live practice group mean response—pretest: M = 3.7, SD = 0.9; posttest: M = 2.3, SD = 1.4; t(11) = 3.14, p = .009—both decreased by a full point from pretest to posttest. These differences were statistically significant, indicating participants in both groups worried less about their performance over time.

For the item, “I think that the live practice/simulator will be beneficial for me,” the simulator group significantly increased their perceptions from pretest (M = 4.2, SD = 0.8) to posttest (M = 4.8, SD = 0.4); t(9) = −2.71, p = .024, indicating that they felt the simulator was more beneficial after participating in the simulator sessions. In contrast, the live practice group did not significantly change their perceptions between pretest (M = 3.7, SD = 0.8) and posttest (M = 3.7, SD = 0.9); t(11) = 0.0, p = 1.0. On the item, “The live practice/simulator will be a useful learning tool,” neither group significantly changed their perceptions of the usefulness of the simulator/live practice as a learning tool, but the mean scores of the simulator group—pretest: M = 4.3, SD = 0.8; posttest: M = 4.7, SD = 0.5; t(9) = −1.81, p = .104—were higher than those of the live practice group, pretest: M = 3.8, SD = 0.9; posttest: M = 3.6, SD = 1.2; t(11) = 0.48, p = .638. This suggests that while both methods of practicing the SLP were considered to be a useful learning tool by the participants, the simulator group viewed the simulator as a stronger learning method. For the final question, “The live practice/simulator is relevant to my program of study,” the simulator group significantly increased their perceptions of the relevance of the simulator to their program of study from pretest (M = 4.2, SD = 0.6) to posttest (M = 4.8, SD = 0.4); t(9) = −2.71, p = .024, whereas there was not a significant change in the live practice group from pretest (M = 4.1, SD = 0.7) to posttest (M = 4.2, SD = 0.8); t(11) = −0.29, p = .777.

A one-way ANOVA compared the responses with the posttest social validity questionnaire between the groups and indicated that the groups felt differently about their experiences. The simulator group felt significantly more nervous, F (1, 28) = 8.50, p = .007, and more worried about how they would perform, F(1, 28) = 4.86, p = .036, than the live practice group. In addition, the simulator group rated the intervention (simulator) significantly more beneficial than when compared with the responses of the live practice group regarding the benefit of the live practice intervention, F(1, 28) = 12.9, p = .001. Furthermore, the simulator group rated the method of learning (i.e., simulator) significantly more useful as a learning tool than the live practice group F(1, 28) = 6.40, p = .017. On the final question (i.e., The live practice/simulator is relevant to my program of study.), there was not a significant difference between the simulator and live practice groups, F(1, 28) = 3.90, p = .058. Both groups considered their intervention method relevant to their program of study.

Discussion

In this study, the authors sought to build upon the previous work in the area of mixed-reality simulations in special education. We replicated the SLP procedures outlined by Vince Garland et al. (2016). We extended this inquiry by using Ely et al. (2018) as a model with peer observation in the simulator and employed the recommendations outlined by Larsen et al. (2020) to improve the participants’ experiences. In addition, we collected social validity and procedural fidelity data.

Hedges’ g results from this study indicate that the simulator produced large gains in knowledge and practice of SLP compared with live practice, which reflects previous research finding that use of a mixed-reality simulator improved participant knowledge and use of instructional strategies (Driver et al., 2018; Vince Garland et al., 2016) and demonstrates the promise of the simulator as an instructional tool for providing targeted practice opportunities. We selected a specific instructional strategy (SLP) that can be utilized across content domains and grade levels to maximize applicability for participants. In that vein, teacher candidates who are struggling to master a particular concept could feasibly work on a specific skill in the simulator to gain more practice and could potentially do so with greater results than live practice in the traditional university classroom. Additional practice coupled with feedback and individualized coaching within the simulator environment ensure that the teacher candidate is making adjustments and improving in their teaching practice. In this way, the simulator could function as a targeted “intervention” of sorts for candidates who need help or additional work in a particular area of their teaching.

In this study, the simulator experience offered consistent skill practice for participants when compared with other live practice models. The mixed-reality scenario is designed to be consistent across users, allowing participants to iteratively hone and refine their skills without unintended influences or disruptions. In this study, participants in the simulator condition also observed their peers working with the avatar. Peer observation was included as an intervention component based on the recommendations of previous research (Ely et al., 2018; Larsen et al., 2020). By ascribing to a set of prenegotiated criteria and expectations, the simulator can offer more streamlined and homogeneous practice experiences across users and sessions that are not always possible in live, peer practice scenarios.

Social validity survey results indicated that while both groups viewed their condition favorably, the simulator group participants rated their experience higher than the live practice group. This reflects previous work around the relevance and utility of mixed-reality simulators (Larsen et al., 2020). While participants initially felt anxious about teaching in both conditions, that anxiety seemed to decrease over time. Interestingly, at posttest, the live practice group rated their own nervousness and anxiety lower than the simulator group. While any conclusions about these social validity results would be speculation, perhaps the novelty of teaching to an avatar in the simulation induces more anxiety compared with teaching a live person, even when that person is a peer or an adult. More work is needed on how practicum (field) and practice environments affect candidates’ feelings and self-efficacy during practice. Although this study was run primarily through our university media lab, for three participants, we did need to move the simulation online (via Zoom) due to COVID-19. While conducting the study online was not our original intent, we found the technology to be flexible. Therefore, we are encouraged by the portability of the tool.

Educational Implications

The mixed-reality simulator offers an opportunity to supplement traditional field-based school placements for teacher candidates in many unique situations. Schools nationwide struggle to recruit and retain special education teachers (Brown et al., 2015; Cross, 2017; Mason-Williams, 2015; Sutcher et al., 2016), which affects not only the education received by K–12 students with disabilities but also the availability of student teaching placements for preservice candidates in teacher preparation programs. The student teaching experience is a critical part of any teacher preparation program as it provides opportunities for hands-on experience and relationships with mentor teachers (Brown et al., 2015) as well as opportunities for preservice teachers to receive direct feedback on their teaching (Leko et al., 2015). Therefore, the simulator can function as a supplement or, in extreme cases, a replacement for traditional field-based face-to-face teaching placements when such placements are not readily available. In addition, the simulator environment can be structured to provide experience with specific types of low-incidence disabilities or student characteristics that may not be as readily available in real-world classes in a geographic location. For example, preservice teachers in more remote locations or in districts with more demographically homogeneous student populations could use the simulator to practice teaching with a student avatar who is an English learner, a student avatar with autism, or a group of virtual students of varying demographic backgrounds. Finally, even in a location where special education placements are plentiful, there is no guarantee that such placements are engaging in evidence-based teaching practices. Schools who do manage to fill special education vacancies often end up hiring teachers who are less prepared, have less hands-on training, and are more likely to have emergency or alternative certification, particularly in high-needs schools (e.g., high poverty, high minority enrollment, rural location; Mason-Williams, 2015). The simulator could potentially provide teacher candidates opportunities to observe high-quality teaching models and gain additional experience using evidence-based practices even in the absence of such elements in their field placements.

The results of this study also provide insight into how much time students need in the simulator to experience its effects. Previous studies using the mixed-reality teaching simulator with candidates have suggested that future studies investigate the length and frequency of experiences needed for participants to produce desired results (Dieker et al., 2017; Driver et al., 2018; Ely et al., 2018). In this study, the authors revealed significant posttest differences between the simulator and live practice groups with participants only participating in the simulator experience twice with about 6 minutes of individual interaction time in each session. This is less time and fewer sessions than the original replicated study (Vince Garland et al., 2016), suggesting that the simulator can be a valuable task for preservice teachers over live practice techniques even when offered in limited frequencies and durations.

The mixed-reality teaching simulator also offers opportunities for preservice teachers to practice their skills when schools are closed. Physical school closings due to inclement weather, school breaks, and unpredictable public health crises like COVID-19 affect preservice teachers’ access to classrooms where they practice their skills. By continuing to offer the simulator experience through the video conferencing program Zoom during COVID-19 closures, preservice teachers in the study had the opportunity to continue practicing the SLP despite their circumstances. This suggests that the mixed-reality teaching simulator has the potential to offer additional opportunities for teachers’ skill development beyond the space confines of physical, formal learning environments.

Limitations and Future Directions

Limitations should be considered when interpreting the results. Given the goal of studying teacher candidates’ implementation of the SLP, there was a challenge associated with implementing the SLP in the simulator. Specifically, participants had to hold up visuals. For example, visual prompts counted only if the participant held up two cards. Gestural prompts only counted if students made a gesture to the visual (e.g., instead of holding the card with two hands). The procedural difference allowed researchers to discriminate between the two levels of prompting. In contrast, teachers have the opportunity to display visual prompts on a table, provide a greater number of pictures at one time, or use physical prompts as the controlling prompt when implementing the SLP with live students.

Although all participants improved their ability to implement the SLP, some participants may have benefited from additional practice sessions. Future research could employ multitiered systems of support logic to differentiate dosage based on response to intervention (Simonsen et al., 2017). This would have provided additional opportunities to practice in the simulator for students who have not yet mastered the skill. Another limitation of this study is that three simulator participants completed the sessions at home on Zoom. Future research should compare the effects of simulation sessions across different locations (e.g., participant home vs. university lab setting). In addition, questions remain about the impact of simulation on the (a) generalization/transfer to the classroom and (b) maintenance of the skill after instruction and practice. Ideally, researchers will address these limitations in future studies.

Summary

In this study, the authors intended to evaluate mixed-reality simulation within a preservice teacher course. We focused on extending the line of inquiry related to SLP within the simulator while addressing the limitations of previous studies. Participants had the opportunity to observe other classmates interact in the simulator. In this study, the authors suggest that mixed-reality simulation is an efficient and effective approach to increasing teacher candidates’ use of SLP practices and may be one way to augment in class activities. This is especially important during university closures and online learning. However, given that all participants received the same amount of instruction (two simulator sessions), research is needed to refine simulator dosage and decision rules for candidates. In addition, further research should explore whether the effects generalize to the classroom when working with students.

Footnotes

Acknowledgements

We would like to acknowledge Marianne Beck and Margaret-Davis Huggins for providing technical assistance. We would also like to acknowledge Drs. Kate Zimmer and Melissa Driver of the Kennesaw State University AVATAR Lab for providing support with this project. In addition, we would like to thank Dr. Krista Vince Garland for providing feedback and resources.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the Clemson University Creative Inquiry Program.

ORCID iDs

Sharon M. Walters

Shanna E. Hirsch

Georgia McKown

Abigail A. Allen

Author Biographies

Sharon M. Walters is a special education doctoral candidate at Clemson University. Her research interests include language interventions for students diagnosed with emotional behavioral disorders and using mixed-reality simulation in preservice teacher preparation.

Shanna E. Hirsch is an Assistant Professor of Special Education at Clemson University. Her current research focuses on implementing positive behavior interventions and supports; supporting novice teachers with classroom management; and implementing evidence-based practices to support students with and at risk for emotional and behavioral disorders.

Georgia McKown is a doctoral student in the Learning Sciences program at Clemson University. Her research focuses on academic development for military connected students in the United States and opportunities to improve teacher education to meet the needs of diverse student populations.

Alex Carlson is a doctoral student in special education at Clemson University. His current research interests include supporting novice teachers, behavioral interventions in the classroom, and legal and policy issues in special education.

Abigail A. Allen is an Assistant Professor of Special Education at Clemson University. Her research interests include assessment and interventions in early writing, learning disabilities, curriculum-based measurement, and data-based instruction.

References

American Council on Education. (2020). Memo to Congress: Supporting higher education during the pandemic. https://www.acenet.edu/Documents/Higher-Education-Supplemental-Summary-Memo-031920.pdf

Bradley

E. G.

Kendall

(2014). A review of computer simulations in teacher education. Journal of Educational Technology Systems, 43, 3–12. https://doi.org/10.2190/ET.43.1.b

Brown

A. L.

Lee

Collins

(2015). Does student teaching matter? Investigating pre-service teachers’ sense of efficacy and preparedness. Teaching Education, 26(1), 77–93.

Brownell

M. T.

Bishop

A. G.

Gersten

Klingner

J. K.

Penfield

R. D.

Dimino

Haager

Menon

Sindelar

P. T.

(2009). The role of domain expertise in beginning special education teacher quality. Exceptional Children, 75, 391–411. https://doi.org/10.1177/001440290907500401

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Rand McNally.

Cook

B. G.

Haggerty

Smith

(2019). Leadership and instruction: Evidence-based practices in special education. In Crockett

Billingsley

Boscardin

M. L.

(Eds.), Handbook of leadership and administration for special education (pp. 18–38). Routledge.

Cross

(2017). Teacher shortage areas: Nationwide listing 1990-1991 through 2017-2018. U.S. Department of Education Office of Postsecondary Education. https://www2.ed.gov/about/offices/list/ope/pol/bteachershortageareasreport201718.pdf

Darling-Hammond

(2010). Teacher education and the American future. Journal of Teacher Education, 61, 35–47. https://doi.org/10.1177/0022487109348024

Dawson

M. R.

Lignugaris /Kraft

. (2017). Meaningful practice: Generalizing foundation teaching skills from TLE TeachLivE™ to the classroom. Teacher Education and Special Education, 40(1), 26–50.

10.

Dieker

L. A.

Hughes

C. E.

Hynes

M. C.

Straub

(2017). Using simulated virtual environments to improve teacher performance. School-university Partnerships, 10(3), 62–81.

11.

Dieker

L. A.

Hynes

M. C.

Hughes

C. E.

Hardin

Becht

(2015). TLE TeachLivE™: Using technology to provide quality professional development in rural schools. Rural Special Education Quarterly, 34(3), 11–16. https://doi.org/10.1177/875687051503400303

12.

Dieker

L. A.

Hynes

M. C.

Hughes

C. E.

Smith

(2008). Implications of mixed reality and simulation technologies on special education and teacher preparation. Focus on Exceptional Children, 40(6), 1–20.

13.

Dieker

L. A.

Rodriguez

J. A.

Lignugaris /Kraft

Hynes

M. C.

Hughes

C. E.

(2014). The potential of simulated environments in teacher education: Current and future possibilities. Teacher Education and Special Education, 37(1), 21–33. https://doi.org/10.1177/0888406413512683

14.

Dieker

L. A.

Straub

Hughes

Hynes

M. C.

Hardin

S. E.

(2014). Learning from virtual students. Educational Leadership, 71(8), 54–58.

15.

Driver

Zimmer

Murphy

(2018). Using mixed reality simulations to prepare preservice special educators for collaboration in inclusive settings. Journal of Technology & Teacher Education, 26(1), 57–77.

16.

Ely

Alves

K. D.

Dolenc

N. R.

Sebolt

Walton

E. A.

(2018). Classroom simulation to prepare teachers to use evidence-based comprehension practices. Journal of Digital Learning in Teacher Education, 34, 71–87. https://doi.org/10.1080/21532974.2017.1399487

17.

Hixon

H. J.

(2009). Technology’s role in field experiences for preservice teacher training. Journal of Educational Technology & Society, 12(4), 294–304.

18.

Institute of Education Sciences. (2020). Cost analysis: A toolkit (IES 2020-001). U.S. Department of Education. https://ies.ed.gov/pubsearch/

19.

Kazdin

A. E.

(2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). Oxford University Press

20.

Kendall

(2003). Designing a research project: Randomized controlled trials and their principles. Emergency Medicine Journal, 20(2), 164–168. https://doi.org/10.1136/emj.20.2.164

21.

Larsen

Hirsch

S. E.

McGraw

Bradshaw

C. P.

(2020). Examining the feasibility and acceptability of using mixed-reality simulator to support pre-service teacher training. Journal of Special Education Technology, 35, 63–75. https://doi.org/101177/0162643419836415

22.

Ledford

J. R.

Gast

D. L.

(2014). Measuring procedural fidelity in behavioural research. Neuropsychological Rehabilitation, 24(3–4), 332–348. https://doi.org/10.1080/09602011.2013.861352

23.

Leko

M. M.

Brownell

M. T.

Sindelar

P. T.

Kiely

M. T.

(2015). Envisioning the Future of Special Education Personnel Preparation in a Standards-Based. Era. Exceptional Children, 82(1), 25–43. https://doi.org/10.1177/0014402915598782

24.

Ludlow

B. L.

(2015). Virtual reality: Emerging applications and future directions. Rural Special Education Quarterly, 34(3), 3–10.

25.

Mason-Williams

(2015). Unequal opportunities: A profile of the distribution of special education teachers. Exceptional Children, 81(2), 247–262.

26.

Mayer

R. E.

(2008). Applying the science of learning: Evidence-based principles for design of multimedia instruction. American Psychologist, 63(8), 760–769. https://doi.org/10.1037/0003-066X.63.8.760

27.

Mayer

R. E.

(2009). Multimedia learning (2nd ed.). Cambridge University Press.

28.

McLeskey

Barringer

M. D.

Billingsley

Brownell

Jackson

Ziegler

(2017, January). High-leverage practices in special education. Council for Exceptional Children & CEEDAR Center. http://ceedar.education.ufl.edu/portfolio/ccsc-2017-high-leverage-practice

29.

Pas

Waasdorp

Bradshaw

C. P.

(2019). Coaching teachers to detect, prevent, and respond to bullying using mixed-reality simulation: An efficacy study in middle schools. International Journal of Bullying Prevention, 1(1), 58–69. https://doi.org/10.1007/s42380-018-0003-0

30.

Peeples

Hirsch

S. E.

Gardner

S. J.

Kelley

R. G.

Sherrow

B. L.

McKenzie

J. M.

Randall

Kennedy

M. J.

(2019). Using multimedia instruction and performance feedback to improve pre-service teachers’ vocabulary instruction. Teacher Education and Special Education, 42, 227–245. https://doi.org/10.1177/0888406418801913

31.

Rock

M. L.

Spooner

Nagro

Vasquez

Dunn

Leko

Luckner

Bausch

Donehower

Jones

J. L.

(2016). 21st Century Change Drivers: Considerations for Constructing Transformative Models of Special Education Teacher Development. Teacher Education and Special Education, 39(2), 98–120. https://doi.org/10.1177/0888406416640634

32.

Sam

, & AFIRM Team. (2015). Prompting. National Professional Development Center on Autism Spectrum Disorder, FPG Child Development Center, University of North Carolina. http://afirm.fpg.unc.edu/prompting

33.

Sayeski

K. L.

Hamilton-Jones

Cutler

Earle

G. A.

Husney

(2017). The role of practice and feedback for developing teacher candidates’ opportunities to respond expertise. Teacher Education and Special Education, 42(1), 18–35. https://doi.org/10.1177/0888406417735876

34.

Simonsen

Freeman

Dooley

Maddock

Kern

Myers

(2017). Effects of targeted professional development on teachers’ specific praise rates. Journal of Positive Behavior Interventions, 19(1), 37–47. https://doi.org/10.1177/1098300716637192

35.

Sutcher

Darling-Hammond

Carver-Thomas

(2016). A coming crisis in teaching? Teacher supply, demand, and shortages in the U.S. Learning Policy Institute.

36.

Vince Garland

K. M.

Holden

Garland

D. P

. (2016). Individualized clinical coaching in the TLE TeachLivE Lab: Enhancing fidelity of implementation of system of least prompts among novice teachers of students with autism. Teacher Education and Special Education, 39(1), 47–59. https://doi.org/10.1177/0888406415600769