Change of Teacher Beliefs,But Not Practices,Following Integration of Immersive Virtual Environment in the Classroom

Abstract

Immersive virtual environments (IVEs) model scientific inquiry practices and can provide rich learning experiences for students. However, the teacher is an essential component of how the students engage with the technology, as they embed the IVE into everyday teaching and learning. Ideally, classroom pedagogies would reflect the scientific practices modeled in the IVE to the best extent possible. In this case study, we explore how one teacher enacted authentic scientific practices after 3 years of IVE implementations plus participation in the corresponding professional development (PD) experience. Although the teacher believed that the inquiry practices modeled in the IVE and the PD were an important part of his teaching, these beliefs were not transferred into practice. Our findings suggest that PD should include opportunities for teachers to reflect on their own practice integrating technology and consider external factors contributing to pedagogical change beyond PD.

Keywords

case study immersive virtual environments mixed methods professional development

Introduction

Emerging technologies, such as immersive virtual environments (IVEs), hold great potential for scaffolding scientific inquiry skills in formal learning environments (Chan & Yang, 2018; Dede, Grotzer, Kamarainen, & Metcalf, 2017; Ketelhut, 2006; National Academies of Sciences and Medicine, 2018). However, these technologies are rarely meant to be standalone or “teacher proofed.” The pedagogy of the teacher is a critical component of how students engage in learning with the IVE (Mishra & Koehler, 2006). A traditional teacher could present an IVE as if there is only one correct method to traverse through the environment. Another teacher could encourage students to explore the many different problem-solving methods and strategies within the virtual environment, acknowledging the merit and challenges of each path. We believe the latter strategy is better aligned with the scientific inquiry-driven pedagogy; many IVEs were designed to facilitate (Barab, Jackson, & Piekarsky, 2006; Chan & Yang, 2018; Dede et al., 2017; Grotzer et al., 2017; Ketelhut, 2007). However, many teachers continued to be challenged to integrate virtual environments into K-12 classrooms in a sustainable and natural way (Ketelhut & Schifter, 2011; U.S. Department of Education, 2017). The aim of our study is to explore factors that impact pedagogical decision-making as teachers integrate IVEs into classroom teaching and learning in order to promote inquiry-based classrooms.

Previous studies have suggested that implementing IVEs provides opportunities to include inquiry into classroom practices (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; Chan & Yang, 2018; Dede et al., 2017; Grotzer et al., 2017; Ketelhut, 2007; Ketelhut, Nelson, Clarke, & Dede, 2010). However, this requires that the teacher be able to design classroom activities that help students make sense of their IVE inquiry experiences. Effectively integrating IVEs into classroom teaching and learning requires the development of teacher knowledge and beliefs to allow the development of complementary experiences to those inquiry-based ones in the virtual world (Mishra & Koehler, 2006). Since early days of the development of IVEs for classroom learning, professional development (PD) purveyors have designed their PD with the intention that teacher practice and beliefs will change, leading to an increased comfort with using these tools and even an use of scientific inquiry in the classroom (Barab et al., 2006; Ketelhut & Schifter, 2011). While the field is beginning to identify aspects of effective teacher PD for the integration of learning technologies (Fishman, Dede, & Means, 2016), there is little research into both teacher practices and beliefs as they integrate IVEs into their teaching.

Through our National Science Foundation-funded project, SAVE Science we explored IVE-based modules designed for assessment of science content and practices in middle school classrooms (Ketelhut, Nelson, Schifter, & Kim, 2013). The study focused on understanding both student and teacher knowledge and scientific practices. We have investigated across time how implementing IVEs, alongside ongoing PD sessions, influences teachers’ beliefs and practices of carrying out scientific inquiry in their classrooms. Through participation in our IVE research project, teachers implemented one or more IVEs with middle school students across several years while being supported with training and ongoing PD sessions. Through this work, we aimed to support teachers in building knowledge and self-efficacy for using technologies such as IVEs while empowering them to integrate IVEs into their own practices and for their own purposes. One goal of our PD and of the larger project was to impact the way teachers thought about and enacted scientific practices in their classrooms.

This study examines the complex development of such teacher change applying Clarke and Hollingsworth’s (2002) interconnected model of professional growth (IMPG) to an illustrative case study answering the research question, How did one teacher’s enactment of science inquiry in his classroom change, if at all, following his participation in an IVE-based curriculum program?

Literature Review

IVEs to Support Science Inquiry

Science inquiry has been considered an integral part of science teaching and learning for the past two centuries (DeBoer, 1991; National Research Council, 1996). Recently, the Next Generation Science Standards specified eight scientific inquiry-related practices such as “Asking Questions” “Planning Investigations,” and “Collecting and Analyzing Data” (National Research Council, 2013) that engage students in authentic science. Despite long-standing nationwide goals to incorporate scientific practices in the classroom (National Research Council, 1996), obstacles for implementation persist, including lack of classroom time due to a heavy emphasis on teaching toward high-stakes assessment, large class size, resources, and teacher knowledge (Ketelhut & Tutwiler, 2017; National Academies of Sciences and Medicine, 2018).

Emerging technologies, such as IVEs, provide an opportunity for students to meaningfully engage in scientific inquiry in formal learning environments in ways that were not always possible before, due to the obstacles mentioned earlier (National Academies of Sciences and Medicine, 2018). An IVE is an IVE that “is based on a certain pedagogical model, incorporates or implies one or more didactic objectives, provides users with experiences they would otherwise not be able to experience in the physical world and redounds specific learning outcomes” (Mikropoulos & Natsis, 2011, p. 770). IVEs can be designed to present an authentic science question for students to investigate within the virtual world. In these scenarios, students can collect various sources of data as they traverse the virtual environment. For example, they can interact with objects within the virtual environment using a set of virtual tools, taking and recording relevant measurements. Students can solve real-world problems by graphing and analyzing different data sources using affordances within the IVE. Designers of IVEs aspire that these innovations will influence science teachers to adjust their pedagogy to include more inquiry-based learning (Barab et al., 2006; Dede et al., 2017). However, the literature is thin on whether and how this happens.

Examples of IVEs that have been implemented in K-12 classrooms are River City, Quest Atlantis, and EcoXPT (Barab et al., 2005; Dede et al., 2017; Ketelhut et al., 2010). These IVEs were designed to facilitate science practices in formal learning contexts by situating inquiry-based science activities and science content within real-world contexts. In these IVEs, students take on a role as a character (avatar) within a virtual world. The avatar is presented with a problem that can be solved by collecting and analyzing data within the virtual world. To conclude their experience in the virtual world, students must synthesize their findings by writing a letter explaining a solution to the problem and supporting their explanations with data they have collected within the virtual world and analyses of that data. Research has indicated that students engage in inquiry-based learning within these virtual environments (Barab et al., 2005; Chan & Yang, 2018; Dede et al., 2017; Grotzer et al., 2017; Ketelhut et al., 2010) and even suggests moderate learning gains for students over traditional pedagogical approaches (Clark, Tanner-Smith, & Killingsworth, 2016; Fletcher & Tobias, 2011; Wouters, Van Nimwegen, Van Oostendorp, & Van Der Spek, 2013; Wouters & Van Oostendorp, 2017). Furthermore, engagement in scientific practices through IVEs has illustrated potential to increase students’ scientific self-efficacy (Ketelhut, 2007).

More recently, educational researchers have explored the potential of IVEs and similar learning technologies as assessments for learning (Jiao & Lissitz, 2018). As such, evaluative strategies can be embedded in learning tasks in ways that are less time-consuming and disruptive to classroom instruction (U.S. Department of Education, 2017). Examples of technologies that have been designed to assess students’ science learning are as follows: The Wells Task (National Assessment of Educational Progress, 2014), SimCityEDU: Pollution Challenge! (Mislevy et al., 2016), and the Tetralouge (Hao, Liu, von Davier, & Kyllonen, 2015; Liu, Hao, von Davier, Kyllonen, & Zapata-Rivera, 2016). In The Wells Task (National Assessment of Educational Progress, 2014), students are presented with a problem-based assessment about a malfunctioning pump in a Nepalese village. Through an interactive simulation, they apply knowledge about scientific principles governing fluid dynamics to create a maintenance plan. In SimCityEDU: Pollution Challenge! (Mislevy et al., 2016), students are presented with an IVE in which to troubleshoot environmental challenges, such as reducing pollution. In Tetralogue (Hao et al., 2015; Liu et al., 2016), students work in pairs to predict a volcano alert in a simulated environment. Understanding how to best implement technology for assessment continues to be an important area of educational research (U.S. Department of Education, 2017).

A significant dilemma in validating these technologies for assessment is that outcomes of student engagement and learning vary widely based on learners and contexts (U.S. Department of Education, 2017). For example, in a study of over 100 schools integrating reading and math software, Dynarski et al. (2007) found that student achievement in some schools improved but declined in others. Many contextual factors can be attributed to the wavering efficacy of technology integration in schools, such as differing student populations, resource availability, parental involvement, and teacher knowledge. Educational researchers have suggested that considering the existing components of classroom ecology may be a more effective strategy to draw on the affordances of technology-enhanced learning environments (National Academies of Sciences and Medicine, 2018). For example, in the Texas SimCalc study (Roschelle et al., 2010), mathematics software was integrated together with paper activities, small group work, and classroom discussions. Teachers were supported with several days of PD and a teacher’s guide with sample student responses. Using this approach, Roschelle et al. (2010) found an increase in mathematics skills across different classrooms.

A critical component of technological integration in schools is teacher attitude and beliefs, which can have a significant impact on practice (Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012; Kim, Kim, Lee, Spector, & DeMeester, 2013). Educational researchers have suggested resources to support teachers implementing technologies for assessment, such as supplemental lesson plans, dashboards, and data reporting in order to connect the virtual and physcial environments (U.S. Department of Education, 2017). In this study, we explore how supporting teachers through PD can impact teacher beliefs and practices about implementing an IVE for assessment.

PD for IVEs

Novel technologies that promote scientific inquiry, such as IVEs, require teachers to integrate technological knowledge into their established teaching practice. Mishra and Koehler (2006) described seven subsets of teacher knowledge required to effectively incorporate technology into the classroom, referred to as the Technological Pedagogical Content Knowledge framework. Technological Knowledge is understanding how to use the technology. Pedagogical Knowledge is familiarity with appropriate strategies for teaching and learning. Content Knowledge is understanding the subject matter. Pedagogical Content Knowledge is best practices to teach subject matter (Shulman, 1986). Technological Content Knowledge is understanding how technology can enhance subject matter. Technological Pedagogical Knowledge is familiarity with appropriate pedagogical applications of the technology. Finally, Technological Pedagogical Content Knowledge is integrating technology in specific subject matter with pedagogically appropriate strategies. Contextual knowledge (XK) is understanding the norms and policies within the communities that the technology is integrated. PD experiences are essential to support the development of this emerging skill set (U.S. Department of Education, 2017). While the field continues to establish effective strategies for PD supporting technology integration (Lawless & Pellegrino, 2007), evidence suggests that characteristics of high-quality PD include prolonged support, just-in-time assistance, collaboration with other teachers, opportunities to inform instruction and alignment with school practices (Fishman et al., 2016).

PD may help guide teacher practices during the implementation of technology, which can impact how much students learn from IVE experiences (Metcalf et al., 2013). Evidence has suggested that teacher implementation of IVEs after participating in PD varies due to both personal beliefs and outside influences. For example, Tutwiler (2014) found that even though teachers were in the same PD, their students showed different patterns of data collection in the IVE, indicating that despite being in the same PD, different teachers were giving different types of student support. While the Tutwiler study did not investigate teacher practices per se, a study by Ketelhut and Schifter (2011) did. They found that essential aspects of PD for improving best practices of implementation included time for teachers to develop comfort with the technology, modeling of successful implementation strategies, and then providing just-in-time technical assistance during implementation. However, even with those, teachers were more likely to continue to use novel technologies over time if researchers and designers engaged in the school community. Based on these prior studies which shed light on the support needs of teachers as they implement novel technologies in the classroom, we designed the PD experiences for teachers in our project to continue throughout their participation, providing just-in-time support. However, what is still less understood is how PD impacts teacher beliefs and knowledge, and how those impact what a teacher does in the classroom.

Theoretical Perspective: Teacher Change

In order for teachers to effectively integrate IVEs into their practice, they must change the ways in which they facilitate learning activities in their classrooms. We seek to understand how to best construct PD experiences that support teachers to develop a scientific inquiry teaching skill set, aligning with the inquiry-infused pedagogy in the IVE. Historically, PD efforts have aimed to change teachers’ beliefs and understandings, with the assumption that teacher practice will change accordingly (Harris, 1980). Guskey (1986) provided an alternative conceptualization for teacher change in which teacher practice was a critical component. Clarke and Hollingsworth (2002) built on Guskey’s model to suggest that PD experiences were connected to teacher beliefs, teacher practices, and student outcomes in a complex, interconnected, and nonlinear relationship, illustrated by the IMPG (Figure 1). This model includes four analytical domains. The external domain refers to the PD experience outside of the classroom. The domain of practice refers to the pedagogies that teachers enact in their classroom. The personal domain refers to teacher knowledge, beliefs, and attitudes about the pedagogy. Finally, the domain of consequence refers to student learning, motivation of management outcomes resulting from implementing a new pedagogy. The model suggests that teacher change requires reflection and enactment between each of these four domains. In addition, the process of teacher change may be encouraged or limited by the community, school, or district, referred to as the change environment.

Figure 1.

The interconnected model of professional growth (Clarke & Hollingsworth, 2002).

We have selected this model to analyze the development of scientific inquiry through integration of an inquiry-infused technology because it emphasizes where change and growth were facilitated by experiences with our project, as well as what obstacles to change were encountered. As we discuss the findings later, we will focus our results on each domain and indicate where enactment and reflection happen.

Methods

Study Context

SAVE Science, a project funded by the National Science Foundation, studied the implementation of IVE-based modules designed to assess middle school science content and practices from 2008 to 2015 (Ketelhut et al., 2013). These assessment modules, focused on questions poorly assessed in traditional school assessments, were designed to evaluate science content and inquiry integrated within a problem-based scenario. There are four modules (plus two introductory modules) designed to evaluate biology, meteorology, physics, or chemistry content, specifically adaptation, fronts, forces and motion, or gas laws.

In this study, we focused on the Basketball module, which simultaneously assessed knowledge of gas laws and elements of scientific inquiry (illustrated in Figure 2). Students were challenged to solve the problem of why identically inflated basketballs were not bouncing the same in indoor (gym) and outdoor conditions (park). To solve this problem, students gathered data using virtual tools such as a measuring tape, a thermometer, and a pressure gauge. Students were also able to transport the basketballs as well as balloons between the indoor and outdoor locations and measure properties of the objects in both environments. Module completion took approximately 20 to 30 minutes. All activities were performed in school, under the direct supervision of the students’ science teachers. Postimplementation, classroom discussions were video recorded to explore how teachers enabled students to construct scientific explanations through classroom discourse.

Figure 2.

Screenshots from the basketball module, showing scenes in the gym and park.

Participants

SAVE Science was implemented by teachers from public urban, near-urban, and suburban middle schools with their sixth through eighth-grade science classes. Teachers volunteered to participate in SAVE Science, implementing one or more IVE science modules with their classes in public middle schools (Grades 6–8) per year. Informed consent was obtained from teachers, parents/guardians, and students. This study focused on one teacher, George (a pseudonym), through 3 years of implementations. George was an eighth-grade science teacher at a suburban, public middle school with 15 years of teaching experience when he first began working with SAVE Science in 2010. George implemented four different assessment modules in 15 different eighth-grade classes over 3 school years.

Between 2010 and 2013, he attended three PD sessions and sat in on one additional training session for new recruits. The PD sessions offered teachers time to learn about the modules, discussion and review of scientific inquiry for middle school science, and development and discussion of lesson plans that connected classroom content to the virtual assessment content in the modules (see Table 2 for a detailed description). In August 2010, George attended a 3-day PD session in which he participated in scientific inquiry activities modeled by the research team and then designed an inquiry-based curriculum related to the content of the IVE. In March 2011, George participated in an afternoon PD session that reinforced the concepts discussed in the summer. A scientific inquiry activity was modeled, and teachers were encouraged to collaboratively design an inquiry lesson for their classrooms. In August 2011, George attended a full-day workshop in which inquiry-infused instruction was emphasized again in addition to investigating student data from the IVE for formative assessment. These same concepts were reinforced again in October 2011. Finally, in March 2012, teachers collaborated in the design and implementation of an inquiry-based activity and discussed how to use data from the IVE to make pedagogical decisions. Many teachers participated in our project for multiple years, developing a cohort for collaborative professional learning.

Data Collection

We were interested in examining the ways in which implementation of the IVE in conjunction with the PD experiences appeared to catalyze change for the participating mentor teachers in terms of their enactment of inquiry-infused practices in their classroom. Our overall corpus of data includes classroom observations, postimplementation classroom discussions, teacher surveys, and teacher interviews.

Researchers took field notes and video recorded a total of 14 classroom observations between 2011 and 2013. These classroom observations occurred during “typical” 50-minute science classes. George implemented an IVE to five classes in 2010 to 2011, four classes in 2011 to 2012, and four classes in 2013 to 2014.

The research team video recorded the implementation and the postimplementation classroom discussion in order to investigate how the teacher enabled students to construct scientific explanations of the IVE-based problem through classroom discourse. Teachers were prompted to ask questions to the class such as, “How did you go about solving the problem? Who did you talk with? What kinds of information did you gather?” Teachers were encouraged to act as facilitators, allowing students to argue and explain how they solved the problem in the virtual world.

In addition, teacher surveys which explored enacted curriculum (Blank, Porter, & Smithson, 2001) and other factors such as science teaching self-efficacy were administered before and after the IVE implementation each year. Examples of survey questions asked the teacher to indicate how much class time students spent doing various activities, such as “doing a laboratory activity, investigation or experiment” or “listening to a teacher explain something to the class as a whole about science.”

Finally, students completed pre- and posttests that assessed their scientific self-efficacy in addition to science content and practices addressed in the IVE. The posttest consisted of five multiple-choice questions and two free-response questions. An example of a question was as follows: “A child riding a bicycle notices that the tires are more inflated on hot days than on cold days, even though no air is being added or removed. How can this be explained?” In Table 1, we summarize data sources contributing to this case study.

Table 1.

Summary of Data Sources.

Data source	Description of data source	Quantity	Year
Classroom observations	Researchers took field notes and video recorded a total of 14 classroom observations between 2011 and 2013. These observations consisted of one period of science teaching.	10	2011–2012
Classroom observations		4	2012–2013
Postimplementation classroom discussions	Postimplementation classroom discussions were video recorded to explore how the teacher enabled students to construct scientific explanations of the problem through classroom discourse. A total of 13 classroom discussions, each following the completion of a SAVE Science module, were audio recorded and transcribed.	5	2010–2011
		4	2011–2012
		4	2012–2013
Teacher surveys	Teacher surveys were administered pre- and postimplementation each year. These surveys took about 20 minutes to complete and explored enacted curriculum (Blank et al., 2001), and other factors such as science teaching self-efficacy.	2	2010–2011
		1	2011–2012
		2	2012–2013
Teacher interviews	Two semistructured teacher interviews were video recorded and transcribed.	1	2011–2012
Teacher interviews		1	2012–2013
Student posttests	Students took a test on science content that was infused in the IVE module after participating in the IVE	107	2010–2011
		75	2011–2012
		85	2012–2013
Student pretests	Students took a test on science content that was infused in the IVE module before participating in the IVE.	74	2012–2013

Note. IVE = immersive virtual environment.

We identify how each data source provided evidence for the domains of the IMPG (Clarke & Hollingsworth, 2002), offering insight about teacher change that was facilitated by experiences with our project. For example, classroom observations and postimplementation classroom discussions provided data about classroom practice (domain of practice). Teacher surveys and teacher interviews contributed information about teacher beliefs and perceptions (personal domain). Student pretests and posttests provide data about student learning gains after the IVE experience (domain of consequence).

Data Analysis

In this mixed-methods case study, we first analyzed each data source independently. The classroom observations were analyzed for inquiry science teaching using the Science Teaching Inquiry Rubric (STIR; Bodzin & Beerer, 2003). On this rubric, there are five aspects of inquiry-based instruction: engaging with scientific questions, prioritizing evidence, formulating evidence-based explanations, evaluating explanations, and communicating and justifying explanations. Each component is scored on a 5-point rubric from teacher-centered (e.g., “no evidence observed” [p. 43]) to learner-centered (e.g., “learner is prompted to formulate own questions or hypothesis to be tested” [p. 43]).

The postimplementation classroom discussions were analyzed for how the teacher enabled students to construct scientific explanations according to the Claim, Evidence, and Reasoning framework (McNeill & Krajcik, 2008). McNeill and Krajcik (2008) defined a claim as “a conclusion that answers the original question” (p. 134). They describe evidence as “scientific data that supports the claim” (p. 134) and reasoning as “a justification that links the claim and evidence” (p. 134). Each component is scored on a 3-point rubric to assess student explanations. A score of 0 on any level would indicate the student was not making a claim, supporting the claim with evidence or providing reasoning. A score of 2 signifies that a student was making accurate claims, supporting claims with sufficient evidence and providing appropriate reasoning that links the claim with the evidence.

Teacher surveys and interviews were analyzed using qualitative coding methods (Saldaña, 2016). In our first round of coding, the research team inductively coded all data sources to generate themes (Braun & Clarke, 2006) related to teachers’ views on integrating inquiry into science teaching. These themes were cross-checked and coordinated by two researchers and triangulated across other data sources (field notes, classroom observations, and classroom discussions) in order to maintain validity. These themes were then applied in a second deductive coding phase using Clarke and Hollingsworth’s (2002) IMPG model as a lens for interpreting our data.

Student pre- and posttests were analyzed to measure student learning. On the posttest, there were five multiple-choice questions. On the pretest, there were seven multiple-choice questions. A paired samples t test was conducted to compare students’ mean scores in pretest and posttest.

We followed a constant comparative process (Kolb, 2012), noting thematic patterns between the interrelated data sets (e.g., survey responses and interview transcript), then comparing themes across different sets. Specifically, we compared how the teacher perceptions and beliefs as evidenced by teacher surveys and interviews (PERSONAL DOMAIN) compared with students’ learning gains as evidenced by pre–post test data (DOMAIN OF CONSEQUENCE) and teacher enactment of inquiry practices as evidenced by classroom observations and whole-class discussions (DOMAIN OF PRACTICE) to uncover how change in each of these domains may contribute to change in teacher practice.

Findings

Overall, we found that participation in our project resulted in some changes to George’s teaching schema, but not in others. After integrating an IVE in his classroom and attending multiple PD sessions, he expressed unwavering belief that inquiry-based instruction was an important part of his practice. However, it is not evident that this belief translated into practice.

External Domain

To support teachers in successfully integrating IVEs, and the student-centered inquiry-based instruction of IVEs, the PD focused in part on how to integrate scientific inquiry modeled in the IVE into everyday teaching and learning. Between 2009 and 2013, multiple PD sessions were offered per year. Each PD session focused on topics related to the included inquiry-based science activities to model the pedagogy; we envisioned to support the IVE module in addition to IVE design and implementation ideas. Later PD was devoted to understanding and data mining student data. For the purpose of this study, we focus on the eight sessions that were highly focused on the scientific inquiry aspects of the PD. In one such activity, participants engaged in an exploration of how structure is related to function by using different kitchen utensils to obtain food in water. In several sessions, teachers built on these activities (geared for the development level of their students) to collaboratively design scientific inquiry-infused lessons to embed in their classroom in alignment with their standards and curriculum. A summary of the scientific inquiry aspects of these eight PD sessions can be found in Table 2.

Table 2.

Summary of Professional Development Sessions.

Date	Description of professional development activities	Number of hours	George attended
April 2010	Researchers introduced the project to teachers and modeled scientific inquiry activities.	6
August 2010	Teachers participated in scientific inquiry activities and designed inquiry-based curriculum to integrate the IVE module into their classroom.	21	✓
October 2010	Researchers modeled a classroom-based implementation of the IVE and provided further examples of inquiry-based instruction.	6
March 2011	Teachers participated in more scientific inquiry activities and designed inquiry lessons to integrate into their classroom.	3	✓
April 2011	Teachers critiqued scientific inquiry lessons of peers and reflected on scientific inquiry activities in their own teaching practice.	3
August 2011	Researchers provided further examples of inquiry-based instruction and reviewed how to generate student data from IVE. Teachers designed inquiry-based lessons.	6	✓
October 2011	Researchers provided further examples of inquiry-based instruction and reviewed how to generate student data from the IVE.	6	✓
March 2012	Teachers collaborated in the design and implementation of an inquiry-based activity and discussed how to use data from the IVE to make pedagogical decisions.	3	✓

Note. IVE = immersive virtual environment.

George participated in five of these eight scientific inquiry-focused PD sessions. He was an enthusiastic participant in the project. In addition to the PD sessions, he attended several advisory board meetings for the research project where he was a vocal proponent of the project in the later years.

Reflecting on the PD, George valued his experience collaborating with other professionals in the community and reflecting on the project. He explained,

This gave me the chance to talk to the other teachers that were doing the modules when we met down at the University and talk, “What did you find? What did you find?” And even if it wasn’t that we were doing the same module, it was how did the students react, what did you find, where and how did you have or solve any issues like this came up? So, it’s always good to talk to other people in the field who are doing the same thing because they’ll have other ideas, other experiences. I gave some ideas to them, they gave some ideas to me and you want to try to incorporate those that are relevant into your training. The best thing is meeting with other people doing it and what did you find? You can’t get any better than that. (Interview, Fall 2011)

He said that while the PD did not improve his own science content understanding, because he had a college degree in science and knew the content, it did provide a new and different pedagogical tool that could enable his students to learn science content.

Domain of Consequence

George’s students demonstrated positive learning gains through their engagement with the IVE. For the 2012 to 2013 school year, students took a pretest before the module in order to measure learning gains from the IVE experience. A paired t samples test was conducted to compare students’ mean scores in pretest and posttest. The sample was 74 students who had completed both the pre- and posttest. The results showed that students’ posttest scores (M = . 673, standard deviation = .26) were significantly higher than pretest scores, M = .463, standard deviation = .22, t(73) = − 6.185, p < .001. Therefore, students earned a higher score on a content related assessment after playing the inquiry-based basketball module. These results suggest the IVE was a positive learning experience for George’s students. Reflecting on his students’ experience with SAVE Science, George said,

It’s interesting in seeing how some of them react, because my classroom is very hands on … so they’ve been working this way. Now, going with the SAVE Science, where now it was strictly video, was interesting watching how some of them adapted quite well to it, while others were more hesitant and you could see they learned a little differently, and going that was harder for them to grasp. Actually, some of my lower students went right to the video very well, other ones that were very hands on, had trouble. I would have maybe thought it would go one way or another but it was a real mix of how they adapted to it. (Interview, Fall 2011)

While he acknowledged the spectrum of student experiences, his overall impression was that the experience benefited student learning. In a PD session, George expressed how wonderful the program was with its emphasis on student-centered inquiry, of which he was a big fan. During one of the advisory board meetings that he attended, he made a point of sharing his reflections that he had learned a great deal from participating in the project and his students benefited as well (private communique).

Personal Domain

Review of surveys and interviews with George revealed that his perception of himself teaching science was in alignment with the student-led inquiry-based science instruction that was modeled in the IVE and in the PD workshops. Later, we identified three categories in which George expressed this perception. The first is his beliefs about his science teaching pedagogy, evidenced by interviews and survey data. The next is his perception about the role of questioning students in his classroom discussion. The last is how he believed he facilitated students to evaluate data in his classroom.

Science teaching pedagogy

George stated, “my classroom is very hands on. They don’t even have assigned textbooks, we work from off the internet, we have handouts, we have a lab manual that kind of guides. And so they’ve been working this way” (Interview, Fall 2011). Consistently for 3 years, George indicated that his class spent the least amount of time participating in whole-class lecture, about equal time participating in laboratory activities and teacher demonstrations, and the most time participating in group work (Figure 3).

Figure 3.

George’s perception of time devoted to classroom activities according to survey of enacted curriculum (Blank et al., 2001). Average of five surveys across 3 years.

Questioning students

George indicated in interviews that he believed questioning students is a very important part of his pedagogy. For example, he stated,

A lot of it is “why,” “what do you mean,” it’s questioning. They’ll go ahead and do something and I’ll say well, what does that mean, how does that relate to something else that we did, try and make them reach out and not just give the very … you don’t want to ask questions that they can answer yes or no with. A lot of it is developing questioning techniques and make them relate that information of what they’re finding to other things or either that lab, or how does this relate to a previous lab that we did. (Interview, Fall 2011)

Evaluating data

Finally, when asked if he asks his students to evaluate information, George responded, “Absolutely.” He went on to give an example, “one of the students got an answer that a person weighs 2000 lbs. and I said is that realistic? Someone’s 2000 lbs?”

In this case, we see that George’s beliefs are supported by reflections about students and their practice. He thinks that students learn best through hands-on, inquiry-based activities. He believes that he facilitated these activities through engaging in high-level questioning and promoting the evaluation of data. He feels that this practice improves student learning.

Domain of Practice

Video recordings and classroom observations of George’s class revealed that his student-led, inquiry-based perception of himself was not in alignment with his actual didactic science instruction. We use the same three categories described earlier to illustrate the contradiction between his perceptions and beliefs described in the previous section and his science teaching practice.

Science teaching pedagogy

Nine of the 13 classroom observations contained majority whole-class lecture. Using the STIR (Bodzin & Beerer, 2003), 69% of observed classes did not provide learners with an opportunity to engage with a scientifically oriented question, and 31% of observed classes provided learners only a basic opportunity to engage with scientifically oriented questions (Figure 4).

Figure 4.

Percentage of opportunities learners were given to engage with a scientifically oriented question during 13 classroom observations over 2 years according to the STIR (Bodzin & Beerer, 2003). STIR = Science Teacher Inquiry Rubric.

Questioning students

Twelve classroom discussions were audio recorded following implementation of SAVE Science. During these discussions, students made approximately 1 to 2 claims about their virtual world experience (an average of 1.33 claims per class). Students sometimes supported these claim with evidence from the virtual world (an average of .92 times per class). Less frequently, students explained their evidence with scientific reasoning (an average of .42 times per class). In all classroom discussion, there were zero instances of rebuttals (Table 3). These results are summarized in Table 3.

Table 3.

Average Number of Student Utterances Per Classroom Discussion Across 3 Years of postimplementation Classroom Discussions Coded According to the Scientific Explanation Framework (McNeil & Krajcik, 2011).

Scientific explanation framework (McNeil & Krajcik, 2011)	Average number of student utterances per discussion
Claim	1.33
Evidence	0.92
Reasoning	0.42
Rebuttal	0

We attribute the low number of claims, and even lower number of evidence, reasoning, and rebuttals to the teachers’ authoritative role. In all 12 classrooms, George consistently prevented opportunities for students to construct explanations by taking a role as the source of authority for knowledge. The following dialogue presented is an example of this. In this example, the teacher quickly provided the answer to the student’s inquiry, with no attempt to facilitate a discussion among classmates. In fact, the teacher’s approach is in direct opposition to how the project staff encouraged teachers to respond to such questions. While the project staff repeatedly suggested that teachers turn a question to the rest of a class, in this case other students that may have had a relevant experience were not given an opportunity to respond and share.

Teacher: What did you tell Julius at the end when you went up to him back in the gym in the wheelchair? Yes? Student 1: Uh, the basketballs outside were more flat because it was too cold. Teacher: It was too cold. Okay, the temperature was causing the basketballs to be flat. Anything else? Anyone say something different? Yes? Student 2: Bounce height of like how much it bounces different. Teacher: Okay, because of that … it’s not going to bounce to the regulation height that it’s supposed to on there. Yes? Student 3: The air particles inside the basketball … Teacher: Okay, because it was cold the particles were moving too slow. Then, the volume ended up getting down a little bit or so on there. That was one thing if you noticed on the circumference, if you measured the circumference. By the way, when you measured the weight, did the weight change? Was the weight any different on any of the basketballs? Did anyone check the weight? Was the weight different? Student 4: No Teacher: The weight, weight would’ve been just about the same. The fact that because of the volume it, it was, what was changing was temperature, the basketballs didn’t have a leak, so they would’ve been filled up with the air like that.

This approach to classroom discussion is also seen in the following dialogue. One student asked the teacher why there were multiple objects to measure. This teacher also quickly and authoritatively responded that just one measurement in science in insufficient to validate a scientific claim. Interestingly, in this situation, three students attempted to engage in argumentation, but the conversation was closed and the opportunity lost by the teacher’s response.

Student 1: Why was there like so many like, do you know how like all the basketballs there was like three on each court? Teacher: Um-hum Student 1: Like why was that because they were almost all the same? No, they were all the same. Teacher: Normally how many, when you do an experiment how many trials do you do? Student 1: One Student 2: Three Student 3: A couple. Teacher: Three. Student 1: Yeah. Teacher: You usually tend to do multiple trials when you do an experiment. And you’ll see when we start on the, uh, energy machines in motion. You’ll be doing three or four trials for everything, because you want to verify your data. Just one trial is not, uh, good enough. So I think that’s what they were getting in that, in that part of it. Why there, why there were three balloons inside or two balloons inside, two balloons outside, three basketballs inside, three basketballs outside. You always want to do more than one trial.

Evaluating data

Using the STIR, 0 of the 13 classroom observations provided learners an opportunity to evaluate their conclusions and explanations (Figure 5).

Figure 5.

Percentage of opportunities learners were given to evaluate their conclusions and explanations during 13 classroom observations over 2 years according to the STIR (Bodzin & Beerer, 2003). STIR = Science Teacher Inquiry Rubric.

George showed little evidence of reflection on his practice. This is likely a contributing factor to the misalignment between his positive beliefs about scientific inquiry and the lack of scientific inquiry in his practice.

Discussion

What becomes evident through review of the classroom observations is that George’s enactment of scientific inquiry (domain of practice) was in direct opposition to how the project staff worked with teachers to develop skills to construct arguments and conduct inquiry in their classrooms in PD sessions (external domain) through the 3 years of his participation in the project. However, a more significant finding of this case study was that George’s surveys and interviews revealed that the enactment of his pedagogy was actually in direct opposition to his own beliefs about his teaching practices (personal domain) and his reflections on his own practice. George believed he was enacting what the PD sessions had helped him see was important and pointed to student outcomes to support his “change.” However, his classroom practice remained didactic and traditional (Figure 6).

Figure 6.

Illustration of the relevant domains and reflection to George’s change.

Figure 6 shows George’s own IMPG. He participated in the external domain through the PD. He implemented the IVE in his classroom and believed he was implementing student-driven inquiry because of its positive impact on student learning. In Figure 6, we use the arrows to show how and where he reflected between the various domains. George’s surveys are clear that his beliefs were changed as a result of his reflections on the domain of consequence and the external domain, and his interviews showed that he reflected on student outcomes to support his decision to change his practice as promoted in the PD. Using the IMPG as an analytical tool, we can see that George enacted in all four domains and reflected between all of them except between the external domain and the domain of practice.

As we have shown through his own words, George believes that inquiry-based pedagogy is important and can actually help students. But, clearly, what his students experienced was not what we had hoped they would. Educational researchers have suggested that IVEs have the potential to impact teacher practice to be more inclusive of inquiry practices (Barab et al., 2006; Grotzer et al., 2017). However, George reveals the complexity of such as claim. While PD and positive student outcomes impacted his beliefs about inquiry-infused, student-centered teaching, these experiences did not afford more inclusive inquiry practices in his teaching. These findings support the complexity of teacher change that the IMPG model suggests. In particular, that a change in teacher beliefs does not constitute a change in teacher practice, as previously believed (Harris, 1980). Our results reflect a familiar challenge in science education: despite efforts to support teachers in facilitating scientific inquiry experiences, obstacles prevent the enactment of authentic scientific practices in the classroom. Using the IMPG as an analytical lens to help us understand why George’s practice did not match the recommendations of the PD, the design of the IVE, or even his own beliefs, we see two elements that might help us explain this outcome. First, George only reflected on his own understanding of how he taught students. At no time, did he actually sit down to review videos of his practice. In Figure 6, thus, we are missing a reflective arrow between the external domain and the domain of practice. We therefore recommend that future research investigate whether PD to PD must also include a reflection on actual practice, not just on beliefs.

Second, the IMPG suggests that even if there is enactment and reflection within and between all domains, the change environment itself might either support or impede change. For change to a more scientific inquiry pedagogy, previous research hints at prohibitory factors in the change environment that may have influenced George. For instance, an overreliance on textbooks, lack of teacher knowledge, or confidence in tried-and-true didactic teaching methods (Ketelhut & Tutwiler, 2017). Moreover, the nationwide emphasis on high-stakes testing could have influenced George to present science as a body of facts, to be tested. Unfortunately, our interview with George did not push him to reflect on the change environment, so these are hypotheses about possible factors to explain the disconnection between his beliefs about his practice and the actual practice. We encourage future researchers to probe for these complications. These factors in the change environment are critical for innovators to consider when designing PD experiences for teachers.

As developers of technology, we hoped that participation in SAVE Science would impact teacher beliefs and practices. While it is possible that participation in the project prompted George to believe that inquiry practices were an important part of his teaching, these beliefs were not transferred into practice. George’s case emphasizes a need for future research to develop teacher learning experiences that help teachers gain a rich, multiperspective understanding of the goals and best practices of IVEs. It may be necessary for training to extend beyond PD sessions and into the classroom in order to promote critical self-reflection of practice. While this single case study highlights essential challenges for the field to consider, more research is required to determine the generalizability of the findings to larger samples of teachers.

Conclusion

While IVEs model scientific inquiry practices and can provide rich learning experiences for students, teacher practices are a critical component of how students engage in learning with the IVE. A goal of our project was to support teachers in integrating inquiry-based practices into their classrooms through the implementation of IVEs coupled with ongoing PD. This case study of one teacher found that teacher beliefs were impacted by participation in our inquiry-infused technology program; however, classroom practice remained traditional and teacher-centered. Our findings present an interesting challenge to bridge the pedagogy of the classroom with the pedagogy of the technology: how to support teacher change when teacher beliefs are misaligned with classroom enactment. They also raise a question about the belief that IVEs can model and promote good scientific inquiry pedagogy. Future research should develop novel teacher-training methodologies that emphasize inquiry-based practice in conjunction with innovative technologies. These experiences in turn should improve teacher efficacy and confidence incorporating IVEs and inquiry-based pedagogies into their regular classroom practice, ultimately benefiting the science, technology, engineering, and mathematics learning of students.

Footnotes

Acknowledgments

The authors gratefully acknowledge the support of the project team, our teacher partners, and National Science Foundation. This article is a greatly expanded version of a paper presentation to the Immersive Learning Research Network’s 2018 conference ().

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the National Science Foundation under Grant No. 0822308.

Kelly Mills is a postdoctoral associate at the University of Maryland, College Park. She studies how teacher education can support the development of best practices with innovative technologies in formal learning environments. She is currently working as a postdoctoral associate on a National Science Foundation (NSF)-funded STEM+C project investigating computational thinking professional learning experiences for preservice elementary teachers. She has worked on several NSF-funded projects focused on developing and integrating innovative technologies in K-12 learning environments. These studies include Science Everywhere, which explored the integration of a social media application for science learning in informal and formal learning environments and SAVE Science, a game-based system for assessing science learning for middle school children. She was a high school science teacher for 8 years. She received a BS in biology, MEd in curriculum and instruction, and PhD in science education from the University of Maryland, College Park. She can be reached at kmills1@umd.edu.

Diane Jass Ketelhut is an associate professor of science and technology education at the University of Maryland. Her research centers on improving science self-efficacy and learning through scientific inquiry experiences within virtual environments. She is currently working as the principal investigator on the National Science Foundation-funded STEM+C project to investigate the integration of computational thinking into elementary preservice science teacher education. She served as principal investigator for the SAVE Science project, a game-based system for evaluating learning in science for middle school children, and project director for the River City project. She has authored/coauthored over 50 publications. Certified in secondary science, she was a science curriculum specialist and science/math teacher for Grades 5 to 12 for 12 years. Diane received a BS in biomedical sciences from Brown University, an MEd in curriculum and instruction from the University of Virginia, and PhD in learning and teaching from Harvard University. She can be reached at diane.jass.ketelhut@gmail.com.

Xiaoyang Gong is an assistant professor at Tianjin Normal University (TJNU), Tianjin, China. Her research interests include how teachers’ technological knowledge relates to their pedagogical strategies of using technology and how students’ achievement emotions function in regular or technology-integrated science classrooms. She has worked on a National Science Foundation-funded project called SAVE Science, which investigated students’ inquiry activities in a game-based assessment system. She is the principal investigator of a TJNU-funded social science project investigating how students’ perceptions of self-efficacy beliefs and scientific values relate to achievement emotions in science classrooms. In addition, she was a high school chemistry teacher for 1 year in China. She received a BS in chemistry from Beijing Normal University and PhD in curriculum and instruction from the University of Maryland, College Park. She can be reached at xiaoyang.gongbnu@gmail.com.

References

Barab, S., Jackson, C., & Piekarsky, E. (2006). Embedded professional development: Learning through enacting innovation. In C. Dede (Ed.), Online professional development for teachers: Emerging models and methods (pp. 155–174). Cambridge, MA: Harvard Education Press.

Barab

Thomas

Dodge

Carteaux

Tuzun

(2005) Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development 53(1): 86–107.

Blank, R., Porter, A., & Smithson, J. (2001). New tools for analyzing teaching, curriculum and standards in mathematics & science. Results from survey of enacted curriculum project (Final Report). Washington, DC: Council of Chief State School Officer.

Bodzin

Beerer

K. M.

(2003) Promoting inquiry-based science instruction: The validation of the Science Teacher Inquiry Rubric (STIR). Journal of Elementary Science Education 15: 39–49.

Braun

Clarke

(2006) Using thematic analysis in psychology. Qualitative Research in Psychology 3(2): 77–101.

Chan, C. K., & Yang, Y. (2018). Developing scientific inquiry in technology-enhanced learning environments. In J. Voogt, G. Knezek, R. Christensen & K-W. Lai (Eds.), Second handbook of information technology in primary and secondary education (pp. 161–180).

Clark

Tanner-Smith

E. E.

Killingsworth

S. S.

(2016) Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research 86(1): 79–122.

Clarke

Hollingsworth

(2002) Elaborating a model of teacher professional growth. Teaching and Teacher Education 18(8): 947–967.

DeBoer

(1991) A history of ideas in science education: Implications for practice, New York, NY: ERIC.

10.

Dede

Grotzer

T. A.

Kamarainen

Metcalf

(2017) EcoXPT: Designing for deeper learning through experimentation in an immersive virtual ecosystem. Journal of Educational Technology & Society 20(4): 166–178.

11.

Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Javitz, H. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort.

12.

Ertmer

Ottenbreit-Leftwich

A. T.

Sadik

Sendurur

(2012) Teacher beliefs and technology integration practices: A critical relationship. Computers & Education 59(2): 423–435.

13.

Fishman, B., Dede, C., & Means, B. (2016). Teaching and technology: New tools for new times. In D. Gitomer, & C. Bell (Eds.), Handbook of research on teaching (pp. 1269–1334). Washington, D.C.: American Education Research Association.

14.

Fletcher

Tobias

(2011) Computer games and instruction, Charlotte, NC: Information Age Publishing.

15.

Grotzer

Metcalf

Tutwiler

Kamarainen

Thompson

Dede

(2017) Teaching the systems aspects of epistemologically authentic experimentation in ecosystems through immersive virtual worlds, San Antonio, TX: National Association for Research in Science Teaching.

16.

Guskey

(1986) Staff development and the process of teacher change. Educational Researcher 15(5): 5–12.

17.

Hao

Liu

von Davier

Kyllonen

(2015) Assessing collaborative problem solving with simulation based tasks, Gothenburg, Sweden: International Society of the Learning Sciences.

18.

Harris

B. M.

(1980) Improving staff performance through in-service education, Boston, MA: Allyn & Bacon.

19.

Jiao, H., & Lissitz, R. W. (2018). Technology enhanced innovative assessment : Development, modeling, and scoring from an interdisciplinary perspective (Vols. 1–1 online resource). Charlotte, NC: Information Age Publishing. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=1535637.

20.

Ketelhut

D. J.

(2006) Do students with higher self-efficacy exhibit greater and more diverse scientific inquiry skills: An exploratory investigation in “River City”, a multi-user virtual environment, Cambridge, MA: Harvard University.

21.

Ketelhut

D. J.

(2007) The impact of student self-efficacy on scientific inquiry skills: An exploratory investigation in River City, a multi-user virtual environment. Journal of Science Education and Technology 16(1): 99–111.

22.

Ketelhut

D. J.

Nelson

B. C.

Clarke

Dede

(2010) A multi-user virtual environment for building and assessing higher order inquiry skills in science. British Journal of Educational Technology 41(1): 56–68.

23.

Ketelhut

D. J.

Nelson

Schifter

Kim

(2013) Improving science assessments by situating them in a virtual environment. Education Sciences 3(2): 172–192.

24.

Ketelhut

D. J.

Schifter

C. C.

(2011) Teachers and game-based learning: Improving understanding of how to increase efficacy of adoption. Computers & Education 56(2): 539–546.

25.

Ketelhut

D. J.

Tutwiler

M. S.

(2017) Science Learning and Inquiry with Technology, Abingdon, England: Taylor & Francis.

26.

Kim

M. K.

Lee

Spector

J. M.

DeMeester

(2013) Teacher beliefs and technology integration. Teaching and Teacher Education 29: 76–85.

27.

Kolb

(2012) Grounded theory and the constant comparative method: Valid research strategies for educators. Journal of Emerging Trends in Educational Research and Policy Studies 3(1): 83.

28.

Lawless

Pellegrino

J. W.

(2007) Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research 77(4): 575–614.

29.

Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata-Rivera, J.-D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrera, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 344–359). Hershey, PA: IGI Global.

30.

McNeill, K. L., & Krajcik, J. (2008). Inquiry and scientific explanations: Helping students use evidence and reasoning. In J. Luft, R. Bell, & Gess-Newsome (Eds.), Science as inquiry in the secondary setting (pp. 121–134). Chicago, IL.

31.

McNeill

K. L.

Krajcik

J. S.

(2011) Supporting grade 5-8 students in constructing explanations in science : The claim, evidence, and reasoning framework for talk and writing, Boston, MA: Pearson.

32.

Metcalf, S. J., Kamarainen, A. M., Grotzer, T., & Dede, C. (2013). Teacher perceptions of the practicality and effectiveness of immersive ecological simulations as classroom curricula. International Journal of Virtual and Personal Learning Environments (IJVPLE), 4(3), 66–77.

33.

Mikropoulos

T. A.

Natsis

(2011) Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education 56(3): 769–780.

34.

Mishra

Koehler

M. J.

(2006) Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record 108(6): 1017.

35.

Mislevy, R. J., Corrigan, S., Oranje, A., DiCerbo, K., Bauer, M. I., von Davier, A., & John, M. (2016). Psychometric considerations in game-based assessment. Technology and testing: Improving educational and psychological measurement, pp. 23-48.

36.

National Academies of Sciences and Medicine (2018) How people learn II: Learners, contexts, and cultures, Washington, DC: National Academies Press.

37.

National Assessment of Educational Progress. (2014). Technology & engineering literacy (TEL). Retrieved from https://nces.ed.gov/nationsreportcard/.

38.

National Research Council (1996) National science education standards, Washington, DC: National Academies Press.

39.

National Research Council. (2013). Next generation science standards: For states, by states. National Academies Press.

40.

Roschelle

Shechtman

Tatar

Hegedus

Hopkins

Empson

Gallagher

L. P.

(2010) Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal 47(4): 833–878.

41.

Saldaña

(2016) The coding manual for qualitative researchers, 3rd ed. Los Angeles, CA: SAGE.

42.

Shulman

L. S.

(1986) Those who understand: Knowledge growth in teaching. Educational Researcher 15(2): 4–14.

43.

Tutwiler, M. S. (2014). Trends in the salience of data collected in a multi-user virtual environment: An exploratory study. (Unpublished dissertation). Harvard Graduate School of Education.

44.

U.S. Department of Education. (2017). Reimagining the role of technology in education: 2017 National Education Technology Plan update. U.S. Department of Education. Retrieved from http://tech.ed.gov.

45.

Wouters

Van Nimwegen

Van Oostendorp

Van Der Spek

E. D.

(2013) A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology 105(2): 249.

46.

Wouters

Van Oostendorp

(2017) Instructional techniques to facilitate learning and motivation of serious games, New York, NY: Springer.