Abstract
The present study used an explanatory mixed-method design to examine the effects of the Achievement Motivation Enhancement (AME)+Cyber enrichment program and teachers’ perception of procedures and outcomes in the context of emergency remote teaching, including online and hybrid formats, during the COVID-19 pandemic. Three teachers implemented the program with 57 students in grades 9 through 12. To evaluate the program, we combined change score analysis of pretest and posttest data on academic self-perception, self-regulation, goal valuation, cyber-related interest, as well as descriptive interpretative analysis of interview data. We found the online learning format was more effective than the hybrid format for the affective and cognitive outcomes. Qualitative findings suggested pedagogical concerns and struggle with online learning due to emergency remote teaching during the COVID-19 pandemic, centered around a lack of student engagement and interaction that appropriate levels of training and practice could remedy. Implications and future research suggestions are discussed.
The coronavirus disease 2019 (COVID-19) pandemic caused drastic changes in educational systems across the world. In the wake of the COVID-19 pandemic, many K-12 schools rapidly shifted to an online mode of teaching with little time to prepare for online teaching and learning. These circumstances were unique and presented various challenges to K-12 educators. Researchers argued this shift to online learning or e-learning is unlike typical digital learning situations and should be more accurately described as crisis learning, quaranteaching (Pace et al., 2020), or emergency remote teaching (Hodges et al., 2020). Many schools were primarily focused on transferring educational content online and not necessarily on online teaching and learning methods.
Before the pandemic, we created a holistic cyber-related talent development program that combined affective support on motivation-related constructs and cognitive enrichment in cyber-related topics: AME+Cyber (Achievement Motivation Enhancement in Cyber-related topics). We designed the AME+Cyber program to support secondary students’ interest in cyber-related topics and prepare them to pursue cybersecurity and digital forensics careers. The global cybersecurity workforce shortage is projected to reach approximately 1.8 million unfulfilled positions by 2022 (Frost & Sullivan, 2017). Therefore, investing in talent development in this area is becoming urgent (Frost & Sullivan, 2017).
The original intent of this study was to pilot the AME+Cyber program at a secondary school and evaluate its effectiveness. However, when general education practices moved online, so did affective support and cognitive enrichment for students with gifts and talents. So, although initially designed for in-person delivery, AME+Cyber was implemented in an emergency remote teaching format. Initially, students participated in learning through an online synchronous form from home. Later, students had the option to participate in online synchronous learning from school. Eventually, AME+Cyber was also implemented in a hybrid format, some students participated in person and some students participated virtually. Given the unique context in which the AME+Cyber program was moved online, this article evaluates the pilot implementation of the AME+Cyber during emergency remote teaching. Specifically, we present an analysis of educators’ perceptions of the online delivery of affective support and cognitive enrichment during crisis.
Literature review
Online learning generally refers to learning experiences that are accessed via some technology (Benson, 2002; Carliner, 2004). In this study, we use the term online learning to describe the process of teaching and learning that happens synchronously online through virtual platforms. We use the term emergency remote teaching to indicate “a temporary shift of instructional delivery to … fully remote teaching solutions for instruction or education that would otherwise be delivered face-to-face or as blended or hybrid courses and that will return to that format once the crisis or emergency has abated” (Hodges et al., 2020, p. 6). An essential difference between the two is online learning is typically planned and designed for online delivery, whereas emergency remote teaching is an urgent and temporary shift.
The use of digital technology or information and communication technology (ICT) has continued to grow rapidly across schools in the United States. In 2018, an average of 60% of high-school teachers reported frequently using digital technology for classwork (OECD, 2019). North America is also leading the widespread adoption of digital learning management systems (LMS) in education (Anderson, 2017). An LMS enables information exchange between educators, students, and parents and typically hosts homework assignments, calendars, grades, messages, and school documents, for example. In general, LMS are well-equipped to support online learning. Some, such as Google Classroom, even have video conference extensions built-in. This widespread use of LMS allowed many schools to quickly transition to emergency remote teaching when the COVID-19 pandemic required it (Lepp et al., 2021). However, although the use of digital technology in education is not new, the use of online learning in a K-12 setting was rare until COVID-19 forced a transition from face-to-face learning to emergency remote teaching.
Generally, proponents of online learning and digital technology in education have argued that there are significant learning benefits and opportunities from using these resources. However, instructional strategies and pedagogical decisions around the optimum use of technology to support learning are more important than the actual technological medium of delivery (Ally, 2008; Clark, 2001; Hew and Cheung, 2013). For example, Hew and Cheung, 2013 conducted a review of 27 empirical studies on the use of Web 2.0 technologies, which is the use of websites and applications allowing the creation and sharing of information and material online, such as social media, blogs, and online virtual environments. They found Web 2.0 technologies’ effects on student learning were fairly weak across the studies, yet generally positive. Hew and Cheung, 2013 argued that overall positive results were likely due to how teachers used the technologies. They concluded a constructionist pedagogy, supported by online activities, was more important than the simple presence of technology for student learning. Similarly, Kimmons (2015) found the adoption of online learning systems did not significantly affect student achievement. Kimmons (2015) argued merely adopting online learning systems is not sufficient. “… educators should continue to proactively assert their place in society as pedagogical professionals to combat steady calls for more and more technocentric solutions to persistent problems” (p. 389). Finally, Comi et al., 2017 evaluated the effectiveness of ICT in education. They found the effectiveness of ICT in school depended on the actual use and teachers’ ability to integrate ICT into their teaching. Thus, across various types of technology, results indicated the benefits of online learning depended on instructional strategies more so than on the technology alone.
To achieve effective online learning, teachers need to understand how to use technology to support meaningful and deep-level learning. Researchers have shown although teachers typically use various ICT in their classrooms, they do not use it as a pedagogical tool to improve student learning (Amhag et al., 2019). Amhag et al. (2019) found one in five teacher educators reported low ICT competence, and 26.3% reported low computer self-efficacy regarding ICT use for teaching and learning. It is problematic that teacher educators responsible for educating pre-service teachers lack training and skills about using technology in education. Therefore, it is unsurprising that multiple researchers have found a clear need for training and support among teachers to use technology in teaching (Amhag et al., 2019; Gil-Flores et al., 2017; Koehler et al., 2013; Louws et al., 2017; Tondeur et al., 2016). Researchers found limited time to become familiar with ICT (Almekhlafi & Almeqdadi, 2010; Ungar & Baruch, 2016), as well as a lack of knowledge and skills about using ICT to support learning (Koehler et al., 2013), counteracted successful ICT integration. On the other hand, training focusing on the pedagogical integration and technical support of ICT, which is directly relevant to teachers' classroom practice, promotes the successful use of ICT for learning (Amhag et al., 2019; Forbes & Koo, 2015; Ungar & Baruch, 2016). Given that teachers may lack the training necessary to successfully engage students in online learning and the urgency with which schools transitioned from face-to-face to online learning during the pandemic, teachers may have been left unprepared.
Online learning has several benefits, particularly for addressing the needs of students with gifts and talents. Researchers have shown students with gifts and talents enroll in online classes because of specific interests, enrichment or acceleration opportunities, and unavailability of in-person classes in their area (Olszewski-Kubilius & Lee, 2004). Online learning can make learning opportunities not offered in local schools or communities more accessible (Cavanaugh et al., 2008; Thomson, 2010) and allows for individualized, student-centered instruction (Potts, 2018; Thomson, 2010). Online learning can also make homogeneous grouping more logistically feasible by offering opportunities for grouping across schools and creating access to out-of-school enrichment programs. Based on data from 28 instructors and 65 students (grades 3–12) who participated in an online learning program, Thomson (2010) concluded both students and instructors had positive experiences with online learning. The online format was particularly useful for differentiated, individualized, and self-directed learning (Thomson, 2010). Similarly, Potts (2018) conducted a case study of five 11- and 12-year-olds who participated in a virtual classroom and found students experienced little difference between the virtual classroom and an in-person one. However, Potts’ (2018) participants discussed several challenges of virtual classrooms besides the benefits.
Although online learning can make learning more accessible, it can also be limiting as technical difficulties and a shortage of digital technology may challenge the online learning experience. Nineteen percent of U.S. school principals reported a shortage or inadequacy of digital technology for instructions had hindered the delivery of quality instruction in their schools (OECD, 2019). Researchers also reported the disuse of available technology and mistakes on the part of educators, parents, and students might affect students' online learning experiences (Potts, 2018; Thomson, 2010). In addition to technical difficulties and shortages, researchers have also found a lack of interaction in online learning formats (Potts, 2018; van der Spoel et al., 2020). This is particularly problematic because a large body of research suggests a critical need to encourage interaction among students and teachers in online learning contexts (Amhag et al., 2019; Kuo et al., 2014).
Finally, as with online learning in general, the rapid and unexpected move from face-to-face to emergency remote teaching presented some benefits and challenges. Although several researchers have positively evaluated various forms of online delivery of enrichment for students with gifts and talents, few researchers have explored the perceived experience of synchronous online delivery of an enrichment program minimally adapted from a face-to-face context. Therefore, teachers’ perceptions of implementing online cognitive and affective enrichment during the COVID-19 pandemic are timely and should be explored.
Present study
The present study used an explanatory mixed-method design to examine the online and hybrid AME+Cyber program’s effectiveness and teachers’ perception of emergency remote teaching. Specifically, the following research questions guided our inquiry: 1. To what extent did students’ academic self-perceptions, goal valuation, and self-regulation change through participation in the AME+Cyber program? 2. To what extent did students’ cyber-related interests change through participation in the AME+Cyber program? 3. How did educators’ perceive the procedures and outcomes in the context of emergency remote teaching?
The present study was a pilot evaluation of the AME+Cyber curriculum. Further, the present study was conducted in unique circumstances, including emergency remote teaching during the COVID-19 pandemic. Therefore, online delivery was not the original intent, nor was the curriculum initially designed for online delivery.
Methods
We used a mixed-methods approach, which involves collecting and analyzing quantitative and qualitative data in a single study (Creswell et al., 2003; Tashakkori & Creswell, 2007). Specifically, we used an explanatory sequential design (Creswell and Plano Clark, 2017). The quantitative data were collected and analyzed first, followed by the qualitative data collection and analysis, as shown in Figure 1 (DeCuir-Gunby & Schutz, 2017). Mixed methods study design.
An explanatory sequential mixed-methods study enabled us to further investigate and explain the quantitative data through qualitative interviews (Creswell and Plano Clark, 2017; DeCuir-Gunby and Schutz, 2017).
Setting
This study was conducted during the 2020–2021 school year in a student-centered, STEM-focused public charter high school (grades 9 through 12) in the Midwestern United States. The school served 529 students at the time of the research, of whom approximately 54% qualified for federal meal subsidy. The ethnic breakdown of the student population was 4% Asian, 36% Black, 17% Hispanic, 10% Multiracial, and 33% White. The school’s minoritized enrollment of 67% is significantly more than the state’s average for public schools of 32%.
We conducted this study during the COVID-19 pandemic when schools moved to virtual and hybrid delivery modes, emergency remote teaching, with little notice and limited planning opportunities. At the time of data collection in the Spring of 2021, students engaged in learning through one of three formats: (1) 2-day hybrid (2 days in-person learning and 3 days e-learning), (2) 4-day hybrid (4 days in-person learning and 1-day e-learning), and (3) entirely e-learning (no in-person learning). There were two groups of 2-day hybrid students. One group attended school in person on Monday and Tuesday. The second group of students attended school in person on Thursday and Friday
The curriculum was implemented in two rounds. The first implementation took place during the early spring semester (mid-January–mid-February), and there were two sessions offered Monday/Tuesday and Thursday/Friday for 5 weeks. The second implementation took place late spring (February–April) and took place Mondays through Thursdays for 5 weeks. The AME+Cyber courses were offered simultaneously. For example, the AME curriculum was offered on Monday and the Cyber curriculum was offered Tuesday.
Participants
Student participants had the opportunity to select the AME+Cyber program for their Spring 2021 schedule. In December 2020, all students were presented with a choice menu of enrichment courses. Students reviewed 30-second videos on enrichment courses of interest and then ranked their top three choices. As a note, not every student was enrolled in their first choice, and students who did not participate in the ranking were randomly assigned a course.
Demographics.
AA = African American; TMR = two or more races.
Group profile
The AME+Cyber program was offered during two implementation rounds at the intervention site. Implementation round one included two outside teachers, Eva and Evalyn (pseudonyms), who were hired to implement the curriculum, and instruction was conducted entirely online. Both Eva and Evalyn had no prior affiliation with the intervention site before this study. Eva, who taught the Monday/Tuesday section, identifies as White and female. She has a Bachelor of Science in Psychology and a Master’s of Science in Gifted Education. Eva had limited K-12 teaching experience and no online teaching experience. Evalyn, who taught the Thursday/Friday section, identified as Asian and female. She had a Bachelor’s of Science in Statistics and a Master’s of Science in Gifted Education and 7 years of teaching experience, with 5 years in gifted education. Evalyn had no online teaching experience. Both Eva and Evalyn had experience coordinating a gifted education program.
The second program implementation round was taught in a hybrid format, as discussed in the Settings section above. Nineteen students participated in the second implementation round. Students were instructed by a teacher from the intervention site, Nick (pseudonym). He had a bachelor’s degree in American history and a transition-to-teaching degree. Nick had 2 years of teaching experience, specifically at the intervention site, and taught social studies and enrichment courses on gaming and computer-related topics.
In addition to the three teachers, one administrator, Wayne (pseudonym), also participated in our study. He oversaw the AME+Cyber implementation at the intervention site. Wayne identified as White and male with 15 years of teaching experience and four years of experience at the intervention site.
Measures
School attitudes assessment survey-revised
The pre-/post-assessment for the AME curriculum included a modified version of the School Attitudes Assessment Survey-Revised (SAAS-R; McCoach, 2002). The SAAS-R is a self-report survey comprised of 35 questions across five subscales: Academic Self-Perception, Attitudes Toward Teachers, Attitudes Toward School, Motivation/Self-Regulation, and Goal Valuation (McCoach, 2002). The present study included three of the five subscales on the AME pre-/post-assessment: Academic Self-Perception (α = .95), Motivation/Self-regulation (α = .90), and Goal Valuation (α = .96) Academic Self-Perception, which refers to students’ perceptions about their academic skills students interests in school, is comprised of seven questions, such as, “I am intelligent” (McCoach, 2002). Goal Valuation consisted of six questions, for example, “I want to get good grades in school.” Goal Valuation refers to students’ beliefs about the importance of tasks in relation to their goals and their interest in tasks. Motivation/Self-Regulation consisted of 10 questions, such as “I work hard at school.” Motivation/Self-Regulation refers to students’ knowledge of cognitive and metacognitive strategies and their motivation to use these strategies to regulate cognition and effort. Participants responded using a 7-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (7).
Cyber survey
The Cyber pre-/post-assessment objective was to assess students’ cyber-related interests. We created three questions to capture students’ interests. One question pertained to a future career and stated, “I want to have a job that involves computer and information technology.”
One question asked about continuing to learn about the topic, “I want to learn more about computer and information technology.” The final question asked about taking courses on the topic, (“I plan on taking more computer and information technology courses even when I don’t have to”). Participants responded using a 5-point Likert scale ranging from Strongly Agree (1) to Strongly Disagree (5).
Observations
Teachers were asked to video record each session of the AME+Cyber program. The first and second authors then reviewed those recordings to capture fidelity of implementation and triangulate the themes derived from the interview data.
Teacher interviews
We created a semi-structured open-ended interview protocol for the teachers and administrator. This protocol included three sections: perception of the procedures, perception of the outcomes, and fidelity of implementation. For the first implementation round, we developed a protocol following preliminary analysis of the quantitative findings to help explain our quantitative results, per the explanatory sequential mixed methods research design. We then updated the protocol in between each interview (Eva and Evalyn) to document follow-up questions. Following a preliminary analysis of round one interview data and observation data from implementation round two, we then updated the round two interview protocol to collect additional data that may help clarify themes. Finally, we revised the protocol once more before conducting the administrator interview. Example questions included, “What, if any, challenges did you face while implementing/overseeing the implementation of this curriculum?” to capture perceptions of the procedures; “In your opinion, has the curriculum helped improve students' interest in cybercrime and security related topics/careers?” to capture perceptions of outcomes; and “How did you prepare for the sessions?” to capture fidelity of implementation.
Procedures
AME+Cyber intervention
AME+Cyber Training
Three teachers individually participated in a one-hour training session, conducted virtually. In this training session, the first and second authors discussed data collection procedures, provided training on listening and responding skills to prepare teachers to engage students in the affective AME curriculum, and provided an overview of both curricula. Teachers also received detailed lesson plans that included relevant background information for each AME and Cyber-related session. Two teachers (Eva and Evalyn) met with the second author weekly to review the Cyber-related curriculum in-depth. During the training and follow-up sessions, we briefly addressed suggestions and adaptations to improve online delivery. Nick opted out of additional support and contacted us with questions as needed.
AME Curriculum
The Achievement Motivation Enhancement (AME) Curriculum (Desmet, 2020) was designed to help increase achievement and achievement motivation through small group discussions. The AME curriculum involved guided discussion sessions on six topics: (1) positive self-concept; (2) goal valuation and goal orientation; (3) self-monitoring; (4) resilience; (5) self-directed learning; and (6) positive self-talk. Each session typically consistsed of a small group discussion, a worksheet activity, and achievement coaching. Topic one focused on positive aspects of students’ self-concepts by having them identify their strengths and talents. Topic two centered on goal valuation, learning or mastery goal orientation, and goal setting strategies. Topic three focused on self-monitoring strategies. Topic four involved a discussion of resilience. Topic five involved reflection on learning patterns and exploring self-directed learning strategies. Finally, topic six involved positive self-talk practice. The AME curriculum was initially designed for in-person delivery. When it became clear that an in-person implementation would not be possible due to COVID-19, minimal changes were made to adapt the curriculum for emergency remote teaching, including online and hybrid delivery.
Cyber-Related Curriculum
The Cyber-Related Curriculum (Cyber) was designed to help increase Internet safety awareness and foster an interest in cyber-related careers. The Cyber curriculum involved course content on five areas which included the (1) Cyber World; (2) Introduction to the Cyber Crime (Parts 1 and 2); (3) The Digital Crime Scene; (4) Digital Citizenship; and (5) Cyber-related Careers. Each session included lecture content, interactive discussion, and activities. The Cyber World session provided an overview of recent advancements in technology, how the Internet changed communications, and the implication of new technologies. The Introduction to Cyber Crime (Parts 1 and 2) provided an overview of the history and prevalence of computer crime, reviewing the ways cyberspace and computers can be used to commit a crime. The Digital Crime Scene discussed the difference between a traditional crime scene and a digital crime scene, types of digital evidence, and the digital forensic process for obtaining evidence. Digital Citizenship focused on being safe online. The Cyber-related careers focused on the shortage of cyber-related careers and the specific types of cyber-related careers in private and public sectors. When it was determined the curriculum would be implemented online, a few adaptions were made. Specifically, the original curriculum included a field trip to a Digital Forensics Lab. Due to the COVID-19 pandemic, this was not possible, and as a result, the interactive field trip content was put into a PowerPoint presentation.
Data collection procedures
The present study consisted of multiple strands of data for both the students and the educators involved. For the students, data included a pre-/post-survey for both AME and Cyber. Both were provided electronically via Google Forms. The pre-survey was provided to each student via email and through the Google Classroom chatbox for implementation round one. For implementation round two, students were only offered the survey via the Google Classroom chatbox. Students were given time at the start of the first AME and Cyber sessions, which ran simultaneously, to complete the survey in both implementation rounds. The same procedures were followed for the post-survey, which was provided on the last AME and Cyber sessions, respectively. Before analyses, the authors linked the pre-/post-surveys and provided each participant a number ID to de-identify the data.
Each teacher video recorded their session to allow the research team to observe the implementation. After each session, the teacher saved and shared the class recording and chatbox. The first and second authors reviewed session recordings. We recorded any deviations from the curriculum, and we took field notes to compare and contrast our observations with teachers’ perceptions. The teachers and administrator also participated in an exit interview following the completion of the intervention. The interviews were video recorded, de-identified, and transcribed for analysis.
We received written permission from the intervention site’s administration to conduct this study. All data were collected as part of regular classroom activities so no signed consent was obtained from individual participants prior to our data collection. Educators verbally consented to participate in interviews which were video recorded and transcribed for analysis. The Institutional Review Board approved the study (IRB-2020-87), and all participants were treated in accordance with the ethical standards set forth by the American Psychological Association.
Data analysis
Per the explanatory sequential research design, the quantitative data were collected and analyzed first. A change score analysis of pretest-posttest data was used to evaluate whether changes in academic self-perceptions, goal valuation, self-regulation, and cyber-related interests existed after participating in the AME+Cyber program. Cohen’s d was calculated to determine the effect size. The quantitative results were then used to help create the interview protocols for the qualitative data collection. Qualitative results were analyzed using a descriptive-interpretive analysis (Elliott & Timulak, 2021) of interview data to capture the educators’ experiences with the online delivery of the AME+Cyber curriculum. We engaged in member checking by sending a copy of the final manuscript to our participants. All participants reviewed the manuscript and agreed with the representation and interpretation of their data in this study.
The AME and Cyber pre-/post-surveys were administered to 33 students in implementation round one and 19 students in the second round. After reviewing the data, we determined 11 individuals in round one and ten individuals in round two completed both the AME pre-/post-survey. This is a participation rate of 33% and 52%, respectively. For the Cyber pre-/post-survey, nine individuals in round one and 12 individuals in round two completed both the pre-/post-survey. This is a participation rate of 27% and 63%, respectively.
Results
The results for each research question are presented next.
To what extent did students’ academic self-perceptions, goal valuation, and self-regulation change through participation in the AME+Cyber program?
AME Change Analysis.
Note. M (SD); Participants responded using a 7-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (7).
To what extent did students’ cyber-related interests change through participation in the AME+Cyber program?
Cyber-Related Interests Change Analysis.
Note. M (SD); Participants responded using a 5-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5).
How did educators’ perceive the procedures and outcomes in the context of emergency remote teaching?
Qualitative exit interviews were analyzed to determine three teachers and one administrator’s perceptions of the procedures and outcomes in the context of emergency remote teaching. All three teachers (Nick, Eva, and Evalyn) discussed a lack of student engagement from students participating online. Overall, teachers attributed this lack of engagement to two things: students’ disuse of technology and pedagogical concerns around online instruction.
Lack of student engagement due to disuse of technology
Nick, Eva, and Evalyn agreed that disuse of technology by students, which included not using a camera during class, lack of audio responses through microphones, and limited use of the chatbox, were the biggest challenge to assessing student engagement. For example, Eva stated, “my students did not want to use video or microphones. It was like pulling teeth.” The disuse of technology also created an uncertainty around students’ presence in online classrooms. Teachers said they could not be confident of who was actually present because a majority of students had their cameras turned off and did not respond verbally or through the chatbox. For example, observations showed that on multiple occasions, teachers would call on students with no response. Evalyn described the experience as, “It’s just kind of hollering at the air, hoping that there’s a response.” Our observations also showed that during the final week of the second implementation round, one student mentioned they were packing a bag and getting ready to leave, so they could not respond timely to the teacher’s inquiries. So teachers’ suspicions that students may not always be paying attention proved right in that case. Evalyn added she had to handle the students' lack of engagement with “a lot of patience,” she said: I have to give them some waiting period and continue to frequently ask if you’re still with me raise your hand and if people don’t raise their hand call their names out. There were times where I fully knew [students] were watching TV or something …. By the end of the third or fourth week there were probably four or five students of which I knew they just logged on and would not reply so …. I didn’t call them out [anymore] or ask them to participate, because I knew they most likely would not.
Overall, each teacher relied on a handful of students to keep the conversation going. This may also explain why our response rate on the surveys was low.
Eva and Nick postulated students might not want to engage in an online environment due to privacy concerns. Due to the online nature and Google classroom format, students could record the session with their computer, a mobile device, or screenshot the chatbox. Eva stated, “[Students] were keenly aware that, whatever they typed into the chatbox, somebody could just screenshot.” Additionally, privacy concerns around homelife were a concern, which was exemplified in the following quote from Nick: “I know a lot of the kids might be embarrassed about where they are and everything.” Further, the teacher felt reluctant to force students’ use of cameras in an effort to not single out students who might be uncomfortable. This concern and perspective was also shared by the administrator, Wayne, who discussed the school’s expectations for students to use their cameras. The administrator stated: “We pre-set expectations, and then we encountered the reality of students, and their vulnerable selves broadcasting live from their home and we had to roll with … what we could.” The administrator recognized this as a barrier to “community connection and belonging.” Due to the challenges associated with the online delivery, all three teachers agreed the curriculum would have been easier to implement and more effective if implemented in person.
Lack of student engagement due to pedagogical concerns
The teachers agreed the lack of student engagement stemmed partly from pedagogical concerns around online teaching and learning. Teachers discussed a lack of connection. For example, Evelyn said, “There were definitely times where it was harder for me to connect with [my students] and further discussion.” Specifically, teachers struggled with establishing a rapport and engaging students in discussion due to limited training and support about online teaching. Techniques that may work in person did not translate well to the online environment. Nick explained: It’s a lot different if I’m in the room with you, then I can look at someone, and they know I’m looking at them. It’s just, it’s much easier. I will embrace the awkwardness, and I’ll just sit there, quiet, until someone feels obligated to talk. And when you’re not in the room, with me, it’s really easy to just sit there forever and not worry about it, where it’ll get really awkward in a classroom full of 15 people with no one saying anything. Someone will eventually speak up, right.
Struggling with establishing interaction and engagement resulted in teachers suggesting the program would have been more effective in person: I think that as a teacher it is going to be a lot more helpful to be in class with [the students]. So, you can gauge facial expressions and see who is paying attention. Those kinds of things are irreplaceable when trying to get at some emotions connected to motivation. So, I think it would go over much better, I think it would be more effective. (Eva)
Nick, who taught in a hybrid format, discussed additional challenges with student engagement due to this hybrid set-up: It’s just I can’t fully interact with either, so I have to always stay somewhere close to my computer and kind of ignore the people in class, or if I minded people on the computer, I’m kind of ignoring the people in front of me.
Overall, teachers struggled with online learning because of pedagogical challenges due to a lack of experience and training. Wayne concluded, “This is actually not a technology problem, this is a pedagogical challenge. How to design and execute lessons or projects or activities that can engage both virtual and in-person [students].” The teachers and administrator agreed these pedagogical challenges could have been remedied through additional training and support.
Discussion
This study was conducted during the COVID-19 pandemic, which required a rapid shift from face-to-face learning to online learning with minimal time to prepare. Therefore, our results should be interpreted in light of these unique circumstances. The overarching research question of this study aimed at evaluating the AME+Cyber program through online delivery in the context of emergency remote teaching. However, it is essential to note that findings from this study cannot be used to decide whether or not to implement online learning. Instead, the results may help provide insight regarding how to support teachers in online learning environments. Overall, the most prominent findings of this study pointed at pedagogical concerns and struggles with online learning, centered around a lack of student engagement and interaction that appropriate levels of training and practice could remedy.
Although teachers discussed student engagement concerns and emphasized the affective curriculum would have been easier and more effective in an in-person format, quantitative results for implementation round one (i.e., online format) showed students’ academic self-perceptions and self-regulation improved after participating in the AME+Cyber program. These findings extend the literature on the AME curriculum’s effectiveness (Desmet and Pereira, 2021) and indicate that students may benefit from engaging in this curriculum and similar affective supports even when conducted online. We also found medium-to-large effects for students’ cyber-related interests. Thus providing preliminary support for the effectiveness of this curriculum when conducted online.
Contrary to our expectations, the hybrid format was less effective than the online format. We found no to small changes in students' academic self-perceptions, goal valuation, self-regulation, and one large negative change for cyber-related interests. These findings may be due to the context of emergency remote teaching or quaranteaching and the specific type of hybrid learning that was implemented rather than hybrid online learning in general. Typically, hybrid learning involves a blended format in which some of the learning happens online and some of the learning happens in-person. However, in the context of emergency remote teaching the school implemented a hybrid teaching format that involved simultaneously teaching to a group of students in-person and a group of students joining online. Our qualitative date indicated that this brought on unique struggles to engage both groups of students and limited the teacher’s ability to properly interact with students. Furthermore, students took turns joining class in-person and online. So, none of the hybrid students joined class in person all the time. The research on this type of hybrid learning is limited. However, researchers have found limitations of online learning during the pandemic, which involved inefficiency and difficulty in maintaining academic integrity (Mukhtar et al., 2020) that may be at play here as well. Further, Hodges et al. (2020) stated, “The rapid approach necessary for emergency remote teaching may diminish the quality of the courses” (p. 8). It is possible that the quality of the course is further diminished when used in this simultaneous online/in-person hybrid format. Emergency remote teaching is not like typical online or hybrid learning in which learning environments are carefully designed for online delivery. Finally, student motivation and engagement are critical to academic success in any environment. Researchers have shown motivating students during emergency remote teaching was a major struggle (Reich, 2020). Our qualitative findings support this assertion. Thus the delivery format was a possible confounding variable that may have affected our evaluation of the AME+Cyber program. Overall, our findings suggest that with appropriate training and necessary adaptions to the curriculum, the AME+Cyber program could be effectively implemented online.
Our qualitative findings illuminate several challenges teachers experienced while implementing the AME+Cyber program in online and hybrid formats in the context of emergency remote teaching. Our most prominent finding was teachers’ perceived struggles with student interaction due to disuse of technology such as not turning cameras on, limited verbal interaction, and limited signs of students’ engagement. Researchers have found online participation drives online learning (Fredericksen et al., 2000; Hrastinski, 2009). Online learner participation positively affects students’ learning (Alavi & Dufner, 2005) and achievement (Johnson et al., 2000). Fredericksen et al., 2000 found interactions with teachers, levels of participation, and interactions with peers were the most significant predictors of learning effectiveness. Therefore, the observed and perceived lack of interaction between teachers and students may explain why we found little to no changes in students’ academic self-perceptions, goal valuation, self-regulation, or cyber-related interest. Our findings thus support the assertion of online participation driving online learning.
This lack of interaction and other pedagogical struggles teachers experienced, such as struggling with online classroom management, could stem from a lack of training and preparation for online learning. None of the teachers had previous experiences with online learning, and they received minimal support during the transition from face-to-face to online learning. Previous research shows common struggles in online learning environments, such as a lack of motivation, community, and engagement, can be addressed by providing teachers with proper practice and training (Andrade, 2015). Cherif et al. (2019) and Sharp and Sharp (2016) attributed successful online learning to instructor preparation to teach in online environments. However, with the rapid move to online learning during the pandemic, time constraints prohibited appropriate training for teachers to create high-quality online learning environments. Thus, our findings support the need for adequate and high-quality training and practice when implanting online learning.
Overall, empirical studies on the effects of emergency remote teaching in high-school settings are limited. Our findings contribute to the growing body of research in this area and support several implications and suggestions for future research discussed below.
Limitations
There are several limitations associated with the present study. First, the research involves the unique and changing circumstances in which data were collected. While the original intent was to pilot the AME+Cyber intervention in person, it was minimally adapted for online delivery to accommodate emergency remote teaching during the COVID-19 pandemic. Center for Disease Control and Prevention (CDC) guidelines for schools changed several times while the intervention was implemented. Therefore, two groups of students initially participated in online learning from home and later attended school while participating in the AME+Cyber classes online. A third group participated in a hybrid format. Although this could be seen as a limitation, it also provided a unique opportunity to capture teachers’ perceptions of online learning within the context of emergency remote teaching and other forms of “quaranteaching.” Nevertheless, readers should keep in mind that this study’s findings are not generalizable to traditional online and hybrid learning settings.
A second limitation has to do with the teachers’ backgrounds. Two of the three teachers, Eva and Evalyn, had a master’s degree in gifted education but limited teaching experience in general and no online teaching experience. The third teacher, Nick, did not have formal gifted education training. None of the teachers had experience delivering affective curriculum or cyber-related curriculum, although Nick had a strong interest and had previously supervised enrichment projects on the topic. Although the teachers received training on the skills required to implement both curricula successfully, it is possible more experienced teachers would have delivered the curriculum more effectively.
A third limitation involves data collection procedures. We used self-report data from students to measure the effectiveness of the AME+Cyber program. Self-report data may not be reliable and valid from youth participants (Teye & Peaslee, 2015). Additionally, administrators at the school site informed us between round one and round two students expressed survey fatigue due to the amount of data being collected at the intervention site. Emergency remote teaching created an interest among several researchers, administrators, and stakeholders. Therefore, students frequently completed surveys for various purposes, not just for our research procedures. We used attention checks in our surveys to improve the reliability of our data. However, the survey fatigue may explain our low response rates.
A fourth limitation was the low response rate. The response rates ranged from 27% to 63% across the various surveys. Therefore, the low response rate may have resulted in Type II error and affected our statistical power to detect small effects.
Finally, other limitations of this study are those associated with one group pretest-posttest designs. For example, instrument reactivity, participant mortality, maturation, and history. Therefore, it is hard to determine the validity of our quantitative findings. Additional research with a more robust design is necessary. The results of this study cannot be generalized beyond this study. However, the findings of this study do highlight important experiences with online learning that could affect curricular decisions and interventions aimed at online learning and particularly the delivery of affective and cognitive enrichment.
Implications
Online learning promotes student-centered learning and provides opportunities for access to education that may not be generally available in person for some students. It is important to note that teachers had minimal time and training in online learning when the pandemic happened. Therefore, the most important implication of this study is that teachers need sufficient training and support to implement high-quality and effective online learning with students. The fundamental problems with online learning identified in our study were lack of teacher–student interaction, inability to make a reliable assessment of understanding, lack of knowledge about online learning, and lack of student engagement. On the other hand, teachers were optimistic that online learning could be successful with the necessary training, support, and adaptions to the curriculum. Therefore, teachers delivering online learning should regularly attend professional development, learn about the LMS and online and digital tools for learning, and adopt a student-centered teaching philosophy that focuses on the optimal integration of digital technology.
Suggestions for future research
Researchers should explore students’ experiences and perceptions of online learning during the COVID-19 pandemic in general and specifically about affective and cognitive support. Students’ perceptions of online learning during a pandemic may significantly differ from their perceptions of online learning in regular times. Comparative and experimental research on the effectiveness of AME+Cyber through online delivery is needed. Future research might also examine students’ perceptions of different hybrid and online learning environments. Researchers may also be interested in exploring online interactions in the context of affective supports for students. All teachers agreed the affective curriculum would have been conducted more effectively in person. So, exploring ways to improve online interaction and establishing psychological safety and trust in an online environment may be of interest.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Purdue University.
