Virtual teaching assistants: A survey of a novel teaching technology

Education technology and tutoring in China

The education technology sector is huge in Asia, especially the focus on tutoring in preparation for standardized exams (Anderson, 2017). Asian parents are spending considerable amounts of their income on tutoring. Concretely speaking, there are three main categories of online AI-enabled education services in China’s market: teaching assistants, that can assist teachers on tasks such as grading homework and test papers autonomously; teaching management for support on matters such as course scheduling; and assistance for self-study and adaptive learning (Zhan, 2019).

Overall, the literature reveals that tutoring is effective, with some findings indicating that it is more beneficial to have teachers or paraprofessionals (those who might not have been ‘full’ teachers but still have experience in that realm) doing the tutoring than non-expert educators or tutors (Nickow et al., 2020).

Remote tutoring through video calls is one common way of providing education online today. A study in Italy found that tutoring program helped boost tutees’ scores not only with standardized testing, but also increased their “psychological well-being” and “aspirations and socio-emotional skills” (Carlana & La Ferrara, 2021). Additionally, tutees not only improved their scores, but tutors also scored higher on empathy assessments, showing that there must be a focus on the effects of remote tutoring on the tutors themselves.

Current solutions in tutoring fall into a few different contexts: tutors in small studio environments, complements to large online classes, and hybrid models in between the two. A study attempted to see the effects of establishing a remote design studio for students who were engaging in co-ops or real-world learning experiences (Nespoli et al., 2021), revealing that design studios for students, especially those intended to supplement real-world work experience, are not necessarily a promising opportunity for remote tutoring applications.

In the case of Massive Open Online Courses (MOOCs), there are multiple case studies of VTAs. Jill Watson helps to offload mundane and routine tasks for teachers so that they can engage more deeply with students, while simultaneously helping students focus on what they know and do not know (OnlineEducation, n.d.). Chatbots like Jill Watson are being used to improve the student learning experience and student services in general, providing 24/7 on-demand learning support for them (Teachonline.Ca, 2021). Students are paired with a “lifelong learning companion” that accompanies them to classes and helps with their homework. These virtual friends are not meant to replace the teachers, but instead support them by reducing their workloads (Furness, 2020).

Emotionally speaking, most VTAs currently exist either in the uncanny valley (in which they imperfectly resemble humans, evoking a sense of “uncanniness”) or tend to be more robotic than human. Belpaeme et al. (2017) argue that robots’ appearance and behavior affect learning outcomes, explaining how robots have a distinct social character which influences the student responses that are conducive to learning. Wang et al. (2021) suggest a “mutual theory of mind” as a theoretical framework for designing long-term human-AI interaction, again using Jill Watson as a case study. For instance, the Kruskall-Wallis tests how anthropomorphic the AI is. Automated sentiment analysis can be employed to test the positive effects that the AI’s responses have on the students' perception of a VTA (Patel et al., 2020).

Technically speaking, robot teachers are not perfect. For instance, in speech recognition and computational vision, there is the risk of inaccurate interpretation of a learner’s behavior (Belpaeme et al., 2018). Moreover, there will often exist gaps between users and developers in expectations and familiarity. Hence, it is crucial for users of chatbots to understand the social and technological limitations of VTAs, to avoid frustration as a result of a mismatch between their expectations and the product’s capabilities (Pang, 2021).

Agile development, a methodology to build software predicated on iterative development and continuous delivery of new features rather than large monolithic releases, is a process for building VTAs. Agile developers can engineer modular microservices communicating with each other to grade homework, provide student feedback, and answer questions. More recently, machine learning algorithms have become a popular approach for VTAs. Reinforcement learning for curriculum planning, natural language processing for grading homework and chatbots, and digital text-to-speech are just a few examples of the types of extant technologies.

Policy climate in China

In the summer of 2021, China’s Ministry of Education introduced the Double Reduction Policy in China (Song, 2022). The policy dictates that tutoring classes for subjects taught at school can only be organized on weekdays before 9pm, that tutoring companies cannot use foreign teaching material or employ teachers based abroad, and teach to students below the age of six (Dandan, 2021). The stated goals of the policy are to lessen the financial burden on parents who pay for-profit tutors for services that the state should provide instead, while also helping improve students’ mental health.

China’s existing policy guidelines are focused on strengthening the ability of its research institutions and empowering people to crest the technological wave rather than foregrounding ethical considerations related to AI or education. On the other hand, the country has imposed stringent rules for electronic devices which are used for intelligent education, with some specific policies requiring these products to comply to high ethical standards. There are some tensions between the positive encouragement for AI development and the strict regulations about the implementations and executions of AI, which can also reflect the huge complexity of the ethical and safety issues of AI and education. These in turn might affect the speed at which such VTA technology can be deployed whilst emphasizing the need of adopting a trial-and-error approach given the policy environment.

Ethical issues surrounding AI in education

The main ethical concerns for AI education (AIED) are data and computational approaches including data collection, how to analyze the data, whether there are biased assumptions, and transparency issues. The demographic imbalances within AI will influence VTA’s response to the students, because it is trained using posts and data from the previous semester (Eicher et al., 2018). Overlooking these issues concerning minorities may create negative emotional impacts on students, which has guided researchers to develop the system so that it can be as inclusive as possible to a broad range of people.

In-depth investigation of teaching activities in the age of AI has shown that AI teaching assistants may have better cognitive ability, which can relieve the cognition burden for teachers in terms of long-term memory, detail recognition, information storing and so on, but the emotional and creative intelligence of teachers cannot be replaced by an AI teaching assistant system. Despite this, AI teaching assistants are better than teachers in emotion recognition and emotional calculation based on facial expressions (Yang et al., 2019).

Aiken and Epstein (2000) proposed some guidelines when they first set up a series of “dimensions of human being”, including moral, aesthetic, intellectual, and other dimensions, and then argued that AIED should not weaken these human dimensions of students, and at least should be able to enhance one of them.

When creating and launching a virtual teaching assistant, there are ethical responsibilities for those people who are involved in the experiment, and in this case, the responsibility is to deal with tension between authenticity and deception (Eicher et al., 2018). When the researchers used an AI teaching assistant, they didn’t tell students that it was not a human answering their questions, but they justified this decision with three reasons: they didn’t want to influence the students’ interaction with their teachers, students were able to discuss the study at the end of the semester, and those students are likely to build similar AI agents themselves in the future, this experience helping them make more thoughtful decisions about what kind of experiences the robot they construct should offer to the users (Eicher et al., 2018).

Positioning VTAs: Users and methods

The greatest potential for VTAs lies in the menial tasks that take up teachers’ time yet could be automated by AI, but VTAs are not a viable alternative to the complexity, understanding, and empathy of human teachers who perform the monumental task of shaping the lives of young students on a daily basis. Given these limitations, we consider which actors should use VTAs, when they should be deployed during a school year, and which subjects these are best suited for.

There are strong incentives for individual students to seek out help from a VTA, given they can help increase their scores by 5-10 points on the gaokao, making them jump up a tier in higher education. Schools are in turn rewarded for improved performance by students on the gaokao (Pan et al., 2016). The incentives therefore align best for selling to schools and designing the VTA to include features relevant to both teachers and students, such as automated grading and personalized reports on student exam performance over time. According to our teacher interviewees, additional homework help, question-and-answer on the lessons given during the day, and self-study are tasks that students generally perform on their own, not with groups of other students. Products that assist students with self-study and adaptive learning have become very popular in recent years, especially up until DRP (Zhan, 2019). Furthermore, VTAs are not effective at assisting creative or collaborative tasks, so they are more well-suited to the tasks listed above. They are particularly useful in the context of review for major exams such as midterm and final tests for three reasons. In preparation for large exams, students can look back on historical reports of their performance in different quizzes and tests over the course of the semester to hone in on weak knowledge points. In this context, VTAs also help teachers grade the large amounts of exercises that students are using for review. Lastly, whilst VTAs are not a perfect solution for online education, their personalized nature, ability to engage students, and integrated use in the curriculum make them appealing as an aid to online education.

Through engagement on the online platform, students can improve information retrieval through spaced practice (Johnson, 2018). Quizzing applications such as Kahoot and Quizlet have made this type of learning possible and effective. Interviewees suggested that material should be scaffolded in such a way that learners are given extra time to go through the content at their own pace. Since the teachers do not know what is going on in the student’s brain, it is important to ensure that online students have plenty of content and time to learn and assimilate. Further, it is not enough to simply grade students and give them their score: students should be taught how to do their own peer assessment.

At the same time, not all subjects are equally well-suited for VTAs; the language arts require more open-ended thinking (Flaherty, 2020), while the sciences generally require more closed-ended questions with discrete curriculum units. Humanities subjects struggle with online teaching and learning at scale, because it is difficult to replicate in-class small group discussions online. Automated grading is better suited to students in STEM courses, especially with recent breakthroughs in NLP such as GPT-3 that have enabled improved accuracy when grading open-ended questions. Erickson et al. (2020) assessed the accuracy of automated grading in open-ended questions within mathematics and found that all models showed at least 37% improvement over the baseline of pure chance. However, grading rubrics are still scripted, and artificial intelligence-driven programs are not necessarily able to catch the nuances that are desired as of now. English language learning is a good candidate for assistance from VTAs, because most of the natural language processing advances were first made in English, so there are relatively high amounts of standardized test questions in that language. For oral learning, an AI program can also judge pronunciation of English words and the intonation of sentences, such as the speech recognition software iFLYTEK.

Pegadogical benefits of VTAs and product features

For teachers, grading is the most challenging and effort-intensive part of the work, some spending up to 4 hours every day on correcting homework. At the same time, they experience pressure from the preparation of materials and curriculum for after school classes. This is even more problematic in schools where students of many levels are packed into a single class, because teachers struggle to assign homework appropriate for students' relative levels while also teaching a standardized curriculum.

Automated grading The grading capability of VTAs for multiple choice questions is already mature, and the current progress in machine learning is directed towards natural language processing for grading open-ended questions, including short answers and fill-in-the-blank. In the context of coding, an ideal VTA solution should guide the student to the correct answer, by telling them which areas of the code are correct or not, rather than simply revealing the whole answer. At present, AI is not necessarily able to catch inefficiencies in the code, or attribute partial credit where students deserve it.

Grading analysis Beyond grading, the customized grading reports generated by VTAs can serve both teacher and student needs by providing the learner with an individualized “performance profile”. One area in NLP that bears great potential is calculating the distance of a student’s answer from a standardized set of answers created by the teacher, taking into account semantic accuracy, logical reasoning, and stylistic excellence. The VTA can take students’ answers and compare them against this baseline. In their study, Ye and Manoharan (2019) found that since all students’ work can be embedded into a continuous vector space, calculating the Euclidean distance between them is trivial. For example, a student’s essay can be used to generate multiple embeddings – in theory, a VTA could learn multiple different models for embedding an essay’s semantics, ranging from style to logical reasoning. By synthesizing insights from students’ entire learning history, these reports can help them have a bird eye’s view of the issues associated with their performance over the course of a long semester. At the same time, teachers can better understand which students need help, and with which specific topics. This is particularly valuable given that those who fall below the average level of proficiency may feel overwhelmed by homework, too intimidated to participate in class discussions, and develop a sense of inadequacy over their abilities, whilst those significantly above the class average may be bored by lessons, disinterested in class discussions, and generally develop a sense of apathy toward their education.

Customized curriculum Customizing the curriculum and homework to all students, especially when there is a large level of variation, is a potential use case for after school VTAs. Whilst also seeing how the class is doing at an aggregate level, teachers can decide which lessons to prioritize, opening up new possibilities for lesson plans by avoiding to spend redundant time reinforcing a topic when it has already been taught and learned. Waikar (2020) describes a Stanford team building an AI-based platform to assist students with the process of navigating open-ended tasks such as coding. They proposed a “Super Teaching Assistant” that provided customized feedback by generating automated reports highlighting the challenges students faced on both a personal and class-wide basis. This applies particularly well in classes where curriculum units are discretely defined, enabling the VTAs to create personalized grading reports for students based on their performance in each unit. Singla et al. (2021) surveyed the existing literature on reinforcement learning (RL) for education, a branch of machine learning which seeks to optimize decisions made in sequential environments where partial information is revealed at each step and the situation evolves at each step, perhaps within a limited set of partially observed states. They found that RL is well-suited for the problem of curriculum planning and student improvement, since it generally proceeds in discrete steps over the course of a school year. This is illustrated in the example below for a maths class (Figure 1).OPEN IN VIEWER

The customized score card could be used not only for exam results or semester-long assessments, but weekly practices or exercises. Teachers could have an input on how often the score cards are generated, depending on the frequency at which students are given exercises to practice on their weaknesses, and the speed at which they progress. Customized lessons then allow teachers to teach students of multiple levels simultaneously by allowing the VTA to design lessons with questions of varying difficulty.

Implementation

In this section, we detail some of our findings surrounding the implementation of VTAs. In particular, this entails aspects of user interface, hosting VTAs on hardware, how to involve relevant stakeholders, the system design principles and human element involved.

User interface and user experience (UI/UX) is critical to adoption by students, because they must feel comfortable and encouraged without trading off the quality and rigor of the educational content. To this end, “intelligent interaction” can be used to inject human emotions in the process of interaction between robots and humans. Sentiment analysis and facial recognition are techniques that can help the VTA grasp both the pedagogical and psychological needs of students. In 2018, the TAL Education Group released the Magic Mirror, a brilliant emotion analysis product driven by technologies such as image recognition, speech recognition, and gesture recognition (Zhan, 2019). This product can help virtual teaching assistants understand students’ emotions and learning interests in real-time. Furthermore, research shows the benefits of the “persona effect”, a well-studied phenomenon in education (Lester et al., 1997), whereby an animated tutor in a dynamic educational situation can have beneficial effects on a student’s feelings towards their learning. This study suggests that a VTA may benefit from a moderate amount of anthropomorphization, whilst ultimately ensuring a balance of engagement and distraction.

When it comes to hardware, around 74% of children under 18 own their smartphones and regularly surf the internet on their devices (Wakefield, 2021). However, The Ministry of Education ruled in 2021 that children in China are not permitted to use their mobile phone at school without written consent from their parents (Wakefield, 2021), and individual schools also have their own guidelines. Instead, schools could purchase a number of laptops or tablets in bulk, and form a separate “computer laboratory” where students can reserve a slot after school or an entire class can reserve an hour for use. Mahalli et al. (2019) found that in a “station learning” system, students can rotate onto the laptops for part of the class period and spend the rest of the period collaborating with each other or working with the human teacher The diversity of learning strategies that station rotation enables has been found to deepen student learning, improve interactivity, boost curriculum flexibility, and augment personalized learning. At the same time, this means that devices cannot be taken home for students to use in the remote learning scenario, but it remains the most effective model for hardware ownership given the current regulations in place and overall cost-effectiveness.

Administrators and technology developers may not have the know-how or experience to predict every need and use-case of VTAs, so it is critical to involve teachers, parents, and students in the implementation phase. Teachers have different needs based on the topic instructed and teaching experience. They should be allowed to raise opinions about desired features, offer suggested adjustments to existing technology, and hypothesize about the optimal rollout of the new program. Given the switch from private tutors to after-school tutoring in public schools, the doubts of parents regarding technology are understandably higher than usual. To assuage their concerns and gain their support, parents should have access to the content created by the VTAs, such as the personalized report cards and customized lessons plans. Furthermore, Wang and Xing (2018) argue that teens whose parents were more involved in their technology usage developed better digital etiquette, privacy habits, and attitudes toward social media. Finally, since students lie at the core of the VTA effort, their wellbeing should be closely monitored throughout the full deployment process. This can be achieved through informal conversations, formal surveys, and special hotlines in cases of great distress or discomfort, and later, ensuring constant feedback loops even after implementation.

Technical challenges

A core challenge facing developers of VTAs is the acquisition of sufficient, high-quality data. The large language models mentioned earlier in the report depend on an enormous amount of input-output pairs, with the input being the essay or short answer, and the output being the grade or teacher’s comments.

Since artificial intelligence relies on historical data to train its algorithms, the questions generated and recommended to students may not correspond to the topics and skills they need to succeed in the dynamic educational environment. Teaching also involves seeing the thought process of students, not just their final answer. If a VTA could help teachers grade homework, it is crucial for the VTA to help track those thought processes of students for teachers. This is particularly true for teachers for whom correcting homework manually is a cornerstone of their approach to personalizing their teaching style. Furthermore, the gaokao exam was reformed to increase the amount of reading comprehension and writing analysis questions, which are challenging for AI to grade accurately without human assistance. However, AI interpretability in education is a burgeoning field (Belinkov et al., 2020). The rise of attention in models like the Transformer and BERT allow models to pinpoint parts of sentences that are the most important for a prediction, which bear great potential for automated grading. In the meantime, if an AI product incorrectly grades a question and then requires human intervention, the time wasted negates the benefit of using a virtual teaching assistant at all.

Finally, VTAs may run into resource limitations given that most schools in China do not allow cellphones in classrooms.

Pedagogical challenges

Since every class session has new content every day, new topics, new homework, and extensions on existing topics, a VTA may have trouble achieving the granularity needed for daily tutoring–this suggests that VTAs may play a better role as occasional assistants rather than daily technology. If students receive additional homework for topics they do not understand, it will increase the burden and actually deviate from the original intention of DRP. Furthermore, automated question generation may not greatly ease the teaching burden, because teachers fear the questions might not be substantial or well written enough. Targeted homework guidance by VTAs may thus only be suitable for certain stages of the learning process, such as the review sessions near the end of a semester rather than at the outset of the semester.

From a teaching perspective, virtual teaching assistants might stifle their creativity by hindering their initiative, because a VTA restricted to a certain question bank and a style of teaching by computer programs may not complement a teacher who wants to use creative ways of teaching, such as student skits, artistic assignments, or peer interaction. Since artificial intelligence can only grade close-ended multiple choice questions and fill-in-the-blank questions, it struggles to communicate lessons involving proper relationships between people, referred to as “moral education”. This is particularly critical for students in hollowed out villages, where parents parents may live only part time at home or, in the worst case, not at all (Liu et al., 2010), in which case educators play a key role.

Finally, some parents fear that if VTAs are taking the place of human teachers, it is because schools are attempting cost-cutting measures and therefore not offering optimal quality education.

Ethical challenges

It is challenging to classify and assess how a VTA performs on the more intrinsic, relationship-focused aspects of a teacher-student relationship. This relationship possesses a friendship dimension, both instrumentally and intrinsically valuable, which is based on variables such as authenticity and non-deception. Some researchers have argued though that deception could be beneficial if its purpose is well defined, such as modeling certain growth behaviors, for example by making a robot intentionally make a mistake, and make it think to itself that it should improve next time.

At present, there are fears that we live in an outcome-driven society which will define an edtech tutoring application as being successful if it helps a child acquire a set of skills, which fails to fulfill the needed emotional connection for students. If a benefit of these robots is cost-effectiveness, and assuming that we want to improve learning and ameliorate our education systems, some researchers suggest it would be more sensible to pour more money into the field, not less.

Applications

VTAs should be implemented for tasks that are more automatable, such as grading, starting with binary question types such as true/false and multiple choice, then later expanding to fill-in-the-blank questions. As for subjects of focus, maths and science, as well as English language, are the best candidates, and we recommend that the VTA begins its operations on subjects such as math and science.

One of the most exciting parts of a VTA is its ability to collect a volume of data about both the class as a whole and individual students. To this end, VTA technology is designed to give teachers a dashboard view of the performance of their class so that they can tailor their future teachings accordingly. The VTA should also provide feedback on an individual basis so that the teacher can provide personalized help to students. Further, it should be able to automatically assist them through providing real-time feedback and detailed explanations for students based on their responses to questions. For example Aidam, Xuebajun’s education robot, can roughly imitate the perception, memory, cognition and analysis capabilities of the human brain. By establishing a knowledge base, the whole process of association, assessment and decision-making can address all questions typically asked in middle school, so that the machine can grade papers, personalize guidance and homework, and improve course preparation (Lv, 2017).

Technology development and rollout

With the goals of equity and access, a technology device rental program that allows schools, regardless of financial status, to be able to access the devices on which VTA technology runs, appears to be the best option. As with all technology, it is not expected that the release of these applications in schools will be perfectly smooth. However, student learning and well-being, as well as the well-being of teachers, is at stake with the launch of VTA technology. To this end, teachers must be involved throughout the development and testing process of its technology. The first year of launching a VTA technology is a trial run where they are continually seeking and receiving input from both students and teachers about the effectiveness of its technology. Importantly, the evaluation data on the success of the technology should be focused not only on how it helps students academic performance, but also how it impacts student mental and emotional well-being. Relatedly, teachers should continually be a part of the development process of VTA technology, such as allowing them to share their VTA-reliant apps with other teachers and also providing input to the question bank-related elements of the VTA.

Education around VTAs

While VTAs hold many exciting promises in terms of increasing the personalization of the student learning experience, they also can inspire new anxieties in students and their families. Relatedly, developers should devise materials which explains to students and families how the VTA is being used and how it can benefit their education. Importantly, it must be emphasized to teachers that they are not being replaced. Rather, it should be explained that VTA technology is making a teacher more efficient, helping them better understand their class on both a macro and individual student basis, and subsequently assists them in providing detailed feedback and personalized lesson plans to students of different abilities.