Abstract
A study was conducted using the Generalized Intelligent Framework for Tutoring (GIFT) intelligent tutoring system (ITS) software and an Army Schoolhouse course. Existing materials were provided and used to create 9 computer-based GIFT lessons that included adaptive tutoring versions of the materials. These lessons included a Pre-Test, 7 content lessons (on the areas of Detect, Engage, and Assess), and a Post-Test. Six participants completed the Pre-Test, Post-Test, and all lessons. Overall Post-Test scores were significantly higher than Pre-Test scores, and remediation also had a positive impact on Post-Test scores. This study demonstrates that completing self-paced ITS lessons prior to initial class instruction can lead to positive learning impacts prior to the start of a class.
Introduction
With recent increases in computer-based and online learning there is an opportunity to leverage intelligent tutoring system (ITS) technologies to help improve and customize the learning experience. There are many ways to configure ITSs, but in general they contain four elements (learner model, pedagogical model, domain model, and a task environment) (Corbett et al., 1997). The learner model is configured to track specific information about the learner, potentially including characteristics (e.g., motivation) and performance. The pedagogical module generally recommends a strategy for the tutoring. The domain model provides the specific content that is associated with the recommended strategy. Once tutoring material is presented to the learner, the performance will be passed to the learner model. The task environment, or tutor-user interface, is how the learner receives information and interacts with the tutor (Corbett et al., 1997; Sottilare, 2015). These elements of an ITS allow for materials to be customized and for the system to provide different tutoring based on learner interactions and performance.
There are many existing ITSs or authoring tools that can be used for many different topics. Examples include ASSISTments (Heffernan & Heffernan, 2014), CTAT (Aleven et al., 2015), and MATHia (Ritter & Fancsali, 2015). There are also generalized frameworks that can be used to create new unique tutors in the topic area that the tutor author has expertise in. The Generalized Intelligent Framework for Tutoring (GIFT) is one such tutoring framework (Sottilare et al., 2017). GIFT is open-source and has tools for authoring lessons, questions, and adaptivity (Sottilare et al., 2017). While GIFT includes the functionality to implement tutoring in applied circumstances (e.g., to provide feedback/assess performance during a serious game), it also includes more basic functionality that allows it to provide more traditional classroom-focused content in the form of slides, images, and general quizzes in an adaptive courseflow (Rowe et al., 2018; Sinatra, 2019). In an authored adaptive courseflow, GIFT can provide tutoring/remediation based on concepts that the student is getting incorrect during the assessments. The goal of this remediation is to provide other opportunities for the student to learn the material on the concept, and to continue to do so until they have passed all the concepts in the assessments of the adaptive courseflow.
There is a potential benefit to creating tutoring and lessons that individuals can complete on their own before starting an in-person course. It provides an opportunity for the students to become familiar and proficient with the material prior to class, and potentially improve their performance in the class.
To initially examine the impact of engaging with online ITS lessons prior to formal classroom based instruction, a series of GIFT-based ITS lessons were created based on existing classroom materials from a Master Gunner course in an Army Schoolhouse. It was anticipated that the students who completed all the GIFT lessons would demonstrate improved performance on a Post-Test and that receiving tutoring/remediation would potentially improve Post-Test outcomes. Specifically, it was hypothesized that the Post-Test performance would be significantly higher than the Pre-Test performance. It was also hypothesized that there would be no significant difference on the Post-Test between those who receive a high amount of remediation (tutoring) and a low amount of remediation. The reasoning behind the second hypothesis was that receiving any remediation should improve overall performance due to having the opportunity to continue interacting with the materials until proficiency is met.
Method
IRB approval was received to conduct the study. Approximately two weeks prior to the start of an in-person session of the Master Gunner course at an Army Schoolhouse, an informed consent form and a GIFT login were sent via email to the students enrolled in the class. It was not required for the students to participate, but they were provided an opportunity to do so if they wished.
Participants
A total of 11 participants completed the Pre-Test. Six of these participants completed all 7 of the lessons and the Post-Test. One participant completed some of the lessons, but not the Post-Test. One participant completed only the Post-Test, and 3 participants only completed the Pre-Test. For the purposes of the analysis, only data from the 6 participants who completed everything were used (the Pre-Test, 7 lessons, and Post-Test).
Apparatus
Participants completed the lessons on their own devices, and chose the time, day, and length of their interactions. Mobile devices, computers, and tablets were all supported ways of interacting with the lessons. Participants logged into an online GIFT website using the information they were provided.
Lessons
Three principles of the DIDEA (Detect, Identify, Decide, Engage, Assess) process were selected by instructors for the lessons to be based on, and materials in the form of existing PowerPoint presentations were provided to those who were creating the lessons in GIFT. The three overall areas covered were: Detect, Engage, and Assess. There were a total of 9 GIFT lessons created and displayed to the students (including the Pre-Test and Post-Test). The GIFT lesson titles were as follows:
1 – Pre-Test
2 – Detect: Scan and Search
3 – Detect: Acquire
4 – Detect: Locate
5 – Engage: Principles of Direct Fire and Types of Fire Commands
6 – Engage: Elements of a Fire Command
7 – Engage: Fire Command Terms
8 – Assess
9 – Post-Test
Lessons were intended to take between 15 to 30 minutes each to complete. If a participant required remediation multiple times, the lesson may have taken more time to complete than a participant that got everything correct the first time through.
The adaptive courseflow in GIFT is based on Merrill’s Component Display Theory (Merrill, 1983). It includes Rules, Examples, Recall, and Practice phases. The Examples and Practice phases are optional. The adaptive courseflow can be authored and provides tutoring based on authored identified concepts.
There were adaptive courseflows created for each of the individual lessons, and these were made up of concepts that were identified in each of the principles of DIDEA that were being covered in the specific lesson. For example the breakdown of the 2 – Detect: Scan and Search lesson is listed below:
Adaptive Courseflow 1: Individual Search
Concepts: Rapid Scan, Slow Scan, Horizontal Scan, Vertical Scan, Estimation of Upper Scan and Search Limits
Adaptive Courseflow 2: Collective Search
Concepts: Overlapping Sectors, Divided Sectors, Near-to-Far Sector
For these lessons, as part of the adaptive courseflow, PowerPoint slides were shown in the Rules phase, which was then followed by the Recall phase which included question bank quizzes based on each concept that the adaptive courseflow covered. For each concept, if the participant did not meet the author-selected number of questions correct for moving forward, the participant received remediation on the concept. After the remediation, the participants engaged in the Recall phase again (they received a random selection of questions covering the concepts from the quiz bank) and continued until they passed all concepts that the adaptive courseflow covered. Some of the lessons included multiple adaptive courseflows and others included only one.
Pre-Test and Post-Test
A Pre-Test and Post-Test were adapted from instructor-provided tests. The tests were adapted to ensure that they had an even number of questions that covered each of the concepts from the lessons. The questions were multiple choice and the Pre-Test and Post-Test contained 20 questions each. In each test there were 8 questions covering Detect, 10 questions covering Engage, and 2 questions covering Assess.
Procedure
Prior to the course start, students enrolled in the Master Gunner course received an email with an informed consent, a link to the GIFT lessons, and GIFT login information. If the student chose to participate, they were asked to complete the Pre-Test, and then lessons during the two week period, and return prior to the end of the period to complete the Post-Test, regardless of the number of lessons they had completed. This Post-Test deadline aligned with the start of the in-person class, and a reminder was also sent out.
When the participants signed into the website, they saw a screen with tiles they could select for each of the 9 GIFT lessons (including the Pre-Test and Post-Test). Each lesson was titled with a number followed by the title (as described in the Lessons section of this paper). Participants could select the lessons in any order that they wished, but were asked to start with the Pre-Test and end with the Post-Test. When the participant clicked on a lesson, it would open the lesson in their device’s browser and provide a sequence of materials followed by an assessment.
Remediation was provided based on the lesson concepts that mastery was not demonstrated on in the assessment. If the participant did not pass the concept, they received remediation material which could be in the form of: the same materials they saw previously (slides; less interactive), a different version of slides covering the same concept (less interactive), or a text prompt that asked them to explain the answer to a question, which was then followed by the expert answer that provided the information (more interactive). The system decided which remediation material to provide based on what they had seen previously and information that it knew about the participant (e.g., high or low motivation based on a questionnaire).
After successfully completing a lesson. the participant would be returned to the main GIFT content page with the tiles, and a blue checkmark would be on the lesson tile they had completed. They could then work on another lesson or log off and return later. Towards the end of the two week period, prior to the in-person class start, the participants were reminded via email to log on and complete the Post-Test. Once the class started, the participants were no longer able to access the online GIFT lesson materials.
Results
The data was extracted from GIFT and organized into an Excel spreadsheet. To determine adaptive courseflow performance and remediation performance, data for each participant was extracted separately from GIFT, and then examined to determine how many times the specific participant failed the recall phase and how many times they received remediation. This data was then entered into a main spreadsheet that included the other data that was directly exported from GIFT (e.g., total Pre-Test and Post-Test scores) and was used for analysis. Analysis was completed using the data analysis tools in Excel.
Overall Pre-Test vs. Post-Test
A one-tailed paired samples t-test conducted in Excel was run to examine the overall scores on the Pre-Test and Post-Test. The tests each were made up of 20 questions, and each had 8 Detect questions, 10 Engage questions, and 2 Assess questions. The Post-Test score (M = 16.67, SD = 2.25) was significantly higher than the Pre-Test score (M = 13, SD = 1.67), t(5) = 3.05, p = .014.
Detect, Engage, and Assess Pre-Test vs. Post-Test Performance
A series of follow up one-tailed paired samples t-tests were performed in Excel to examine if any of the specific topic areas (Detect, Engage, Assess) were showing higher performance. For these tests, the percentage correct was used for the analyses to make them easier to interpret, as each main topic area had a different number of questions covering it.
For Detect, the Post-Test percentage correct (M = 79.17%, SD = 12.91%) was higher than the Pre-Test percentage correct (M = 66.67%, SD = 20.41%), however, it was not significantly different, t(5)= -1.37, p = .115.
For Engage, the Post-Test percentage correct (M = 85%, SD = 13.78%) was significantly higher than the Pre-Test percentage correct (M = 61.67%, SD = 9.83%), t(5) = -3.5, p = .009.
For Assess, the Post-Test percentage correct (M = 91.67%, SD = 20.41%) was higher than the Pre-Test percentage correct (M = 75%, SD = 27.39%), however, it was not significantly different, t(5) = -1.58, p = .087.
High vs. Low Remediation
For each participant, the number of times that they received remediation was calculated. Based on this calculation, and a median split, the participants were broken into two separate groups: high remediation and low remediation. They were in the high remediation group if they received remediation 5 or more times (N = 3). They were in the low remediation group if they received remediation 4 times or below (N = 3).
If the participant received remediation a high number of times, it means that during the adaptive courseflows, they missed questions and did not always initially pass the recall phase. As a result of not passing a concept in the adaptive courseflow, they would have received remediation. They would not be able to move forward in the adaptive courseflow until they passed all concepts. This remediation was intended to provide the participant an opportunity to learn and understand the material. If the participant received remediation a low number of times, it indicates that they passed the recall phases of the adaptive courseflow phases more efficiently and that they may have had an initial understanding of the material based on the lesson itself.
Excel was used to conduct a two-tailed paired samples t-test examining the Post-Test scores of the Low Remediation and High Remediation participants. There were no significant differences found between the Post-Test scores of the High Remediation (M = 17.33, SD = 2.89), and Low Remediation (M = 16, SD = 1.73) participants, t (2) = -2, p = .184. Of note is that those who received more remediation actually scored higher on average than those who received less remediation, even though it was not significantly higher.
General Feedback from Instructors and Students
General feedback received from the instructors included that the current system did not provide an easy way to visualize the output of the student performance during the tutoring. For this interaction, the individual who created the GIFT lessons exported the data and arranged it in a spreadsheet in a human-readable way. The feedback received was that the lack of an easy to view gradebook interface was a current limitation of the ITS software, which should be improved in the future.
Students generally provided favorable ratings of the system including general agreement with statements that the system was easy to navigate, easy to log in to, and that it was a useful tool.
Discussion
The results were consistent with the hypotheses. Overall, Post-Test scores were significantly higher than Pre-Test scores, which suggests that learning did occur due to the interaction with the tutoring. The further analysis indicated the specific area of Engage may have benefitted the most from the tutoring. This is an interesting finding as Detect and Assess are more straightforward topic areas that require more rote learning/memorization. Whereas the area of Engage is more conceptual and a more advanced topic. It is possible that in a more applied concept area this type of tutoring may have a bigger impact on learning.
There were no significant differences between the Post-Test scores of High and Low Remediation participants. This was as expected and suggests that those who received remediation throughout their experience with GIFT were performing at the same level as those who learned the material more quickly and initially passed more of the adaptive courseflows with less tries. This suggests that the tutor had a positive impact on the learning of the participants in the High Remediation group.
Overall feedback from the students who engaged with the lessons was positive. However, this experience identified a current limitation of the GIFT software, which was mentioned by the instructors, which is that it does not currently include easy gradebook export and visualization tools. In order for GIFT to be an easy-to-use option for instructors who wish to implement an ITS prior to or during their course, it is important that the GIFT software continues to improve the way that data can be exported. This would also be beneficial to students who would be able to see their performance during tutoring if these functions were improved.
One of the overall limitations of the current study is that the sample size was small and became even smaller when examining data of the participants that completed all of the lessons. In the future it may be beneficial to send additional reminders to participants who engage with the system so that they are more likely to complete the lessons.
Overall, the results of the study suggest that including access to a self-paced ITS online prior to the start of the course can result in improvements in understanding of material as measured on a Pre-Test and Post-Test. This tutoring experience may also make it easier for students to understand the material once the class begins.
Footnotes
Acknowledgements
The research described herein has been sponsored by the U.S Army Combat Capabilities Development Command – Soldier Center. The statements and opinions expressed in this paper do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.
Partial initial results were presented in a virtual Poster during the 2022 Army University Learning Symposium to share the status of the project. There was no proceeding paper publication as a result of the previous poster, and the current paper includes additional data, analysis, and interpretation.
We would like to thank the Maneuver Center of Excellence (MCoE), including Dr. Jay Brimstin, Dr. Rory O’Brien, Mr. Rich Eggers, Mr. Steve Krivitsky, Mr. Anthony Kunigan and Mr. Travis Williams for assistance in planning/coordinating the study, and in reviewing the GIFT materials. We would also like to thank Dr. Wade Elmore (Army University) for his initial work on the study, & Drs. Joan Johnson and Jeanine DeFalco (Soldier Center) for their help writing assessment questions and testing the GIFT lessons.
