Automatic Detection of Student Mental Models Based on Natural Language Student Input During Metacognitive Skill Training

This article describes the problem of detecting the student mental models, i.e. students' knowledge states, during the self-regulatory activity of prior knowledge activation in MetaTutor, an intelligent tutoring system that teaches students self-regulation skills while learning complex science topics. The article presents several approaches to automatically detecting students' mental models in MetaTutor based on paragraphs generated by students during prior knowledge activation. Three major categories of methods (content-based overlap methods; cohesion analysis of text; and tf-idf based weighted representations) were developed and combined with machine learning algorithms in order to automatically infer the underlying parameters. A detailed comparison among the methods and across all machine learning algorithms is provided. The evaluation of the proposed methods is performed by comparing the methods' predictions with human judgments on a set of 309 prior knowledge activation paragraphs collected from experiments with the MetaTutor system on college students. According to the experiments, a word-weighting method, which uses tf-idf values calculated from the corpus, combined with a Bayes Nets machine learning algorithm, offers the most accurate results. Second best performance is given by a Latent Semantic Analysis-based approach enhanced with lexical features and combined with the machine learning algorithm of Logistic Regression.