Reply to Wiebe et al.

We appreciate Eric Wiebe, Isaac Thompson, and Tara Behrend’s (2015) reactions to our article, “Moving Through MOOCs: Understanding the Progression of Users in MOOCs,” published in the December 2014 issue of Educational Researcher (Perna et al., 2014). One goal of our project was to stimulate dialogue about foundational issues pertaining to research on massive open online courses (MOOCs). Consequently, we are pleased that at least three other people read our article and took the time to write. Some of Wiebe et al.’s suggestions have virtues, but others include vulnerabilities. We offer reactions to their suggestions here.

One virtue is Wiebe et al.’s (2015) emphasis on the importance of using an appropriate theoretical framework to guide any empirical examination. The theory and related conceptual model, of course, should be aligned with a study’s research questions. Reflecting our interest in developing a systems-level perspective of MOOC users, our analyses were not data mining. Rather, they were designed to address a particular set of research questions with a relevant conceptual model that focused on events across MOOC courses. In an effort to identify categories of MOOC users, Wiebe et al. address different research questions, drawing on different theoretical perspectives and methodological approaches.

A further virtue of Wiebe et al.’s (2015) response is their reminder about “person-centered” methods. David Magnusson (2000) proposed using cluster-analytic approaches to address questions about human development so as to develop a more holistic approach to data analysis. Lars Bergman (e.g., Bergman, Magnusson & El-Khouri, 2003) developed a longitudinal clustering algorithm (SLEIPNER) that became the hallmark of the person-oriented or person-centered approach. With the more common use of methods such as latent class models (uncovering subgroups based on observed categorical variables), latent profile models (uncovering subgroups based on observed continuous variables), and growth mixture models (uncovering subgroups based on longitudinal trajectories), the methods subsumed under the rubric of person-centered methods have expanded. Each of these methods answers a very specific type of research question. As in all properly conducted research, the statistical method should not drive the research question.

A third virtue lies in Wiebe et al.’s (2015) encouragement to cluster users into groups using sophisticated statistical techniques. The newer varieties of statistical analyses for uncovering latent clusters, such as latent profile analysis (LPA), are tantalizing and potentially important, but the practical import of such categorization schemes must also be established. To understand the generalizability of the findings to other MOOCs, more information is also needed about the population and sample used to generate the LPA trajectories. The one “teacher professional development MOOC” that Wiebe et al. use as an example appears to be aimed at a particular population: K–12 teachers who might be particularly motivated to take and complete the course.

Researchers must also take into account the sometimes subtle differences among different methods. Although the manuscript includes insufficient information about the details of the variables and analyses to fully assess the appropriateness of the methods used, Wiebe et al. (2015) seemed to concentrate on users who persisted through the course. This focus could explain the recommendation to use a latent growth curve (LGC) approach. LPA uses clusters based on continuous observed variables and was originally developed as a continuous variable analog to latent class analysis (LCA). Wiebe et al. cite Bauer and Shanahan (2007), who compared LPA to a relatively simple logistic regression using seventh-grade variables to predict 11th-grade dropout. This approach is not appropriate for our research questions and data set, which focus on the many opportunities for dropout and different potential sequencing of milestones. Wiebe et al. also consider relatively few occasions of observation. This approach enabled them to consider latent transition analysis (LTA), which is typically limited to a few occasions and variables; larger models tend to result in too many parameters and an unidentified model. Growth mixture modeling (GMM) is appropriate for clustering trajectories. Dean, Bauer, and Shanahan’s (2014) discrete time event history mixture model produces clusters while allowing different milestones to be modeled in the same analysis along with appropriate covariates. This approach is beyond the aims of the Perna et al. (2014) article.

We share Wiebe et al.’s interest in developing a comprehensive picture of MOOC users’ behavior, experiences, and outcomes. We also appreciate the anecdotal information about the perceived value that instructors, students, and others assign to MOOCs. But developing a complete and comprehensive theoretically grounded understanding of MOOCs requires more extensive data. Wiebe et al.’s (2015) response understates the challenges of acquiring and making sense of complex records from MOOC courses and ignores the very limited measures that are typically available in such data for providing rigorous tests of complex psychological theories of learning. Wiebe et al. include no information on the variables used to produce cluster profiles or the distance indicators that were exploited to define a cluster or determine overlap in clusters, but the click-based indicators that were available to our research team were too limited to provide robust tests of complex theoretical constructs. MOOC researchers should encourage the collection of better data and the establishment of standards and guidelines that can guide MOOC instructors about the questions to ask MOOC students at beginning, middle, and end of the course, so that we may learn, for instance, the correlates of persistence, characteristics of engagement, and numerous learning and other benefits that might result from even sporadic contact with a MOOC.

No one study or methodological approach can address the many questions about MOOCs. We look forward to continued dialogue about the most fruitful approaches to develop a solid base of knowledge that brings scientific order to the MOOC mayhem.