Informing Pedagogical Action

Abstract

This article considers the developing field of learning analytics and argues that to move from small-scale practice to broad scale applicability, there is a need to establish a contextual framework that helps teachers interpret the information that analytics provides. The article presents learning design as a form of documentation of pedagogical intent that can provide the context for making sense of diverse sets of analytic data. We investigate one example of learning design to explore how broad categories of analytics—which we call checkpoint and process analytics—can inform the interpretation of outcomes from a learning design and facilitate pedagogical action.

Keywords

learning analytics learning design pedagogical intent

This article examines two relatively new concepts within education, learning analytics, that is, the collection, analysis, and reporting of data associated with student learning behavior, and learning design, that is, the documented design and sequencing of teaching practice, and how together these may serve to improve understanding and evaluation of teaching intent and learner activity. Learning analytics offers a passive method of gathering information on how learners are interacting with learning resources, each other, and their teachers. Unlike traditional surveys or focus groups, which rely on participants both opting to provide feedback and accurately remembering and reporting past events, learning analytics captures data on specific, observable behavior in real time. Although this overcomes data accuracy difficulties, the challenge posed by learning analytics is interpreting the resulting data against pedagogical intent and the local context to evaluate the success or otherwise of a particular learning activity (Dawson, Bakharia, Lockyer, & Heathcote, 2010). Learning designs, which document pedagogical intent and plans, potentially provide the context to make sense of learning analytics data. Essentially, learning design establishes the objectives and pedagogical plans, which can then be evaluated against the outcomes captured through learning analytics.

In this article, we explore how learning design might provide the framework for interpreting learning analytics results and apply the concept to a sample learning design. Taking the example of design for case-based learning, we investigate how learning analytics can help to evaluate whether a learning design is achieving its intended purpose. Using a framework we call checkpoint and process analytics, we consider how expected learner behaviors and interactions to intended outcomes of the learning design. We argue that the resulting information allows a learning design to be evaluated in context, with a rich set of real-time, behavior-based data on how learners are currently interacting within the learning environment.

We begin with definitions and a discussion of the importance of learning analytics and learning design as fields that inform educational practice, and then turn to an example learning design that illustrates the potential for juxtaposition of learning design and learning analytics.

Overview of Learning Analytics

Over the past few decades, there has been increasing government, public, and industry interest in developing indicators of the quality of learning and teaching practices (Bloxham & Boyd, 2012; Coates, 2005, 2010; King Alexander, 2000). Arguably, shrinking fiscal resources and the expansion of a global competitive education market have fueled this increasing pressure for educational accountability. The offshoot of these economic drivers has been the development in the education sector of standardized scalable, real-time indicators of teaching and learning outcomes. However, creating such standards is a complex task given the diversity of student engagements, systems, learning outcomes, and teaching practices that are enacted across any educational institution. Any attempt to introduce wide-scale educational analytics and accountability processes thus requires a thorough understanding of the pedagogical and technical context in which the data are generated.

In universities, learning quality assurance data are generally derived from student experience surveys alongside measures of attrition, progression, and assessment scores. These data are commonly used retrospectively by university administrators and teachers to improve future iterations of courses, to determine impact on learning outcomes, and to provide a benchmark on overall performance (Coates, 2005). The high adoption of education technologies, such as learning management systems (LMS), has resulted in a vast set of alternate and accessible learning data (Greller & Drachsler, in press; Pardo & Kloos, 2012). Student interactions with the course activities via the LMS are captured and stored. The resulting digital footprints can be collected and analyzed to establish indicators of teaching quality and provide more proactive assessment of student learning and engagement. This area of applied research is becoming known as learning analytics.

As defined for the first Learning Analytics and Knowledge Conference in 2011, the study of learning analytics is the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (https://tekri.athabascau.ca/analytics/). Research in learning analytics interrogates the data associated with a learner’s online interactions to create predictive models for performance (Macfadyen & Dawson, 2010) and attrition (Campbell & Oblinger, 2007), as well as more complex learning dimensions such as dispositions and motivations (Buckingham Shum & Deakin Crick, 2012; Dawson, Macfadyen, & Lockyer, 2009). These forms of data inform decisions about future learning and teaching practice. The emergent field is multidisciplinary and draws on methodologies related to educational data mining, social network analysis, artificial intelligence, psychology, and educational theory and practice.

Learning analytics integrates and analyzes the “big data sets” available in educational contexts to gain a better understanding of student engagement, progression, and achievement. Although the field is still in its infancy, learning analytics can help teachers interpret learner- and instructor-centric data for informing future pedagogical decisions. To date, learning analytics studies have tended to focus on broad learning measures such as predictors of student attrition (Arnold, 2010), sense of community and achievement (Fritz, 2011), and overall return on investment of implemented technologies (Norris, Baer, Leonard, Pugliese, & Lefrere, 2008). However, learning analytics also provides additional and more sophisticated measures of the student learning process that can assist teachers in designing, implementing, and revising courses. Although there is a vast potential for this field, there remains much work to be done to build the theoretical and empirical base that provides clear evaluative procedures for matching observed student interaction behaviors with course- and program-level learning goals and outcomes (Pardo & Kloos, 2012). These forms of analytics and the associated data sets, tools, and models for analysis can be increasingly important for informing teachers on the success and outcomes of their design of learning experiences and activities alongside monitoring student learning for direct support during the academic semester.

Overview of Learning Design

The field of learning design emerged in the early 2000s as researchers and educational developers saw the potential to use the Internet to document and share examples of good educational practice. Here, we use the term learning design, but work in the same vein has been carried out under such names as pedagogical patterns, learning patterns, and pattern language. Learning design describes the sequence of learning tasks, resources, and supports that a teacher constructs for students over part of, or the entire, academic semester. A learning design captures the pedagogical intent of a unit of study. Learning designs provide a board picture of a series of planned pedagogical actions rather than detailed accounts of a particular instructional event (as might be described in a traditional lesson plan). As such, learning designs provide a model for intentions in a particular learning context that can be used as a framework for design of analytics to support faculty in their learning and teaching decisions.

The learning design field has developed in part as a response to the discourse about teaching and learning in higher education of the preceding decades. Higher education treatises of the early 1990s (e.g., Laurillard, 1993; Ramsden, 1992) called for more effective teaching in higher education, a move away from reliance on the traditional, didactic large group lecture, and an assumption that information and communication technology would help revolutionize higher education pedagogy. Thus, the broad field of learning design was underpinned by two main aims: to promote teaching quality and to facilitate the integration of technology into teaching and learning.

A main premise of learning design has been reusability across educational contexts, based on the notion that if good teaching practice in one educational context could be captured in a description, that description could be read, interpreted, and adapted for reuse in another context. Research and development work in this area have included the creation of online repositories of learning designs that teachers could read, interpret, and adapt to their own practice (e.g., Agostinho, Harper, Oliver, Hedberg, & Wills, 2008; Conole & Culver, 2010) and the development of technical languages and tools designed to make learning designs machine readable and adaptable (Koper, 2006; Masterman, 2009).

These efforts required researchers to identify and evaluate examples of good practice. In this field, the notion of good practice manifests in such a way that most learning designs shared through repositories focus on alternative pedagogies for higher education, and most emphasize the use of technology (see, e.g., the Learning Designs site, http://www.learningdesigns.uow.edu.au/, and the Pedagogical Pattern Collector, http://193.61.44.29:42042/ODC.html). This stems from the field’s development as a response to quality teaching in higher education and assumptions that constructivist approaches to teaching and learning would support quality designs and practices. The importance of engaging and challenging the learner are among the underlying principles of good practice (Boud & Prosser, 2002). As such, we find an emphasis in learning design on project, experiential, and inquiry-based pedagogies that place importance on learner communication and interaction, often facilitated by technology.

The learning designs come in many forms and level of detail. Some draw on an architectural model to describe textually solutions to common educational problems (McAndrew & Goodyear, 2007). Some use common representations such as process diagrams, flowcharts, and tables (Falconer, Beetham, Oliver, Lockyer, & Littlejohn, 2007), and others combine text descriptions with graphical representations (Agostinho et al., 2008). Regardless of the format in which learning designs are documented, essential elements include identifying the key actors involved (teachers and students), what they are expected to do (teaching and learning tasks), what educational resources are used to support the activities, and the sequence in which the activities unfold. These essential elements may be presented with great detail and provide a highly contextualized description of a particular unit, covering specific topics. Or they may be presented more generically, free of the detail of any particular implementation of the design. Learning designs also range in granularity from presenting a teaching and learning process that might occur for an entire semester-long course to that which might occur in only one class.

Considering this variability in presentation, detail, and granularity, research in the field has focused on the dissemination, adoption, use, and usability of learning designs. The learning design approach has been found to be useful for faculty to document their own practice, for instructional designers to document the practices of those they may work with, and for both faculty and designers to interpret the practices of others (Agostinho, 2011). In particular, graphical representations of learning designs have been found to stimulate design ideas for teachers who are engaged in designing a course (Bennett, Lockyer, & Agostinho, 2004; Jones, Bennett, & Lockyer, 2011). Whether they are using learning designs documented by a graphical representations and/or a textual description to stimulate ideas and support their own design practices, teachers seem to find specific examples of learning designs—those that retain information about the original context for the design—more valuable than generic designs (Bennett et al., 2004). This suggests that teachers can use specific, detailed learning designs as examples and are able to adapt the ideas to their own context.

The most easily understood and adapted common elements within all learning designs include the following:

A set of resources for the student to access, which could be considered to be prerequisites to the learning itself (these may be files, diagrams, questions, web links, prereadings, etc.)

Tasks the learners are expected to carry out with the resources (prepare and present findings, negotiate understanding, etc.)

Support mechanisms to assist in the provision of resources and the completion of the tasks; these supports indicate how the teacher, other experts, and peers might contribute to the learning process (e.g., such as moderation of a discussion or feedback on an assessment piece; Bennett et al., 2004)

Figure 1 provides an example learning design visual representation showing three common categories of resources, tasks, and supports.

Figure 1.

A learning design visual representation showing three common categories (resources, tasks and supports).

Although learning designs can provide a description of pedagogical intention, they do not identify how students are engaged in that design during or postimplementation. This is where learning analytics can provide information for a more holistic perspective of the impact of learning activities.

Using Learning Analytics

Learning analytics has the potential to draw on a variety of data sources that are collected in a range of institutional systems, including student information systems (Lauría, Baron, Devireddy, Sundararaju, & Jayaprakash, 2012), library interactions (Bichsel, 2012), LMS (Dawson, 2010; Liaqat, Hatala, Gašević, & Jovanović, 2012), admissions systems (Dawson, Macfadyen, Lockyer, & Mazzochi-Jones, 2011), and grades (Macfadyen & Dawson, 2010). However, at present, the predominance of learning analytics research centers on the types of data available in institutional LMS. Current commercial and open-source LMS provide a level of student tracking data that can be made available to teachers as reports and tables indicating, for example, student time spent online, page views, and number of posts in a discussion forum. At present, these ubiquitous data are underutilized as an indicator of student engagement and learner progress. This is largely the result of the lack of conceptual frameworks and resulting common understanding of how to use and interpret such data, and models that can validly and reliably align such data with the learning and teaching intent (Ferguson, 2012; Mazza & Dimitrova, 2007). At present, the available LMS data are neither easily understood by teachers as they align with individual and group student engagement behaviors (activity patterns) nor presented in ways that provide easy interpretation. One approach that can assist teachers to interpret these data is via visualizations (Dawson, McWilliam, & Tan, 2008). Various approaches to learning analytics, and visualizations in particular, are discussed next.

A variety of learning analytics tools are available that summarize and visualize various elements of student behavior and activities (see Table 1).

Table 1.

Examples of Learning Analytics Tools and Visualizations.

Visualization	Available tools	Description	Framework
Reports	BlackBoard, Moodle, Desire 2 Learn	Individual user tracking, course based	Individual and cohort monitoring
Social network analysis	SNAPP—Social Networks Adapting Pedagogical Practice	Extracts and visualizes student relationships established through participation in learning management system discussions (Dawson, Bakharia, & Heathcote, 2010)	Social-constructivist models of learning
Student dashboards and monitoring	Student Activity Meter	Visualizations of student activity for promotion of self-regulated learning processes (Govaerts, Verbert, Duval, & Pardo, 2012)	Self-regulated learning—monitoring of individual behaviors and achievement to guide learning process
Individual and group monitoring	GLASS: Gradient’s Learning Analytics System	Visualizations of student and group online event activity (Leony et al., 2012)	Individual and cohort monitoring
Learning content interaction	LOCO—Analyst	Provides insight into individual and group interactions with the learning content (Jovanović et al., 2007)	Individual and cohort monitoring
Discourse analysis	Cohere	Supports and displays social and conceptual networks and connections (De Liddo, Buckingham Shum, Quinto, Bachler, & Cannavacciuolo, 2011)	Social learning and argumentation theory

One type of visualization noted in Table 1 is social networks. These network diagrams can be applied in education to depict teacher and learner online communication patterns. Tools such as SNAPP (Dawson, Bakharia, & Heathcote, 2010) draw on data from the LMS to represent visually patterns of user interactions. Figure 2 illustrates how such tools can present interaction data visually to the teacher from within the LMS. A caveat is that although visualizations offer effective ways of making sense of large data sets, they still require familiarity and expertise to fully appreciate their results. Social network diagrams in particular require some degree of literacy in interpreting the results, for example, in understanding the meaning of actor locations and nuances associated with which data were used to draw connections between actors.

Figure 2.

SNAPP visualization tool embedded in a learning management systems discussion page.

The interpretation of visualizations also depends heavily on an understanding the context in which the data were collected and the goals of the teacher regarding in-class interaction. Interpretation of the analysis thus requires alignment with the original teaching context if it is to be useful as feedback on whether the learning design has achieved its intent. Interpretation requires an understanding of the relationship among technology functionality, observed interactions behaviors, and educational theory (Heathcote, 2006). It is the conceptual bridging and understanding between the technical and educational domains that remains problematic for learning analytics (Dawson, Heathcote, & Poole, 2010). This leads to questions surrounding how analytics can begin to bridge the technical–educational divide to provide just-in-time, useful, and context-sensitive feedback on how well the learning design is meeting its intended educational outcomes. Here we argue that a critical step for moving forward on this agenda entails the establishment of methods for identifying and coding learning and teaching contexts. This requires a marriage of the processes and methodologies associated with the fields of learning analytics and learning design, one that is illustrated in the example discussed below.

Learning Analytics to Evaluate Learning Design

To explore the importance of understanding learning design (or pedagogical intent) for accurate interpretation of social network analysis in learning contexts, we examine a particular instance of interaction, as illustrated in the social network diagram presented in Figure 3.

Figure 3.

Facilitator-centric social network pattern—ego network. Each node (bubble) represents a student or instructor.

The social network diagram shows a facilitator-centric pattern. Interaction in this discussion forum may be seen to be dominated by a central participant—in this example, the central actor is the instructor. Various learning designs should result in this pattern if they are considered as successful (i.e., reflect achievement of the intended learning design). For example, if this diagram represents a question and answer (Q&A) forum on course content, then the network is well aligned with the pedagogical intentions. Q&A forums commonly represent situations where one-to-one relationships mediated by an instructor are expected. If the instructor was absent from a configuration like this, it might indicate either successful delegation of the answering of student queries to other students or, if the intent was not to delegate answering responsibility, might indicate an absent instructor and potentially frustrated students. Alternatively, if the network showed a pattern where one student facilitated the particular topic, one would expect the central node to represent the facilitating student in the early phase of discussion on the topic, where he or she mediates and clarifies understanding for his or her peers. Conversely, if the intent of the forum was to promote learner-to-learner interaction for co-construction of knowledge, then the pattern seems polarized from the intended aim. The learning design and intent of the forum clearly needs to be established before the analytics visualization provides useful evaluative insight.

Thus, to interpret the data that learner environments generate, it is important to combine learning analytics with learning design. Although learning designs provide theoretical, practice-based, and/or evidence-based examples of sound educational design, learning analytics may allow us to test those assumptions with actual student interaction data in lieu of self-report measures such as post hoc surveys. In particular, learning analytics provides us with the necessary data, methodologies, and tools to support the quality and accountability that have been called for in higher education.

Aligning Learning Analytics With Learning Design

Although theoretically learning designs and learning analytics may be seen to provide compatible information, to be truly useful a framework is needed to align the two concepts. Our discussion of learning design and learning analytics focuses on two broad categories of analytic applications. The first relates to what we term checkpoint analytics, that is, the snapshot data that indicate a student has met the prerequisites for learning by accessing the relevant resources of the learning design. For instance, checkpoint analytics would relate to metrics such as log-ins into the online course site, downloads of a file for reading, or signing up to a group for a collaborative assignment. Although these forms of analytics may be valuable for providing lead indictors of student engagement, they do not, in isolation of other data, provide insight into the learning process or understanding of how students are learning and what they are learning. As checkpoint analytics exclusively measures access to the resources included in a learning design, its value lies in providing teachers with broad insight into whether or not students have accessed prerequisites for learning and/or are progressing through the planned learning sequence (akin to attendance in a face-to-face class). Data on whether or not students have accessed prereadings or organized themselves into groups for upcoming assignments could be considered checkpoints that indicate whether the foundations for learning have been established, and thus checkpoint analytics concentrates on highlighting which students have completed these learning prerequisites and which have not.

The second type of learning analytics we term process analytics. These data and analyses provide direct insight into learner information processing and knowledge application (Elias, 2011) within the tasks that the student completes as part of a learning design. For example, social network analysis of student discussion activity on a discussion task provides a wealth of data that can offer insight into an individual student’s level of engagement on a topic, his or her established peer relationships, and therefore potential support structures. The inclusion of content analysis adds further scope for determining the level of understanding and learning models established.

The articulation of the nature of support available within learning designs helps to interpret process learning analytics. These supports give an indication of what roles we can expect to see learners and teachers taking within collaborative spaces such as discussion forums (e.g., whether we would expect exclusively student-to-student interactions in a group discussion on construction of a group assignment or facilitator-centric interactions in the Q&A portion of the forum). In this way they help to provide an expected configuration based on what support was built into the learning design.

Learning Design and Analytics Investigation

The following investigates a theoretical scenario to illustrate the potential for leveraging learning analytics in support of and for evaluation of learning design. The scenario uses a learning design drawn from a repository established through an Australian project that identified, reviewed, and documented examples of university courses that effectively used technology to facilitate flexible learning (Agostinho et al., 2008; http://www.learningdesigns.uow.edu.au/). The design selected for illustration here comprises individual, small group, and large group learning tasks and use of online resources and discussion forums.

The following describes the selected learning design, and then we discuss the types of analytics that may inform the teacher about (a) how students are learning during implementation of the design and (b) how the design might be adapted or redesigned for further iterations.

Case-Based Learning Design

The learning design investigated here provides a sequential representation and brief description of the learning design. The central point of the diagram (Figure 4) is the learning task (represented as a square with green shading) that the students are expected to do, the associated content resources (represented as a triangle with blue shading), and the teacher and/or peer support that facilitates the tasks (represented as a circle with pink shading). Each step in the sequence of the learning design is associated with potential analytics: checkpoints (represented as crosses) and processes (represented as open circles).

Figure 4.

Case-based learning design.

This semester-long learning design involves students working collaboratively on group projects relevant to the students’ future professional practice. As such, this design is typically used in professionally focused programs such as architecture, business, teaching, and multimedia design. The tasks may be carried out fully online, in face-to-face meetings, or a combination of both. The learning objectives for this design are these:

To develop specific knowledge and skills related to the project

To develop an understanding of how theory relates to practice

To develop skills in case analysis and reflection

To develop teamwork and project skills

The learning design is grounded in a case-based reasoning approach that helps students link theory to practice through a series of case analyses and project tasks. First, students individually engage in a case analysis task in which they explore and analyze a real-life case. Students may choose from a number of cases that are similar to their later project task. The cases students were to examine were made available online by the teacher; cases provided detailed descriptions of realistic situations, problems encountered, solutions used, and their outcomes. As a group and/or whole class, students then discuss the problems, solutions, and complex circumstances of their chosen cases. Next, the project task involves students working in groups to develop a written project proposal and a relevant project output (e.g., blueprint, business case, lesson plan, website). The project is designed to enable students to put theoretical concepts identified in the cases into practice, learn or reinforce skills, and deal with complex, authentic situations. At the conclusion of the project students engage in a reflection task in which they consider their experiences and extract lessons for future practice. The reflections are undertaken individually and as a group and may take the form of a discussion and/or written submission.

How Analytics Can Support Implementation of a Learning Design

Stage 1: Case Analysis Task—Checkpoint Analytics

Learning analytics can generate reports of student log-in behaviors and access to individual cases; these provide the teacher with indicators of when students have commenced the learning sequence. The opportunity then exists for automatic or teacher-generated reminder alerts that can be incorporated to prompt late starters to initiate the learning activity (such as Arnold, 2010).

Stage 2: Case Analysis Discussion Task—Process Analytics

Once students analyze their case individually, they then share their ideas with their project group members. They identify issues that arose in these cases and consider how they may be applicable to the project they are about to undertake. A network diagram generated from the online discussion forum can help the teacher identify the effectiveness of each group’s interaction process.

For example, Figure 5 illustrates what a discussion dominated by a single student would look like using social network analysis; Figure 6 shows an example of greater diversity of interaction. These forms of expected interactions and learner-behavior patterns can be used to identify deviations between interaction as anticipated from the learning design and as actual outcome. For example, if this learning design called for a student leader to facilitate peers in sharing the ideas and analysis of the cases they have considered, Figure 5 might demonstrate achievement of that design. However, if all students were equally expected to share and comment on each other’s cases, Figure 6 might be expected.

Figure 5.

Discussion dominated by one student.

Figure 6.

Equal distribution of student contribution in discussion.

Stage 3: Whole-Class Discussion Task—Process Analytics

After the project groups discuss their case analyses, the learning design calls for the teacher to facilitate a whole-class discussion. If successful, the social network analysis of discussion forum posts should illustrate the teacher as central in the network. However, as student discussion increases, the teacher may be expected to become less dominant, with discussion increasing among students. Figures 7 and 8 illustrate the expected change in instructor network position as the student discussion and facilitation process evolves.

Figure 7.

Red central node represents the instructor—typical social network visualization during the early facilitation phase, where the instructor is mediating a discussion.

Figure 8.

Social network example indicating strong student peer interaction.

Stage 4: Project Proposal Task—Process Analytics

At this stage, students begin their project task. In the first part of the project task, students work in a small group to collaborate on their project proposal. If the task is completed within a discussion forum, a social network diagram could be used to indicate established density and connections of participation as well as outliers or disconnected students disengaged from the task. If the collaboration on the assignment occurs within a document-sharing tool such as a wiki or Google Docs, student content, timing, and versions are available for analysis. A checkpoint analytic may be included here to indicate students participating or not participating in the development of the shared group document.

Stage 5: Project Development Task—Checkpoint or Process Analytics

As students work on their project task, analytics might include checkpoints to verify that students have accessed the teacher’s feedback on the proposal. Process analytics might visualize group collaboration on developing the project after receiving teacher feedback.

Stage 6: Reflection Task—Checkpoint or Process

The final reflection task can be assessed using both checkpoint and process analytics. The checkpoint is to verify whether the self-reflection template has been accessed or uploaded with student changes. In addition, further content analysis can be undertaken to map student reflections and changes over an extended period. Self-reflection requires strong metacognitive capacities that have been demonstrated to be essential for developing the skills necessary for lifelong learning (Butler & Winne, 1995).

How Analytics Support Implementation and Redesign

Overall, these kinds of instrumental checkpoint analytics and the more interpretive process analytics provide the teacher with indicators of student engagement. This can be used both during the course and after. During the delivery of a course, the teacher may use these analytics to intervene when learning behavior does not match the theoretical expectations from the learning design. Such intervention may involve the teacher sending reminders to students about the suggested progression through the task, emailing students with prompting questions to promote deeper investigation of content, or moderating a planned group discussion to stimulate more equal contribution. This is the kind of intervention that a teacher would normally undertake during implementation of the course. Traditionally, this kind of intervention relies on the teacher noticing the unanticipated or detrimental learning behavior. This awareness of learner behavior is more difficult to do in the online environment than in a face-to-face context where teachers have visual cues to draw on.

Analytics can also help with course redesign. Traditionally, educators draw on their past experience when they teach a course or when they are designing a new course. For many teachers this may be an informal process that relies on recall, student surveys, and/or teacher notes recorded during or after the teaching session. Revisiting the learning analytics collected during the course can support teachers when they are planning to run the course again or when the learning design is being applied to a different cohort or context.

The review of the checkpoint and process analytics of case-based learning design discussed here provides important data to assist a teacher in refining the overall design and integration. Updated case resources might account for examples of past student projects and the problems encountered (as indicated by the analytics), or additional resources might provide students with guidance on strategies for effective project team processes. The analytics may also help a teacher plan for points in the task sequence in which they may need to provide additional support to students.

Conclusion

This article argued that the evaluative potential of learning analytics would be significantly enhanced by reference to the learning design that documents pedagogical intent. In addition, we looked at the value that the accountability and quality agenda might gain from the ability to use learning analytics for real-time evaluation of learning within a specific pedagogical design. An example of a learning design was explored to outline how analytics tools of two types—checkpoint and process analytics—could generate analytics that allow for comparison of expected behaviors and interactions with outcomes of a learning design. Using the example of case-based learning design, we saw how the resulting learning analytics allows a learning design to be evaluated in light of its pedagogical intent, using a rich set of real-time, behavior-based data on learner interaction within the learning environment. The next stages of research and development include several parallel directions: engaging teachers and students in understanding and using visual patterns of interaction as a means to encourage learning activity; scaling up to larger numbers of classes, providing a base for comparing statistically the observed to expected analytics of behaviors and interactions; and using results to provide meaningful feedback to teachers on how their learning design is meeting their pedagogical goal and to assist them in decisions around design and pedagogical change in real time.

As the field of learning analytics continues to evolve and the diversity of data sources increases, there will be an associated rise in the number and accuracy of predictive models (logistic regressions, decision trees, support vector machines, etc.) for student performance and progression. As these models come into the mainstream, there is an opportunity to leverage analytics tools and visualizations to establish pedagogical recommendations. However, as noted above, any user interaction behavior must be analyzed in the specific education context such as the learning design and course modality. An understanding of the learning design context is imperative for establishing accurate predictive models alongside pedagogical recommendations. Establishing a conceptual framework for typical learning analytics patterns expected from particular learning designs can be considered an essential step in improving evaluation effectiveness and to build the foundation for pedagogical recommender systems in the future.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Lori Lockyer is the Vincent Fairfax Family Foundation Chair in Teacher Education and head of the School of Education at Macquarie University. Her research focuses on learning design, teacher design thinking and practices, and the use of information and communication technology to support learning in K-12 education, professional education, and health education.

Elizabeth Heathcote is manager, online development at the University of Adelaide. She has a professional background in instructional design, project management, change management, and leadership in educational technologies. Her particular interests are learning activity design in online environments and change management in higher education.

Shane Dawson is the deputy director of the Learning and Teaching Unit and associate professor of technology enhanced learning at the University of South Australia. His research activities focus on learning analytics and social networks to inform educational theory and practice. He has also been involved in developing pedagogical models for enhancing creative capacity in undergraduate students. He is a cofounder and executive member of the Society for Learning Analytics Research.

References

Agostinho

(2011). The use of a visual learning design representation to support the design process of teaching in higher education. Australasian Journal of Educational Technology, 27(6), 961-978.

Agostinho

Harper

Oliver

Hedberg

Wills

(2008). A visual learning design representation to facilitate dissemination and reuse of innovative pedagogical strategies in university teaching. In Botturi

Stubbs

(Eds.), Handbook of visual languages for instructional design: Theories and practices (pp. 380-393). Hershey, PA: Information Science Reference, IGI Global.

Arnold

K. E.

(2010). Signals: Applying academic analytics. EDUCAUSE Quarterly, 33(1). http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/SignalsApplyingAcademicAnalyti/199385

Bennett

(2002). A technology supported constructivist learning environment that uses real-life cases to support collaborative project work. Retrieved from http://www.learningdesigns.uow.edu.au/exemplars/info/LD1/index.html

Bennett

Lockyer

Agostinho

(2004, December). Investigating how learning designs can be used as a framework to incorporate learning objects. In Atkinson

McBeath

Jonas-Dwyer

Phillips

(Eds.), Beyond the comfort zone: Proceedings of the 21st ASCILITE conference (pp. 116-122). Retrieved from http://www.ascilite.org.au/conferences/perth04/procs/bennett.html

Bichsel

(2012). Analytics in higher education: Benefits, barriers, progress and recommendations. Louisville, Colorado: EDUCAUSE Center for Applied Research.

Bloxham

Boyd

(2012). Accountability in grading student work: Securing academic standards in a twenty-first century quality assurance context. British Educational Research Journal, 38(4), 615-634.

Boud

Prosser

(2002). Appraising new technologies for learning: A framework for development. Educational Media International, 39(3-4), 237-245.

Buckingham Shum

Deakin Crick

(2012, May). Learning dispositions and transferable competencies: Pedagogy, modelling and learning analytics. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver, Canada.

10.

Butler

D. L.

Winne

P. H.

(1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281.

11.

Campbell

Oblinger

(2007). Academic analytics. Retrieved from http://connect.educause.edu/library/abstract/AcademicAnalytics/45275

12.

Coates

(2005). The value of student engagement for higher education quality assurance. Quality in Higher Education, 11(1), 25-36.

13.

Coates

(2010). Defining and monitoring academic standards in Australian higher education. Higher Education Management and Policy, 22(1), 41-58.

14.

Conole

Culver

(2010). The design of Cloudworks: Applying social networking practice to foster the exchange of learning and teaching ideas and designs. Computers & Education, 54(3), 679-692.

15.

Dawson

(2010). “Seeing” the learning community: An exploration of the development of a resource for monitoring online student networking. British Journal of Educational Technology, 41(5), 736-752.

16.

Dawson

Bakharia

Heathcote

(2010, May). SNAPP: Realising the affordances of real-time SNA within networked learning environments. Paper presented at Networked Learning—Seventh International Conference, Aalborg, Denmark.

17.

Dawson

Bakharia

Lockyer

Heathcote

(2011). “Seeing” networks: Visualising and evaluating student learning networks. Final Report 2011. Canberra: Australian Learning and Teaching Council. Retrieved from http://research.uow.edu.au/content/groups/public/@web/@learnnet/documents/doc/uow115678.pdf

18.

Dawson

Heathcote

Poole

(2010). Harnessing ICT potential: The adoption and analysis of ICT systems for enhancing the student learning experience. International Journal of Educational Management, 24(2), 116-128.

19.

Dawson

Macfadyen

Lockyer

(2009). Learning or performance: Predicting drivers of student motivation. In Atkinson

R. J.

McBeath

(Eds.), Same places, different spaces. Proceedings of the 26th Annual ASCILITE International Conference (pp. 184-193). Auckland: University of Auckland, Auckland University of Technology, and Australasian Society for Computers in Learning in Tertiary Education.

20.

Dawson

Macfadyen

Lockyer

Mazzochi-Jones

(2011). Using social network metrics to assess the effectiveness of broad-based admission practices. Australasian Journal of Educational Technology, 27(1), 16-27.

21.

Dawson

McWilliam

Tan

J. P. L.

(2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. In Hello! Where Are You in the Landscape of Educational Technology? Proceedings ASCILITE Melbourne 2008. Retrieved from http://www.ascilite.org.au/conferences/melbourne08/procs/dawson.pdf

22.

De Liddo

Buckingham Shum

Quinto

Bachler

Cannavacciuolo

(2011, February). Discourse-centric learning analytics. Paper presented at the 1st International Conference on Learning Analytics and Knowledge, Banff, Canada.

23.

Elias

(2011). Learning analytics: Definitions, processes and potential. Retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

24.

Falconer

Beetham

Oliver

Lockyer

Littlejohn

(2007). Mod4L final report: Representing learning designs. Final project report for Joint Information Systems Committee (JISC) Design for Learning Program. Glasgow, UK: Glasgow Caledonian University.

25.

Ferguson

(2012). The state of learning analytics in 2012: A review and future challenges (Tech. Rep. No. KMI-12-01). Milton Keynes, UK: Open University, Knowledge Media Institute. Retrieved from http://kmi.open.ac.uk/publications/techreport/kmi-12-01

26.

Fritz

(2011). Classroom walls that talk: Using online course activity data of successful students to raise self-awareness of underperforming peers. Internet and Higher Education, 14(2), 89-97. doi:10.1016/j.iheduc.2010.07.007

27.

Govaerts

Verbert

Duval

Pardo

(2012, May). The student activity meter for awareness and self-reflection. Paper presented at the ACM Annual Conference on Human Factors in Computing Systems, Austin, TX.

28.

Greller

Drachsler

(in press). Translating learning into numbers: A generic framework for learning analytics. Educational Technology and Society.

29.

Heathcote

(2006). Learning design templates—A pedagogical just-in-time support tool. In Minshull

Mole

(Eds.), Designing for learning: The proceedings of Theme 1 of the JISC Online Conference: Innovating e-Learning (pp. 19-26). Bristol, UK: JISC.

30.

Jones

Bennett

Lockyer

(2011). Applying a learning design to the design of a university unit: A single case study. In Bastiaens

Ebner

(Eds.), Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 3340-3349). Chesapeake, VA: AACE.

31.

Jovanović

Gasevic

Brooks

Devedzic

Hatala

Eap

Richards

(2007). Using semantic web technologies to analyze learning content. IEEE Internet Computing, 11(5), 45-53.

32.

King Alexander

(2000). The changing face of accountability: Monitoring and assessing institutional performance in higher education. Journal of Higher Education, 71(4), 411-431.

33.

Koper

(2006). Current research in learning design. Educational Technology and Society, 9(1), 13-22.

34.

Lauría

E. J. M.

Baron

J. D.

Devireddy

Sundararaju

Jayaprakash

S. M.

(2012, May). Mining academic data to improve college student retention: An open source perspective. Paper presented at LAK 12: 2nd International Conference on Learning Analytics and Knowledge, Vancouver, Canada.

35.

Laurillard

(1993). Rethinking university teaching: A framework for the effective use of educational technology. London, UK: Routledge.

36.

Leony

Pardo

de la Fuente Valentin

De Castro

D. S.

Kloos

C. D.

(2012, May). GLASS: A learning analytics visualization tool. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver, Canada.

37.

Liaqat

Hatala

Gašević

Jovanović

(2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58(1), 470-489.

38.

Macfadyen

Dawson

(2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599.

39.

Masterman

(2009). Activity theory and the design of pedagogic planning tools. In Lockyer

Bennet

Agostinho

Harper

(Eds.), Handbook of research on learning design and learning objects issues applications and technologies (Vol. 1, pp. 209-227). Hershey, PA: Information Science Reference.

40.

Mazza

Dimitrova

(2007). CourseVis: A graphical student monitoring tool for supporting instructors in web-based distance courses. International Journal of Human-Computer Studies, 65(2), 125-139.

41.

McAndrew

Goodyear

(2007). Representing practitioner experiences through learning design and patterns. In Beetham

Sharpe

(Eds.), Rethinking pedagogy for a digital age (pp. 92-102). London, UK: Routledge.

42.

Norris

Baer

Leonard

Pugliese

Lefrere

(2008). Action analytics: Measuring and improving performance that matters in higher education. EDUCAUSE Review, 43(1), 42-67.

43.

Pardo

Kloos

C. D.

(2012, May). Stepping out of the box. Towards analytics outside the learning management system. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver, Canada.

44.

Ramsden

(1992). Learning to teach in higher education. London, UK: Routledge.