Engaging with assessment: Increasing student engagement through continuous assessment

Engagement and assessment

Student engagement, first introduced by Astin (1999) as student involvement in 1984, was initially defined as ‘the amount of physical and psychological energy that the student devotes to the academic experience’ (p. 518). Since then, there has been much debate about the meaning of student engagement (Dunne and Owen, 2013; Trowler, 2010) with more recent definitions incorporating the role of an institution in student engagement, for example, Exeter et al. (2010) suggest that student engagement refers to the ‘time, energy and resources spent on activities designed to enhance learning at university’ (p. 762). Student engagement is currently an important topic in higher education, with learning, marketing, retention rates and economics all influenced by the level of student engagement at an institution (Trowler, 2010). Student engagement is known to be correlated with student satisfaction (Kuh et al., 2007) and positive engagement by students is acknowledged to be an important indicator of quality (Coates, 2005). It is therefore important for higher education institutions to focus on increasing student engagement.

Student engagement has been noted to be ‘a useful proxy for what happens in the learning environment’ (Zepke, 2013: 99). Kuh (2009) recognises that student engagement can be influenced by practices at their institution and states, ‘Student engagement represents the time and effort students devote to activities that are empirically linked to desired outcomes of college and what institutions do to induce students to participate in these activities’ (p. 683). Robinson and Hullinger (2008) hold the view that a faculty member should provide courses that promote student engagement, with particular mention of the online learning environment. They also note that when designing a curriculum there should be a focus on increasing interaction with class materials (Robinson and Hullinger, 2008). Mitchell and Carbone (2011) suggest that engagement can be enhanced depending on the nature of the task the students are carrying out.

Assessment is a vital part of all university studies (Higher Education Academy, 2012). It is thought that the ‘single biggest influence on how students approach their learning’ (Rust et al., 2005: 231) is assessment, while Race (2009) states that ‘The most important thing we do for our students is assess their work’ (p. 3). ‘Backwash’ is a term which refers to the effect assessment has on student learning (Biggs and Tang, 2011; Elton, 1987), recognising that what and how students learn is largely determined by assessment. As well as providing a grade or mark to a student, assessment should also be designed to help students to learn and to engage with their studies.

Students tend to approach learning in one of two ways, broadly defined as ‘surface’ and ‘deep’ (McMahon, 2006). Surface learning aims to meet course requirements with minimal effort (Biggs and Tang, 2011), for example, memorising facts without understanding, while deep learning aims to engage with the task meaningfully, with a focus on understanding underlying theories and principles. The approach a student takes to learning will frequently depend on the task (Ramsden, 2003), and it has been stated that ‘one cannot be a deep or surface learner, one can only learn the content in a deep or surface way’ (p. 49). A student will vary their approach depending on the situation. Exeter et al. (2010) state that an engaged student will be ‘one who is a deep learner’ (p. 762). One of the tasks of an effective teacher is to try to encourage all students to take a deep approach to learning (Biggs and Tang, 2011; McMahon, 2006). It is important to try to encourage deep learning through assessment, and a positive learning experience can be created through careful alignment of assessment to learning outcomes (Biggs and Tang, 2011).

Student effort (and engagement) is known to often be unevenly distributed during a module (Gibbs and Simpson, 2004). Many students will not carry out work, or spend little time on it, if it does not contribute to their grade (Rust, 2002). Students are often strategic in their approach to university study, and many ‘only seriously engage with learning tasks if they are going to be assessed, with marks attached’ (Rust, 2002: 153). Lovatt et al. (2007) found that ‘there is a tendency for students to concentrate their study into exam study weeks rather than engaging with the material throughout the semester’ (p. 40). In many cases, this means that students are only seriously engaging with a module at perhaps two or three points during that module, when they are being assessed. Working with this knowledge and the fact that, in most cases, ‘assessment defines what students regard as important, how they spend their time’ (Brown et al., 1997: 7), it should be possible to encourage increased motivation and engagement with a module by careful design and development of the assessment scheme (Biggs, 1996).

Van Gaal and De Ridder (2013) studied the impact of introducing assessment tasks at three points during a module on students’ examination performances and found that, in general, students performed better in the examination following the introduction of the assessment tasks. Marriott and Lau (2008) found that the use of online phased assessments encouraged students to work consistently through the year. The use of continuous assessments has also led to increased, and more uniform, attendance and improved examination performance (Cole and Spence, 2012). Continuous assessment, defined as ‘the use of tests over a learning unit, and the accumulation of results in a final grade’ (Miller et al., 1998: 34), has been described as ‘more formative than summative’ (Le Grange and Reddy, 1998: 10). Trotter (2006) used the term ‘continuous summative assessment’ to refer to this method of assessment and found that, although it was relatively time-consuming for the lecturer, the positive effects on student motivation and learning were great. There is concern that continuous assessments encourage surface learning (Jordan, 2009; Tan, 1992), however, with considered design this can be avoided, allowing testing of higher cognitive functions such as critical thinking and analysis skills (Brady, 2005; Draper, 2009; Leung et al., 2008).

Student engagement with virtual learning environments (VLEs)

VLEs are now used at most higher education institutions. They are frequently used as a repository for module information including lecture materials, assessment information and communications, with limited use of interactive activities within the VLE (Blin and Munro, 2008). Hsu (2011) suggests that VLEs should present materials in a number of ways in order to accommodate different learning styles and therefore encourage greater engagement by students. Hart and Rush (2007) suggest that the use of VLEs may necessitate an adjustment to the learning styles and approaches taken by individual students, and in doing so, may encourage them into modes of deep learning. Knight (2010) found that students who interacted with the online resources in a VLE consistently through a module adopted a deep learning approach. The design and functionality of a VLE is important and plays an important role in student engagement (Beer et al., 2010; Dale and Lane, 2007).

Lovatt et al. (2007) found that student use of a VLE decreased as the module progressed, and suggested that student interaction should be encouraged using methods including continuous assessment. Dale and Lane (2007) carried out an investigation of student engagement with a VLE with the intention of enhancing student engagement with online learning through the development of learning and teaching strategies. It is known that not all students will engage with a module (however interactive) to the same extent (Hart and Rush, 2007) and that motivating factors are needed to encourage engagement. Dale and Lane (2007) suggest that as student motivation is largely assessment-driven it would be a sensible approach to link VLE-based activities with assessment. Marriott and Lau (2008) found that the use of online assessments had a positive impact on student motivation and engagement. It has been suggested that a student’s engagement throughout their degree is influenced by their first-year at university (Kuh et al., 2007) and ‘that patterns of online learning appear to be fixed early on in students’ university careers’ (Maltby and Mackie, 2009: 58).

VLEs can provide much data about student interaction with learning materials (Beer et al., 2010). As delivery moves from face-to-face methods to online methods, the teacher lacks the usual visual clues relating to student engagement, therefore monitoring VLE interaction is one way of tracking student engagement (Bodger et al., 2007; Ferguson, 2012). As well as engagement, the learning approaches of students have been investigated through their use of a VLE (Knight, 2010; Phillips et al., 2012) while learning style has also been studied (Hsu, 2011). Recently, a field of research into analytics in higher education has evolved (e.g. Campbell and Oblinger, 2007; Ferguson, 2012; Van Barneveld et al., 2012), originally under the umbrella of ‘academic analytics’ the new field of learning analytics has emerged. Learning analytics has been defined as ‘the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs’ (Society for Learning Analytics Research definition in Ferguson, 2012: 314). Van Barneveld et al. (2012) suggest that learning analytics can have a number of levels of focus including the individual learner, the instructor, the department and an institution. It is hoped that analysis of such data will allow instructors to monitor student engagement and evaluate the impact of learning and teaching practices (Chen et al., 2008; Dawson et al., 2008; Dawson and McWilliam, 2008; Henrie et al., 2015; Van Barneveld et al., 2012).

Different metrics have been investigated as indicators of engagement with a course VLE (Henrie et al., 2015). Ramos and Yudko (2008) found that student success in a course was related to the total number of page hits in a VLE (i.e. learner–content interactions), a similar finding to Beer et al. (2010) who found a statistically significant relationship between student engagement (as recorded by number of clicks in a VLE) and student success. Agudo-Peregrina et al. (2014) studied student VLE interactions and analysed the data to find which interactions were important in influencing student academic performance, and whether VLE use had an impact on student success. They found that there was a significant relationship between student VLE interactions and success on online delivery courses, whereas this relationship was not significant for face-to-face courses supported by a VLE (Agudo-Peregrina et al., 2014). Mogus et al. (2012) investigated the impact of student activity in a VLE by comparing student VLE usage (by studying the activity logs) with their final grades. They found that there were some positive correlations between students’ activity logs and their final grades, with the strength of the correlation with final grade varying for different VLE-based activities. Henrie et al. (2015) suggest that more research is needed to understand what the VLE usage data can tell us about the student learning experience in order that teaching can be improved to better help students learn.

While there is much that we know about student engagement, there is a need to further explore the monitoring of engagement through VLE usage, whether curriculum changes increase student engagement with a VLE, and whether continuous e-assessments increase student engagement. This study seeks to investigate the impact that the introduction of a weekly low-stakes online test had on the engagement of students with a module as measured by their activity within the module VLE.

Methods

This investigation is centred on an optional 20-credit second-year undergraduate Physical Geography module (referred to as Module A) delivered at the University of Northampton, UK. Module A was delivered as a 2 hour session (consisting of lectures, seminars and practicals) weekly for 24 weeks. The VLE was used as a repository for module information (including lecture slides, handouts, reading lists, assignment information, links to news articles and TV programmes of interest), for assessments (e-submission and tests) and as a communication tool, and all students were encouraged to use the VLE as part of a blended learning approach. In 2011–2012, 20% of the module was assessed using one in-class test; in 2012–2013, this was changed to weekly online tests (taken in the VLE in the students’ own time), with 20 tests each worth 1% of the assignment grade (20% in total). Both types of test (in-class and online) comprised multiple-choice questions, fill in the blank questions, short answer questions and data interpretation questions and were designed to assess the same learning outcomes. In both types of test, there was a greater focus on testing students’ understanding rather than recall of facts in order to encourage a deeper approach to learning. The other 80% of the module was assessed by coursework.

Data pertaining to two other optional 20-credit second-year undergraduate Physical Geography modules (referred to as Modules B and C) were also collected. Modules B and C were delivered in the same way as module A (a 2 hour session (consisting of lectures, seminars and practicals) weekly for 24 weeks) with the module VLE used as a repository for module information, for submission of assessments and as a communication tool. The modules were assessed using a combination of examination (Module B), in-class tests (Module C) and coursework (Modules B and C).

All modules contained a mix of students typical of the University (a mix of males and females, mostly full-time students, the majority were home and European Union (EU) students with some international students, a few mature students and some students with specific learning difficulties). In all three modules, the VLE was continually updated with new material being provided for students throughout the duration of the module.

Results

Did the introduction of continuous e-assessments change student engagement with a VLE?

Student use of Module A VLE in 2011–2012 (in-class test) and in 2012–2013 (online tests) was investigated by analysing the total number of hours each student spent in the VLE and the total number of logins of each student to the VLE. The total number of hours a student spent in Module A VLE during 2011–2012 and 2012–2013 ranged from 0.7 to 226.2 hours (Table 1). The mean number of total hours was highest in 2012–2013 and varied significantly from 2011–2012 (two-sample t(19) = −10.22, p = 0.000). The total number of logins to Module A VLE ranged from 13 to 334 (see Table 2). The mean number of total logins was highest in 2012–2013 and was significantly different from the mean number of total logins in 2011–2012 (two-sample t(20) = −12.39, p = 0.000). It is clear that the student engagement with the module VLE was significantly higher in 2012–2013 than in 2011–2012.

OPEN IN VIEWER

Table 1.

Total number of hours students spent in Module A VLE during the 2011–2012 and 2012–2013 academic years.

Academic year	Minimum number of total hours	Maximum number of total hours	Mean number of total hours	Standard error mean	Standard deviation
2011–2012	0.7	19.8	7.7	1.4	5.4
2012–2013	11.9	226.2	135	12.4	55.3

OPEN IN VIEWER

Table 2.

Total number of logins by each student to Module A VLE during the 2011–2012 and 2012–2013 academic years.

Academic year	Minimum number of total logins	Maximum number of total logins	Mean number of total logins	Standard error mean	Standard deviation
2011–2012	13	44	30.4	2.5	9.3
2012–2013	107	334	188.3	12.5	55.9

Did students engage more with the VLE of a module using continuous e-assessments than with VLEs of modules using only traditional assessment methods?

The engagement of students with the VLEs of two modules which use traditional assessment methods was studied and compared with the engagement of students with Module A VLE. The total hours each student spent in the VLE, the total number of logins of each student to the VLE and all user activity within the VLE were studied. The total number of hours a student spent inside the VLE during the 2012–2013 academic year for Modules A, B and C (see Table 3) varied from 0.14 (Module B) to 226.2 (Module A). The mean number of total hours spent in a module VLE was highest for Module A, followed by Module C and then Module B (see Table 3). The mean number of total hours in a module VLE varied significantly between the three modules (F(2, 56) = 11.94, p = 0.000). Post hoc tests showed that the mean number of total hours spent in Module A VLE was significantly different from the mean number of total hours spent in the VLE of both Modules B and C, while Modules B and C did not differ significantly from each other.

OPEN IN VIEWER

Table 3.

Total number of hours students spent in the VLE module sites for three second-year undergraduate modules during the 2012–2013 academic year (Modules A, B and C).

Module	Minimum number of total hours	Maximum number of total hours	Mean number of total hours	Standard error mean	Standard deviation
Module A	11.9	226.2	135	12.4	55.3
Module B	0.14	173.3	63.1	11.4	45.4
Module C	21.3	163.3	82.47	7.99	38.32

The total number of times each student logged in to the VLE for Modules A, B and C during the 2012–2013 academic year (see Table 4) varied from 17 (Module B) to 334 (Module A). The mean number of logins to a module VLE was highest for Module A, followed by Module C and then Module B (see Table 4). The mean number of logins to a VLE varied significantly between the three modules (F(2, 56) = 19.41, p = 0.000). Post hoc tests showed that the mean number of logins to Module A VLE was significantly different from the mean number of logins to the VLEs of both Modules B and C, while Modules B and C did not differ significantly from each other.

OPEN IN VIEWER

Table 4.

Total number of logins made by each student to the VLE module sites for three second-year undergraduate modules during the 2012–2013 academic year (Modules A, B and C).

Module	Minimum number of total logins	Maximum number of total logins	Mean number of total logins	Standard error mean	Standard deviation
Module A	107	334	188.3	12.5	55.9
Module B	17	189	90.3	12.2	48.8
Module C	27	233	104.7	12.6	60.3

Total user activity within the content area of a module’s VLE during the spring term of the 2012–2013 academic year varied from 867 (Module B) to 5567 (Module A), as can be seen from Table 5. Total daily activity within the content area of the VLE sites ranged from 0 (all modules) to 251 (Module A), as can be seen from Figure 1 and Table 5. All modules had days with no activity in the content areas (Module A = 1 day; Module B = 16 days; Module C = 7 days). It is clear that peaks of user activity coincide with assessment deadlines (Figure 1). Mean daily activity was highest for Module A, followed by Module C and then Module B (see Table 5). The mean total daily user activity differed significantly between the three modules (F(2, 240) = 41.64, p = 0.000). Post hoc tests indicated that the mean daily total user activity within the content areas of the VLE sites were significantly different between all three modules.

OPEN IN VIEWER

Table 5.

All user activity within content area of module VLE sites during the spring term (07 January 2013 to 28 March 13) for three second-year undergraduate modules.

Module	Total user activity within site during study period	Minimum daily activity	Maximum daily activity	Mean daily activity	Standard error mean	Standard deviation
Module A	5546	0	251	68.5	6.69	60.2
Module B	867	0	121	10.7	1.78	16.0
Module C	2291	0	189	28.3	3.91	35.2

OPEN IN VIEWER

Figure 1.

All user activity inside content area of VLE module site during the spring term (07 January 2013 to 28 March 2013) for three second-year undergraduate modules. Traditional assessment deadlines are marked by the grey bars; Continuous e-assessment deadlines are marked by the white bars (a) Module A, (b) Module B and (c) Module C.

The data from all three metrics show that the module which used continuous e-assessments experienced significantly increased student engagement with the VLE compared to the two modules which used traditional assessment methods.

Discussion and conclusion

The adoption of continuous e-assessments into a module led to increased student engagement with that module, as measured by interactions with the VLE. For many students, much of their interaction with a module now occurs in a VLE, and this offers an opportunity to monitor student interaction and engagement with the module, at both individual and cohort levels (Van Barneveld et al., 2012). Monitoring student use of a VLE does seem to provide useful information about engagement, although deciding upon the metric(s) to be used to represent a student’s engagement is important (Henrie et al., 2015). This study found that it was necessary to look at a number of factors representing VLE usage; it is not sensible to look just at data on the hours spent in the VLE (cf. Beer et al., 2010; Dawson et al., 2008). Engagement with a VLE was investigated by looking at the total hours spent in the VLE, total number of logins to the VLE and total interactions with content in the VLE. These parameters gave a reasonably good picture of how students used the VLE, but caution is needed when interpreting such data. For example, the student who spent the shortest time in Module A VLE in the 2012–2013 academic year achieved the highest grade in the e-assessments. This student also accessed the most items in the VLE and had a number of logins greater than the mean. The students used their time in the VLE effectively, downloading and saving files for later use. The use of only one metric (in this case time in VLE) would not have provided a useful measure of this individual’s engagement with the module. The collection and analysis of data relating to student interactions with a VLE has proven to be a useful method of inferring engagement, but it should be noted that there are a number of legal and ethical issues associated with learning analytics which need to be considered (Jisc, 2015; Sclater, 2014).

The introduction of low-stakes continuous e-assessments into a module (Module A) led to increased engagement with the module VLE throughout the academic year when compared to the same module of the previous year. The increased interactions with the VLE were much greater than could be expected if all the ‘additional’ interactions (logins and time in VLE) in 2012–2013 when compared to 2011–2012 were directly linked to students actually taking the e-assessments (this would account for a maximum of 10 hours and 20 logins per student). Interaction with the VLE of Module A (which uses e-assessments) was significantly higher than that with the VLEs of Modules B and C (which are assessed using more traditional assessments). The introduction of continuous assessment to Module A seems to have increased the total engagement as well as ensuring that students engage throughout the year. There were still clear peaks in engagement with Module A VLE associated with assessment deadlines, but the weekly nature of the assessment encouraged students to engage with the VLE consistently throughout the module. In the modules with traditional assessments (Modules B and C), it is clear that student engagement peaked a few times through the year (the days before the assessment deadlines). These findings corroborate those of earlier research which found that continuous assessment led to enhanced student engagement (Marriott and Lau, 2008) and support the suggestion that continuous assessment could help to increase engagement with a VLE throughout a module (Dale and Lane, 2007; Lovatt et al., 2007), perhaps leading to higher levels of attainment by students (cf. Knight, 2010).

Although engagement has been measured via interaction with the module VLE in this study, it was noticeable that class attendance of Module A increased from the previous year (Holmes, 2015). The students in Module A were surveyed and questions asked included whether students read through their lecture notes after a lecture, and whether students accessed the lecture PowerPoints via the VLE after the lecture (Holmes, 2015). In 2011–2012, 28% of the students stated that they did read through their lecture notes after a lecture, whereas 82% of the 2012–2013 students did this (Holmes, 2015). In all, 25% of the students on the module in 2011–2012 said that they accessed the lecture PowerPoints via the VLE compared with 88% of those on the module in 2012–2013 (Holmes, 2015). These findings all support the interaction with VLE evidence of the increased engagement of students with Module A in 2012–2013 when compared with 2011–2012. Holmes (2015) investigated students’ perceptions of their learning and engagement in response to the use of continuous e-assessment (in Module A) and found that students considered that their increased engagement, and improved learning and understanding were a direct consequence of the use of the continuous e-assessments with one stating that the tests ‘make sure I actually read my notes and understand what was taught each week’ (Holmes, 2015: 8). The frequent testing and associated immediate feedback motivated and encouraged students to engage with the module material and to perform well throughout the module (Holmes, 2015). The average grade for Module A was higher in 2012–2013 than in 2011–2012, with a greater percentage of students achieving the equivalent of a first-class or upper-second-class grade in the module in 2012–2013 than in 2011–2012. These findings, along with the VLE usage data, suggest that careful changes in curriculum can successfully increase student engagement with a module.

An important limitation of this study is that it investigated the engagement of small numbers of students on just three modules at one higher education institution within one country. A larger study, including students studying a variety of subjects in a number of institutions, including those from other countries, would be useful. Investigating the thoughts of students on the impact that the design of a VLE has on their engagement with it, and exploring how different students use the VLE to support their learning would also be beneficial, ensuring that a VLE can be designed to help all learners. It could also be valuable to compare the VLE activity of individual students taking Module A across different modules to explore whether they change their learning behaviour in other modules as a result of an assessment change (introduction of continuous e-assessments) in one module. This study only presents a snapshot of information; a longitudinal study following the VLE interactions of these individual students over the remainder of their course would allow the investigation of whether there are any long-term benefits to their engagement with their studies. If it is found that students do change their behaviour as a result of curriculum change then it may be beneficial to include low-stakes continuous e-assessment in first-year undergraduate modules to encourage all students to engage with the VLE throughout their degree programme.

There is currently a move towards greater blended learning and online delivery. It is possible for online learning environments to have as good student engagement, and therefore student satisfaction levels, as face-to-face learning environments. Engagement with a VLE is influenced by the VLE itself, and therefore, it is important that VLEs are designed to encourage high student engagement and not just to act as an information repository. It is clear that careful curriculum design, including assessment activity, can significantly increase student engagement, thereby enhancing the student experience and leading to greater levels of student satisfaction.