Active learning: The importance of developing a comprehensive measure

Abstract

This article reports on an investigation into the validity of a widely used scale for measuring the extent to which higher education students employ active learning strategies. The scale is the active learning scale in the Australasian Survey of Student Engagement. This scale is based on the Active and Collaborative Learning scale of the National Survey of Student Engagement. The particular focus of the study was to investigate effects resulting from the addition of a small number of items to the active learning scale designed to capture some highly engaging, mostly online, activities. The items were developed in response to concerns that students studying in distance mode often report lower average scores on active learning scales than their on-campus counterparts. The additional items relate to activities such as working online with other students and faculty. The findings show that average scores on the Australasian Survey of Student Engagement/National Survey of Student Engagement scale increase significantly when the new items are included and that some differences between on-campus and distance education students narrow significantly. These findings have implications for the development of more robust and comprehensive instruments to measure active learning.

Keywords

active learning Australasian Survey of Student Engagement distance education flow National Survey of Student Engagement online learning

Measuring active learning

Active learning is a very broad concept that covers or is associated with a wide variety of learning strategies. This research is motivated by reports, discussed in detail below, that suggest that students working online score lower on some commonly used instruments designed to measure active learning than students learning on campus. This appears to run counter to the experience of many educators and is not in line with the considerable body of research demonstrating that distance education delivers equivalent (or better) educational experiences and learning outcomes to that of traditional on-campus education (Bernard et al., 2004; Means et al., 2009).

Active learning is sometimes described by contrasting it with what it is not as Petress (2008) does in stating that active learning is ‘the opposite of passive learning’ (p. 566). In support of this contention by Petress (2008), Chickering and Gamson (1987) assert that ‘students do not learn much just by sitting in classes listening to teachers, memorising pre-packaged assignments, and spitting out answers’ (p. 4). Instead, active learning ‘calls forth images of active, student-centered, participatory learning’ (O’Loughlin, 1992: 792), where students must process content themselves in order to learn (Chickering and Gamson, 1987; Powner and Allendoerfer, 2008). Active learning is commonly associated with learning strategies such as experiential learning, learning by doing and service learning, peer tutoring, laboratory work, role-playing and the use of case studies (Chi, 2009). It is also commonly associated with learning activities involving technology (Laird and Kuh, 2005), including simulations or games (Scott et al., 2009) and the use of mobile or classroom-based devices such as smartphones or clickers (Dyson et al., 2009). Active learning strategies take many forms – not all of which are experiential and not all are synonymous with active learning. Active learning is often associated with constructivist theories of learning, where students learn by actively challenging and critiquing concepts developed through their own experiences or the experiences of others, possibly under the guidance of a teacher who encourages the necessary cognitive conflict (Ford, 2010). Active learning is also commonly associated with learning that involves interpersonal interaction between students and others (Chi, 2009; Chickering and Gamson, 1987; Hendriks and Maor, 2004; Moore, 1989). Finally, in active learning, student control, autonomy, self-regulation and power relationships are often seen as important, with terms such as ‘student-directed learning’ often associated with active learning (Chi, 2009; Hagel et al., 2012; O’Loughlin, 1992; Petress, 2008; Quinton and Smallbone, 2010; Rae and Cochrane, 2008).

With such a broad range of activities associated with the term ‘active learning’, it is not surprising that instruments designed to measure active learning can fail to capture some experiences that are important indicators of the concept. Such deficiencies in instruments relate to their content validity – an important basic notion of validity to do with the selection of items from the ‘content universe’ that are intended to capture a concept (Cronbach, 1971; Fitzpatrick, 1983; Neuman, 2011). If an instrument does not have good content validity, then data obtained via that instrument may be of limited use. In particular, if important indicators are not included in measures of active learning, it may be difficult to sensibly compare scores from different cohorts of students.

Evidence of possible problems with this type of validity has been observed for one widely used active learning scale: the Active Learning benchmark scale in the Australasian Survey of Student Engagement (AUSSE). The AUSSE is a survey of the engagement of undergraduate students that has been widely used annually across Australian and New Zealand higher education institutions since 2007. The AUSSE was adopted, with only some minor word changes, from the National Survey of Student Engagement (NSSE) which has been used in the United States since 1999. The AUSSE contains a number of benchmark engagement scales; one is the Active Learning benchmark scale, the equivalent of which in the NSSE is the ‘Active and Collaborative Learning’ Scale. The AUSSE/NSSE defines active learning as ‘students’ efforts to actively construct their knowledge’ (Australian Council for Educational Research (ACER), 2007: 2). In providing a rationale for the Active and Collaborative Learning scale, NSSE (2001) states that ‘Students learn more when they are intensely involved in their education and are asked to think about and apply what they are learning in different settings’ (p. 35). Accordingly, the scale used in the 2008 AUSSE comprised the seven items shown in Exhibit 1.

Exhibit 1.

The active learning items from the AUSSE.

In your experience at your institution during the current academic year, about how often have you done each of the following?

Worked with other students on projects during class

Made a class or online presentation

Asked questions or contributed to discussions in class or online

Participated in a community-based project (e.g. volunteering) as part of your study

Worked with other students outside of class to prepare assignments

Discussed ideas from your readings or classes with others outside class (e.g. students, family members, co-workers)

Tutored or taught other university students (paid or voluntary)

AUSSE: Australasian Survey of Student Engagement.

Source: Australian Council for Educational Research (ACER) (2010).

Items are answered on a scale of 1–4, where 1 = ‘never’ and 4 = ‘very often’. Students are instructed to leave the response box blank if the item does not apply.

Evidence of a problem was contained in a report from the analysis of data on the active learning scale provided by the organisation that manages the AUSSE: ‘Notable differences exist in relation to distributed learners’ participation in active learning … mean scores [of distributed learners] are lower than the overall Australasian mean’ (ACER, 2008: 5). Here, ‘distributed learners’ refers to students who reported being enrolled in their course of study by external mode (i.e. distance education) or reported being enrolled in their course of study part-time, undertook at least half of their study online, spent less than 5 hours per week on-campus or worked more than 15 hours each week off-campus (ACER, 2008). Typically, such students are mature-aged, in full-time employment and/or have family responsibilities. Students who live in remote and regional locations are also proportionately more likely to be classified as distributed learners (Richardson and Friedman, 2010). The most important feature of distributed learners is that they are very likely to do much of their learning in the online environment.

The difference between the Active Learning scores of external/distance students and their on-campus (non-distributed) peers reported in ACER (2008) was 10 points after the scores had been scaled out of 100. The pooled standard deviation of the AUSSE Active Learning scores is approximately 15 (based on data from the 2010 AUSSE), so a 10-point difference represents a standardised mean difference (Cohen’s d) of d = 0.67. This is a medium effect size according to the rules of thumb for the standardised mean difference proposed by Cohen (1969). Such a significant difference would be counter to the experience of many educators that distributed students are, on average, no less active than non-distributed students. In fact, there is considerable research demonstrating that distance education delivers equivalent educational experiences and learning outcomes to that of traditional on-campus education (Bernard et al., 2004; Means et al., 2009).

One possible reason for the differences between the two cohorts of students reported by ACER (2008) is that although the word ‘online’ appears twice in the seven items shown in Exhibit 1, the items tend to emphasise activities that students do in or outside of class, as well as emphasise those activities that are more likely to be performed by traditional, school leaver and on-campus students, for example, tutoring other students or volunteering as part of study. Yet, research has shown that students can find online activities highly engaging. For example, Scott et al. (2009) in their analysis of 1866 student responses to a question about the best aspects of their university study found that students valued active uses of information and communication technologies such as online searches and discussions, quizzes and receiving feedback on their learning. The AUSSE/NSSE active learning scale may therefore fail to capture some important online activities in which learners engage. This is especially important since the online environment is becoming more and more important for learners (Allen and Seaman, 2013; Bokor, 2012; Concannon et al., 2005). Measures of active learning that fail to capture important online activities may therefore have weak content validity.

The above discussion points to issues with the validity of a commonly used scale for measuring active learning. The rest of this article reports results from the 2010 AUSSE survey which enabled an investigation of the effects of adding additional items to the active learning scale. The additional items were intended to capture important activities, most in the online learning environment, that all scales designed to measure active learning may need to encompass.

Development of an extended active learning scale

The investigation reported here uses previous work of Carr et al. (2010) that proposed items designed to capture student involvement in highly engaging activities. Carr et al.’s research involved a series of in-depth interviews with students who were asked to describe situations that they had experienced in their university studies in which they felt highly engaged in their learning. From the students’ descriptions of their experiences, Carr et al. (2010) identified activities commonly experienced by students that represented non-passive, engaged learning and which were not encompassed in existing questions in the AUSSE. These activities included students working actively with other students online for learning purposes, students making sustained and concentrated efforts in searching online for learning resources, students participating in activities where they felt highly engaged by the online feedback they received that was either computer or human generated and students persisting in difficult tasks to arrive at a resolution or solution. To reflect these activities, Carr et al. (2010) proposed a number of question items, all of which they considered were related to active learning but that were missing in the existing seven-question AUSSE scale. From these items, four were accepted by the ACER for inclusion in the 2010 AUSSE survey. The four items, as worded in the AUSSE survey, are shown in Exhibit 2.

Exhibit 2.

Additional items for the active learning scale.

In your experience at your institution during the current academic year, about how often have you done each of the following?

Communicated or worked online with other students

Searched online for resources relevant to your studies

Obtained timely online feedback from assessments

Persisted with challenging learning activities despite initial setbacks

The four items are discussed in more detail in the following subsections.

Communicated or worked online with other students

Interpersonal interaction is an important dimension of active learning, and while this new item was related to two existing AUSSE/NSSE items (‘worked with other students on projects during class’ and ‘worked with other students outside of class to prepare assignments’), it explicitly includes the word ‘online’ and does not mention assignments, readings or classes. So the new question may capture additional activities that signal active learning by students, particularly those who study online. For example, it may capture activities where students work collaboratively on learning activities using online technologies (Arbaugh, 2005; Kim et al., 2011), possibly facilitated by social media (Schroeder et al., 2010). Collaboration may also involve information-sharing activities that are not directly related to assignments such as those that involve feed-forward, peer assessment or analysis of the work of other students for self-directed learning (McLuckie and Topping, 2004; Resta and Laferriere, 2007; Thorpe, 2008). Other captured activities may involve communicating feedback or other information from peers to provide context to learning, such as relating ideas to a job or work (Ferguson, 2010). Online technologies may also be used for community building and bonding activities (Ferguson, 2010; Laffey et al., 2006; Rovai, 2007) or peer tutoring and mentoring (McLuckie and Topping, 2004; Smith-Jentsch et al., 2008). Such social interaction is important for learning (Chou and Min, 2009; Laffey et al., 2006) and represents non-passive engagement by students with their learning.

Searched online for resources relevant to your studies

A clear link exists between information literacy, which includes activities such as searching online for resources, and student engagement (e.g. Gratch-Lindauer, 2007; Kezar, 2006; Kuh and Gonyea, 2003; Mark and Boruff-Jones, 2003). In particular, Gratch-Lindauer (2007) reports on findings from a set of 10 experimental items about information literacy that were included in the 2006 NSSE and that showed ‘modest to high positive significant correlations’ (p. 343) with four of the five NSSE benchmark scales, including Active and Collaborative Learning. Kezar (2006) provides some qualitative support for the contention that students can find searching for resources active and engaging with the example of the following quote from a student:

I get bored in the classroom; sometimes faculty are just lecturing at me. What I get really excited about is doing research, and I found the library a great location for looking at archives, learning about key information sites on the Internet, and searching specialised databases. (p. 23)

Similarly, an example from Carr et al. (2010) illustrates the active nature of these types of experiences for students:

With the science one, there was so much stuff and I eventually had to stop. Looked at my watch, it was an hour and a half later. I had to stop, I just need a couple of lesson plans, but I was just like, ‘oh another link’, and ‘oh that looks good’, and … (p. 174)

Item 2 (from Exhibit 2) is designed to capture student engagement in these types of non-passive, self-directed learning experiences.

Obtained timely online feedback from assessments

Feedback often involves interaction between staff and students. The questionnaire used in the 2007 AUSSE contained the following question that captures some aspects of this dimension:

(How often have you) received prompt written or oral feedback from teachers/tutors on your academic performance?

Appropriately, this important aspect of engagement was included in the AUSSE (2007) ‘Staff and Student Interactions’ benchmark scale. However, there are important aspects of feedback that may not directly involve staff–student interaction. For example, considerable research indicates that computer-generated feedback can encourage active learning, especially if it is provided rapidly. An example provided by Carr et al. (2010) describes feelings of engagement that derive from the instantaneous feedback provided by a multiple-choice test activity. Jordan and Mitchell (2009) provide examples of natural language systems where a computer reads and interprets textual material submitted by students and provides appropriate feedback, instantaneously. Even the anticipation of rapid feedback may encourage active engagement (Kettle and Häubl, 2010). Similarly, feedback from peers, rather than staff, such as in a peer-assessment regime (Crossman and Kite, 2012) or through a wiki (Su and Beaumont, 2010), can actively engage students (Price et al., 2007). Such feedback can lead to further student–student interactions (Crossman and Kite, 2012) and is related to improved learning outcomes (Li et al., 2010; Reuse-Durham, 2005). The new question (Item 3, Exhibit 2) is expected to capture these types of valuable active learning experiences that are not teacher- or tutor-initiated but which sustain students’ active engagement in their learning through feedback processes.

Persisted with challenging learning activities despite initial setbacks

Carr et al. (2010) found that persistence with difficult tasks was a common experience reported by students when they felt highly engaged in learning activities, online. Learning activities that are designed to ensure a match between the level of challenge of the task and the skills of the learner engage students by capturing and maintaining their attention (Nakamura and Csikszentmihalyi, 2009). If the challenge of the task is too low, the learner will experience boredom; if the level of challenge is too high, the learner will experience anxiety (Nakamura and Csikszentmihalyi, 2009). Carr et al. (2010) proposed a new item to capture the circumstance where a learner is engaged in an activity that matches their skill set but that they initially find difficult (Exhibit 2). The item is intended to elicit recall of activities where students actively pursue an answer or solution to a difficult problem and experience high engagement because of the balance between task challenge and personal skills. When students persist, they are actively working to construct their knowledge. As noted, earlier, this is essentially the definition the AUSSE gives to the concept of active learning (ACER, 2007). However, none of the existing items on the AUSSE 2007 questionnaire appear to capture the notion of persistence.

The four items discussed above appear to relate to aspects of active learning that are not captured by the seven-item active learning scale in the AUSSE/NSSE; the current research investigates the effects when these items are included in the AUSSE/NSSE Active Learning benchmark scale. No specific claims are made about individual items, and no thorough investigation of the content validity of the scale is carried out. The general research question examined is exploratory in nature, namely, what are the effects on scores when the additional four questions are included into the AUSSE/NSSE active learning scale?

Method

In 2010, 32 Australian universities participated in the AUSSE (ACER, 2010). The survey was conducted in July and August 2010. It was administered to samples of undergraduate students at various year levels by means of both online and paper-based instruments. There were three online versions of the questionnaire. One of the versions included the four new questions and was administered to a subset of students at each participating Australian university.

The four new questions were included as part of Question 1 of the questionnaire. This question comprised approximately 30 items with the same stem: ‘In your experience at your institution during the current academic year, about how often have you done each of the following? Mark your answers in the boxes. Leave blank if the item does not apply’. Students were asked to respond to each scale item on a Likert-type scale with four levels: Never, Sometimes, Often and Very often.

The sample design for the data collection adopted by the ACER included a target response rate of 20% per institution. Actual response rates of institutions varied between 5.9% and 49.8% (ACER, 2010). Approximately 89% of all respondents completed the online version of the questionnaire (ACER, 2010). In total, 5887 students completed questionnaires that contained the additional four active learning questions. Table 1 presents a profile of these respondents.

Table 1.

Profile of respondents.

	Number of respondents
Total number of respondents	5887
Number of Australian universities	32
Location of study
On one or more campuses	5027
Mix of external/distance and on-campus	460
External/distance	400
Proportion of study online
None	1213
About a quarter	2296
About half	1675
All or nearly all	703
Sex
Male	1761
Female	4126
Attendance type
Part-time	733
Full-time	5154
Broad field of education
Natural and physical sciences	658
Information technology	119
Engineering and related technologies	413
Architecture and building	127
Agriculture and environmental studies	84
Health	1311
Education	625
Management and commerce	772
Society and culture	1415
Creative arts	358
Other (not analysed)	5
Socioeconomic status (SES) (based on postcode)
Low SES	1027
Middle SES	2770
High SES	2090

Results

For analysis, the responses to the active learning questions were coded 0 = Never, 1 = Sometimes, 2 = Often and 3 = Very often. Scores for each student were then computed for the following two scales:

AL7: the original seven AUSSE active learning scale questions;

AL11: the seven original questions together with the four new questions.

The scores were computed by taking the mean of the relevant questions, provided the student had responded to at least half of the relevant questions. The score for any student who responded to less than half of the questions was set to ‘missing’. This is the same procedure used by ACER when determining scores for the active learning scale, except that ACER also applies a non-response weighting to scores depending on demographic variables such as gender, socioeconomic status, proportion of study online and attendance type (part-time or full-time). Finally, the difference between the AL7 and AL11 scores was computed for each student.

Figure 1 summarises the mean scores for AL7 and AL11 for the groups of students shown in Table 1. The number shown on each bar is the value of the sample mean. The error bars represent 95% confidence intervals for the population mean. The samples may be subject to some sampling (non-response) bias, so these confidence intervals need to interpreted with some care.

Figure 1.

Mean scores for AL7 and AL11 for various groups of students (with 95% confidence intervals and Cohen’s d).

The following sections highlight some important results illustrated in Figure 1.

Overall increase in scores (‘All’ in Figure 1)

The non-overlapping error bars in the section labelled ‘All’ indicate that there is a significant overall increase in the active learning scales from AL7 to AL11. Overall, there is an increase of 0.19, from 1.19 to 1.38. This represents an increase of approximately 15%. The standard deviation of the differences between the AL7 and AL11 scores (determined from the paired data) was approximately 0.19, so the increase represents a large effect size (Cohen’s d (for a single-sample t-test) = 1.0).

Increase in scores for external/distance students

The increase in the active learning scale score for external/distance students (the fourth pair of bars from the top) is very large. The increase of approximately 0.36, from 0.83 to 1.19, represents a 44% increase. The standard deviation of the differences between the AL7 and AL11 scores (determined from the paired data) for this group of students was approximately 0.21, so an increase of 0.36 indicates a large effect size (Cohen’s d = 1.7). There is a large increase for students studying all or nearly all online (0.28, from 1.08 to 1.36 – see the pair of bars 8th from the top in Figure 1) (s = 0.21, Cohen’s d = 1.3).

Persistence in differences between external/distance/online students and on-campus students

While the greatest increase in scores is for external/distance students, the new (AL11) score for external/distance students is still significantly less than the scores for on-campus students: the difference between the AL11 scores for these two groups is 1.40 − 1.19 = 0.21. The pooled standard deviation of the AL11 scores for these two groups was approximately 0.42, so a difference of 0.21 units represents a medium effect size (Cohen’s d = 0.5).

Increase in scores for all cohorts of students

The error bars do not overlap for nearly all the pairs of bars in Figure 1, no matter which group of students examined: males/females, part-time/full-time, students from different disciplines or students from different socioeconomic backgrounds. In fact, with the stronger analysis using the paired data noted above, all differences are statistically significant at the 0.001 level for all 25 cohorts. (The largest p-value for any of the tests is approximately 10⁻¹³, so the differences remain statistically significant at the 0.001 level if we control for a family-wise error rate using the Bonferroni correction.) Cohen’s d comparing the AL7 and AL11 scores for each of these was at least 0.8, indicating large effect sizes (see values to the right of the bars in Figure 1). These differences indicate that the observed increase in active learning scores when the four new questions are included applies to a very wide variety of students.

While there is an overall increase in the scores for all cohorts of students, the increase is not the same for all cohorts. For example, the scores for students in different disciplines increase by 0.17 for students enrolled in courses in Engineering, Architecture and Building, and Education to 0.21 for humanities students (Society & Culture in Figure 1). With the large sample size, such differences are still statistically significant (p < 0.001), but the effect size is relatively small (Cohen’s d < 0.25), suggesting that any differences between disciplines may not be of practical importance.

Large relative standard deviations for all questions

Analysis was also carried out to determine the extent to which the new questions capture aspects of engagement that vary from student to student. This is important, since a question such as ‘how often did you a use a computer in your studies?’ is not very useful because, although it might indicate active learning, essentially all students are likely to answer ‘Very often’, and the standard deviation of such a question would be close to zero. More useful questions will have relatively large standard deviations. Table 2 summarises the means and standard deviations for each of the 11 items in the AL11 scale. The relatively large standard deviations, including those for the four new questions, indicate that all the questions capture useful information that varies from student to student.

Table 2.

Summary measures for the 11 active learning questions.

Active learning item	Mean	Standard deviation
Asked questions or contributed to discussions in class or online	1.67	0.84
Made a class or online presentation	1.28	0.89
Worked with other students on projects during class	1.54	0.92
Worked with other students outside class to prepare assignments	1.56	0.91
Tutored or taught other university students (paid or voluntary)	0.39	0.73
Participated in a community-based project (e.g. volunteering) as part of your study	0.45	0.79
Discussed ideas from your readings or classes with others outside class (students, family members, co-workers, etc.)	1.61	0.85
Communicated or worked with students online	1.33	0.89
Searched online for resources relevant to your studies	2.37	0.75
Obtained timely online feedback from assessments	1.38	0.79
Persisted with challenging learning activities despite initial setbacks	1.77	0.74

Table 2 also shows that the two highest scoring questions are two of the new questions. Because they both have means that are well above the overall mean for AL11 (46.1), these questions act to increase the overall scores.

Discussion and conclusion

The results presented above, while applied to the AUSSE/NSSE active learning scale for this research, have implications for all measures of active learning. In particular, the significant large increase in the scores on the AUSSE active learning scale that resulted from addition of the four new questions shown in Exhibit 2 (that relate to online activities and the idea of persistence) suggests that care needs to be taken to ensure instruments designed to measure active learning capture important key activities in which students engage. The increase in scores on the AUSSE scale was statistically significant overall and for all cohorts of students with all increases corresponding to large effect sizes. The increase was largest for distance/external students and students studying ‘all or nearly all online’ (Cohen’s d = 1.7 and 1.3, respectively). This suggests that the four new questions capture aspects of active learning that were not captured by the original seven scale items, especially online activities that are most relevant to distance/external students and students studying online. This finding supports those of others such as Scott et al. (2009) (relating to active uses of information and communication technologies) noted earlier. Alternative instruments to measure active learning might need to ensure they also include similar items.

The results comparing the increase in scores for external/distance/online students and on-campus students from the AUSSE survey suggest that caution needs to be exercised when comparing active learning scores on instruments like the AUSSE/NSSE for these groups of students. For example, the results for the AUSSE suggest that the differences between the active learning scores reported in ACER (2008) may be due, in part, to the way in which the concept is measured in the AUSSE/NSSE: the gap between the two groups narrows significantly (with a medium effect size) when the four new items are included.

It is important to note that while the gap between external/distance/online students and on-campus students in the AUSSE survey narrowed when the extra items shown in Exhibit 2 were included in the AUSSE active learning scale, the gap still persisted and was statistically significant and corresponded to a medium effect size. However, almost certainly the gap is still due, in part, to the way active learning is measured in the AUSSE/NSSE. Even with the addition of four new items, the AL11 scale still has an emphasis on activities that students do in or outside of ‘class’ and those that are more likely to be performed by traditional, school leaver and on-campus students. It is also likely that there are other important engaging online activities that have not been captured in AL11. Further research is required to identify such activities and to conduct thorough assessments of the content validity of active learning scales.

Others have observed that the level of student engagement as measured by instruments like the NSSE/AUSSE varies depending upon the discipline of study (e.g. Brint et al., 2008). The findings in the research described in this article lend some support to such observations, but it is important to note that all of the differences between disciplines, while statistically significant, were associated with a small effect size. It is still possible that ‘active learning’ might need to be measured differently in different disciplines, but the current findings suggest that overall discipline-independent measures of active learning may still be useful. Further research is needed to investigate discipline differences.

The assessment of the validity of the AUSSE/NSSE active learning scale conducted for this study shows that including additional items in the scale resulted in significant increases in the scores on that scale. However, had the items been included in another of the AUSSE’s engagement scales, it is possible that the addition of the items may have had a similar impact. Furthermore, while the items were developed from students’ descriptions of occasions when they were highly engaged in their learning – providing justification for the inclusion of these items as part of an active learning scale – further research is needed to determine more accurately whether the additional items best relate to active learning or some other form of engagement. Such research could involve the use of a panel of teachers or similar experts (as recommended by Crocker and Algina, 1986).

As discussed in the first section of this article, active learning is a broad concept covering many different types of learning strategies. Useful measures of the concept that apply across, and allow comparisons between, many different types of students therefore need to include items that capture a good selection of active learning activities that students employ. In particular, the research reported in this article highlighted the importance of capturing some highly engaging activities associated with learning online or that relate to persistence by demonstrating significant increases in benchmark scale scores when items designed to capture such activities are incorporated. These activities are especially important for students learning in the online environment, which, as previously noted, is becoming increasingly important for most, if not all, learners. Comprehensive measures of active learning should include these types of items.

Footnotes

Acknowledgements

The authors wish to thank staff at the Australian Council for Educational Research, in particular Hamish Coates and Ali Radloff, for their support. The authors also wish to thank staff in Deakin University’s Strategic Intelligence and Planning Unit for supporting incorporation of the new questions into the 2010 AUSSE

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was financially supported by Deakin University in the earlier phases of this research project.

Author biographies

Rodney Carr is a Senior Lecturer and Director of the Doctor of Business Administration programme in the Faculty of Business and Law at Deakin University. His research interests include student engagement and knowledge creation and sharing in and between small businesses. Address: Faculty of Business and Law, Deakin University, PO Box 423, Warrnambool, VIC 3280, Australia. [email: rodney.carr@deakin.edu.au]

Stuart Palmer is Associate Professor of Integrated Learning in the Faculty of Science, Engineering and Built Environment at Deakin University. His research interests include frequency domain image analysis, student evaluation of teaching and the effective use of digital/online technologies in learning and teaching. Address: Faculty of Science, Engineering and Built Environment, Deakin University, 75 Pigdons Road, Waurn Ponds, VIC 3126, Australia. [email: stuart.palmer@deakin.edu.au]

Pauline Hagel is an Adjunct Associate Professor at the Education Institute at the University of Canberra. Her research interests include student university choice, student engagement, outcomes evaluation and improving the access and retention of equity groups. Address: University of Canberra, University Drive, Bruce, ACT 2617, Australia. [email: pauline.hagel@gmail.com]

References

Allen

Seaman

(2013) Changing course: Ten years of tracking online education in the United States, Babson survey research group. Available at: www.onlinelearningsurvey.com/reports/changingcourse.pdf (accessed 24 June 2014).

Arbaugh

(2005) Is there an optimal design for on-line MBA courses? Academy of Management Learning & Education 4(2): 135–49.

Australian Council for Educational Research (ACER) (2007) AUSSE 2007 development and methodology. Available at: http://web.archive.org/web/20091010131234/http://www.acer.edu.au/documents/AUSSE_2007DevelopmentAndMethodology.pdf (accessed 24 June 2014).

Australian Council for Educational Research (ACER) (2008) Enhancing the engagement of distributed learners. Research Briefing 3. Available at: http://research.acer.edu.au/cgi/viewcontent.cgi?article=1008&context=ausse (accessed 24 June 2014).

Australian Council for Educational Research (ACER) (2010) Australasian Survey of Student Engagement 2010 institution report. Available at: http://www.acer.edu.au/files/AUSSE_2010_Institution_Report.pdf (accessed 24 June 2014).

Bernard

Abrami

Lou

. (2004) How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research 74(3): 379–439.

Bokor

(2012) University of the Future: A Thousand Year Old Industry on the Cusp of Profound Change. Ernst & Young. Available at: http://www.ey.com/Publication/vwLUAssets/University_of_the_future/$FILE/University_of_the_future_2012.pdf (accessed 24 June 2014).

Brint

Cantwell

Hanneman

(2008) The two cultures of undergraduate academic engagement. Research in Higher Education 49(5): 383–402.

Carr

Hagel

Hellier

(2010) Measuring student engagement: Using ‘flow’ theory to guide question development. In: Refereed papers from the 33rd HERDSA annual international conference, Melbourne, VIC, Australia, 6–19 July, pp. 167–77. Milperra, NSW, Australia: HERDSA. Available at: http://www.herdsa.org.au/wp-content/uploads/conference/2010/papers/HERDSA2010_Carr_R.pdf (accessed 24 June 2014).

10.

Chi

MTH

(2009) Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science 1(1): 73–105.

11.

Chickering

Gamson

(1987) Seven principles for good practice in undergraduate education. AAHE Bulletin 39(7): 3–7.

12.

Chou

Min

(2009) The impact of media on collaborative learning in virtual settings: The perspective of social construction. Computers & Education 52(2): 417–31.

13.

Cohen

(1969) Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press.

14.

Concannon

Flynn

Campbell

(2005) What campus-based students think about the quality and benefits of e-learning. British Journal of Educational Technology 36(3): 501–12.

15.

Crocker

Algina

(1986) Introduction to Classical & Modern Test Theory. New York: Holt, Rinehart & Winston.

16.

Cronbach

(1971) Test validation. In: Thorndike

(ed.) Educational Measurement, 2nd edn. Washington, DC: American Council of Education, pp. 443–507.

17.

Crossman

Kite

(2012) Facilitating improved writing among students through directed peer review. Active Learning in Higher Education 13(3): 219–29.

18.

Dyson

Litchfield

Lawrence

. (2009) Advancing the m-learning research agenda for active experiential learning: Four case studies. Australasian Journal of Educational Technology 25(2): 250–67.

19.

Ferguson

(2010) Peer interaction: The experience of distance students at university level. Journal of Computer Assisted Learning 26(6): 574–84.

20.

Fitzpatrick

(1983) The meaning of content validity. Applied Psychological Measurement 7(1): 3–13.

21.

Ford

(2010) Critique in academic disciplines and active learning of academic content. Cambridge Journal of Education 40(3): 265–80.

22.

Gratch-Lindauer

(2007) Information literacy-related student behaviors. College & Research Libraries News 68(7): 432–41.

23.

Hagel

Carr

Devlin

(2012) Conceptualising and measuring student engagement through the Australasian Survey of Student Engagement (AUSSE): A critique. Assessment and Evaluation in Higher Education 37(4): 475–86.

24.

Hendriks

Maor

(2004) Quality of students’ communicative strategies delivered through computer-mediated communications. Journal of Interactive Learning Research 15(1): 5–32.

25.

Jordan

Mitchell

(2009) e-Assessment for learning? The potential of short-answer free-text questions with tailored feedback. British Journal of Educational Technology 40(2): 371–85.

26.

Kettle

Häubl

(2010) Motivation by anticipation: Expecting rapid feedback enhances performance. Psychological Science 21(4): 545–7.

27.

Kezar

(2006) Librarians enhancing student engagement: Partners in learning that build bridges. In: Gibson

(ed.) Student Engagement and Information Literacy. Chicago, IL: American Library Association, pp. 16–32.

28.

Kim

Hong

Bonk

. (2011) Effects of group reflection variations in project-based learning integrated in a Web 2.0 learning space. Interactive Learning Environments 19(4): 333–49.

29.

Kuh

Gonyea

(2003) The role of the academic library in promoting student engagement in learning. College and Research Libraries 64(4): 256–82.

30.

Laffey

Lin

(2006) Assessing social ability in online learning environments. Journal of Interactive Learning Research 17(2): 163–77.

31.

Laird

TFN

Kuh

(2005) Student experiences with information technology and their relationship to other aspects of student engagement. Research in Higher Education 46(2): 211–33.

32.

Liu

Steckelberg

(2010) Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology 41(3): 525–36.

33.

McLuckie

Topping

(2004) Transferable skills for online peer learning. Assessment and Evaluation in Higher Education 29(5): 563–84.

34.

Mark

Boruff-Jones

(2003) Information literacy and student engagement: What the national survey of student engagement reveals about your campus. College and Research Libraries 64(6): 480–93.

35.

Means

Toyama

Murphy

. (2009) Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Washington, DC: US Department of Education.

36.

Moore

(1989) Three types of interaction. The American Journal of Distance Education 3: 1–7.

37.

Nakamura

Csikszentmihalyi

(2009) Flow theory and research. In: Snyder

Lopez

(eds) The Oxford Handbook of Positive Psychology, 2nd edn. New York: Oxford University Press, pp. 195–206.

38.

National Survey of Student Engagement (NSSE) (2011) Annual results 2011: Fostering student engagement campuswide. Available at: http://nsse.iub.edu/NSSE_2011_Results/index.cfm (accessed 24 June 2014).

39.

Neuman

(2011) Social Research Methods: Qualitative and Quantitative Approaches, 7th ed. Boston, MA: Allyn & Bacon.

40.

O’Loughlin

(1992) Rethinking science education: Beyond Piagetian constructivism toward a sociocultural model of teaching and learning. Journal of Research in Science Teaching 29(8): 791–820.

41.

Petress

(2008) What is meant by ‘Active Learning?’. Education 128(4): 566–9.

42.

Powner

Allendoerfer

(2008) Evaluating hypotheses about active learning. International Studies Perspectives 9(1): 75–89.

43.

Price

O’Donovan

Rust

(2007) Putting a social-constructivist assessment process model into practice: Building the feedback loop into the assessment process through peer review. Innovations in Education and Teaching International 44(2): 143–52.

44.

Quinton

Smallbone

(2010) Feeding forward: Using feedback to promote student reflection and learning – A teaching model. Innovations in Education and Teaching International 47(1): 125–35.

45.

Rae

Cochrane

(2008) Listening to students. Active Learning in Higher Education 9(3): 217–30.

46.

Resta

Laferriere

(2007) Technology in support of collaborative learning. Educational Psychology Review 19(1): 65–83.

47.

Reuse-Durham

(2005) Peer evaluation as an active learning technique. Journal of Instructional Psychology 32(4): 328–45.

48.

Richardson

Friedman

(2010) Australian Regional Higher Education: Student characteristics and experiences. Research Report. Available at: http://research.acer.edu.au/higher_education/22 (accessed 24 June 2014).

49.

Rovai

(2007) Facilitating online discussions effectively. Internet and Higher Education 10(1): 77–88.

50.

Schroeder

Minocha

Schneider

(2010) The strengths, weaknesses, opportunities and threats of using social software in higher and further education teaching and learning. Journal of Computer Assisted Learning 26(3): 159–74.

51.

Scott

Grebennikov

Gozzard

(2009) ICT-enabled learning: The student perspective. Journal of Institutional Research 14(2): 1–16.

52.

Smith-Jentsch

Scielzo

Yarbrough

. (2008) A comparison of face-to-face and electronic peer-mentoring: Interactions with mentor gender. Journal of Vocational Behavior 72(2): 193–206.

53.

Beaumont

(2010) Evaluating the use of a wiki for collaborative learning. Innovations in Education and Teaching International 47(4): 417–31.

54.

Thorpe

(2008) Effective online interaction: Mapping course design to bridge from research to practice. Australasian Journal of Educational Technology 24(1): 57–72.