A comparative study of three teaching methods on student information literacy in stand-alone credit-bearing university courses

Abstract

Three teaching methods, applied to credit-bearing information literacy (IL) university courses, were evaluated and compared. The effects of lecture-based learning (LBL), project-based learning (PjBL) and problem-based learning (PBL) were investigated using the information literacy test (ILT) as an assessment tool, with regard to the total ILT score, specific IL contents according to the five ACRL standards and students’ mental skills according to the Bloom’s cognitive categories. While all three teaching methods showed a significant improvement in the ILT post-test, the active-learning groups of PjBL and PBL scored significantly better than the LBL group. The most notable positive difference was observed in students’ effective access to information related to database searching skills, in the intellectual property/ethics issues and in the cognitive category of comprehension. The PjBL and PBL post-test results did not differ significantly, indicating that both active learning methods resulted in similar improvements of students’ IL.

Keywords

assessment information literacy teaching methods

1. Introduction

In our previous work [1], an information literacy test (ILT) for higher education was developed, tested and validated. The ILT was applied as a measuring instrument to assess the level of information literacy (IL) of university students. In the continuation of our research, described in the present article, we explored the effects of different teaching methods, applied in credit-bearing IL courses at university level, on the general level of IL, on knowledge and skills according to the individual IL standards in higher education, and on the cognitive levels of IL. Although the assessment of teaching methods in IL has been researched before [2 –4], the present article takes a new approach in analysing the impact of teaching methods on IL content and cognitive skills, as well as directly comparing two active teaching methods with a traditional teaching approach.

The research was performed within stand-alone credit-bearing IL classes. From a pedagogical perspective, such classes deserve attention in their own right, not just from the standpoint of improving learning in other disciplines [5].

The three teaching methods compared were lecture-based learning, project-based learning and problem-based learning. The first of these is also called traditional or teacher-centred, and the latter two belong to the group of active or learner-centred learning methods.

1.1. Information literacy and cognitive levels

The definition of IL used in this article is based on a model by the Association of College and Research Libraries (ACRL, part of the American Library Association) [6], which specifies that information literacy is ‘a set of abilities requiring individuals to recognise when information is needed and have the ability to locate, evaluate, and use effectively the needed information’.

Information Literacy Competency Standards for Higher Education [6] were developed by the ACRL to provide a framework for assessing the information literate individual, who must be able to:

determine the extent of information needed;

access the needed information effectively and efficiently;

evaluate information and its sources critically and incorporate selected information into one’s knowledge base and value system;

use information effectively to accomplish a specific purpose;

understand the economic, legal and social issues surrounding the use of information, and access and use information ethically and legally.

Recently, the ACRL adopted the Framework for Information Literacy in Higher Education [7] to upgrade/replace the existing IL standards and associated learning objectives with the threshold concept of frames, defined by knowledge practices/abilities and dispositions. Although some suggestions for the application of the new framework have already been made [8], our paper still uses for the analytical purposes the structure of ACRL standards, as they contain precise and well-structured descriptions of students’ performance indicators and expected outcomes.

According to Bloom’s taxonomy of learning domains [9], the cognitive domain involves knowledge and the development of intellectual skills. This includes the recall or recognition of specific facts, procedural patterns and concepts that serve in the development of intellectual abilities and skills. There are six major cognitive categories of increasing difficulty:

knowledge/remembering (retrieving knowledge from memory);

comprehension (extracting meaning from messages);

application (using a procedure in a situation);

analysis (determining the relationship of parts in a whole);

synthesis (generalisation, specialisation);

evaluation (value, decision-making, ideas).

It has been established by Wiener [10] that the cognitive functions enunciated by Bloom can be translated into tasks by using the attributes as portals to additional descriptors or actions. Reece [11] claimed that the ACRL standards are connected with Bloom taxonomy levels, and that, in learning to evaluate a source, students develop the highest taxonomic skill on Bloom’s scale. It was recommended by Keene et al. [12] to choose an IL teaching method based on the cognitive skills required. For the execution of mechanical skills, a mixture of lectures, demonstrations, on-line tutorials and worksheets was suggested as suitable; for comprehension and tasks involving higher-order cognitive skills (analysis, synthesis, evaluation), the inclusion of student-centred activities (active learning) was recommended. Walton and Hepworth [13] also found that a number of cognitive states (knowledge and application) were evident at the face-to-face stage; however, the higher cognitive states of analysis, synthesis and evaluation appeared to be most evident in those students who have experienced the online social network learning process. In a review discussing searching as learning, Rieh et al. [14] associated types of searching strategies with cognitive learning modes and linked them with Bloom’s cognitive categories. Harmer and Lee [15] noticed that student-centred learning had a prominent role in the new ACRL Framework and presented a map of abilities and skills, as defined in the standards, with regard to the frames, also in view of Bloom taxonomy.

1.2. Problem- and project-based learning

The beginnings of problem-based learning (PBL) date back to the 1960s in medicine, where PBL is still most widely used, although it has since been applied in many other disciplines. Given a problem, learning occurs in a three-phase process, beginning with a tutor-supervised collaborative phase where learning needs are identified, followed by self-directed study and concluded by collaborative knowledge application and summation [16]. PBL is consistent with the principles of instruction arising from constructivism [17]. The premise of PBL is that new knowledge is constructed on the basis of what is already known [18]. A great deal of emphasis is placed on the tutor’s role as the facilitator of learning, the self-direction and self-regulation responsibility of students, and the design process of suitable problems [19].

The older of the two active methods, project-based learning (PjBL), was first used back in 1921. Blumenfeld et al. [20] define PjBL as a comprehensive perspective focused on teaching by engaging students in investigation. There are two essential components of projects: they require a question or problem that serves to organise or drive activities; and the activities result in a series of artefacts or products that culminate in a final product that addresses the driving question. According to Savery [19], the objective is to design a product based on a given set of specifications. Learning is more oriented towards following procedures. Problems encountered are tackled by teacher intervention through providing expertise, feedback and suggestions. In project-based learning, considerable emphasis is placed on time/resource management and role differentiation.

1.3. PBL and PjBL in IL teaching

When speaking about PBL/PjBL research in relation to IL, it is necessary to differentiate between studying active approaches to teaching IL (as in the present article) and the impact that active methods applied in teaching a specific discipline have on IL. Many authors, such as Kenney [21], limit the application of PBL to one-shot IL lessons or subject courses with integrated IL instruction [22, 23], which differs from our approach, in which the method was applied throughout a stand-alone, credit-bearing IL course. Applications of problem-based learning in IL teaching often centre on information-retrieval tasks in which real-life problems that can be solved by database searching are presented, or such problems are simulated. However, this is not sufficient. Macpherson [24] states that successful information retrieval requires not only teaching question analysis, a problem-solving heuristic and search strategy design, but also teaching the concepts and skills associated with the broader spectrum of information literacy: critical thinking skills, involving evaluation of sources for credibility and relevance. Various authors point out the advantages of using PBL for teaching IL and give examples of good practices. Besides presenting a number of cases of the application of PBL, Snavely [25] adds learning by doing as another advantage. Wenger [26] suggests that more time in IL instruction should be spent on helping to articulate students’ information needs, even at the expense of the time spent searching library resources. Spence [27] emphasises the importance of collaborative learning through practical activity, aligning PBL with sociocultural theories of learning. Diekema et al. [28] suggest that PBL might offer another, less discipline-centred strategy for introducing students to a more situated experience of IL. Cook and Klipfel [29] provide specific principles and strategies for promoting knowledge retention and transfer within student-centred IL instruction. As well as being beneficial when used for IL instruction, PBL has a positive effect on IL when used in teaching other subjects. According to Dodd [30], the introduction of PBL in the curriculum has increased the need for information skills and resulted in students visiting the library earlier, having a better understanding of how to use resources, becoming more discerning regarding information and learning how to integrate information effectively. There is, however, another connection between PBL and IL, namely the similarity between the PBL methodology and the definition of IL according to the ACRL, as has been pointed out by Smith Macklin [31]. The abilities of fact gathering based on prior knowledge, knowledge need identification, problem statement and solution hypothesis, information search and result evaluation in PBL are comparable to the abilities of recognising needs and locating, evaluating and using information in the ACRL.

Despite active learning methods becoming more and more commonplace in IL instruction in all types of learning and on all educational levels, few researchers have compared teaching methods directly. In IL instruction for engineering students, problem- and lecture-based approaches were compared by Hsieh and Knight [32], with the former proving to be more effective. Another research project [2] compared PBL (situated learning) with the classical approach in distance IL teaching, finding that PBL increases the possibility of knowledge transfer. Classical versus problem-solving approaches to IL instruction integrated into science learning were also studied by Chen et al. [33]. The problem-based approach proved to be better for comprehension and solving problems, while the classical approach was better for memorising. In comparing passive (traditional) and active methods of IL instruction in terms of the effect on students, it was found [34] that active instruction had a direct effect on achieving positive student results, such as decreased anxiety or increased self-efficacy, improved perception of library resources, time savings and effort reduction in finding information, while passive instruction did not yield such benefits. Boustany [35] reported that students’ self-assessments showed seemingly increasing difficulties with IL tasks over their years of study, possibly indicating that students become more critical of their own skills with each passing year. A phenomenographical study by Maybee revealed that students experience information use in a complex, multi-tiered way. To enhance their IL, educators should be attempting to guide learners to conceptualise information use in a variety of ways [36]. Two active learning methods for IL teaching—guided and self-directed instruction—were compared by Schroeter and Higgins [37], with the former method proving to be more effective than the latter. The active learning approach was not always more successful. In another study, a comparison of two active methods and traditional IL teaching [3] found that the traditional method was slightly more successful than the active methods. In recent research, Hsieh et al. [4] compared four teaching methods, including the traditional approach and active learning in IL instruction, regarding the first two ACRL standards. An improvement was observed for all methods, with no substantial differences between the methods. Regarding the effect of PBL on IL, traditional teaching was compared with PBL in a subject-based course by Eskola [38] in terms of the type of IL skills acquired during the course (developed, simple and undeveloped). It was found that developed skills appeared more often when PBL was used. Other active teaching styles have also been employed in IL education; for example, a task-centred approach, involving more direct instruction than the problem-based approach [39]. In addition to investigating methods, research has also been focused on the mode of delivery of IL instruction, with e-learning modules being developed [28] as well as the integration of e-learning into traditional models, such as in blended learning [40].

2. Motivation and research objectives

The above review of available literature reveals the advantages of active methods over traditional methods of IL instruction [2, 32], as well as cases to the contrary [3], but no superiority on the content level, based on individual ACRL standards, was reported [4]. Comparisons on the cognitive level [33, 38], albeit with PBL in a slightly different setting (the effect of IL on subject teaching, the effect of subject teaching on IL), also found that PBL was better in higher cognitive categories. To our knowledge, no research has yet been conducted comparing the efficiency of PjBL, PBL and traditional lecture-based learning (LBL) according to all five ACRL standards, along with comparison of the cognitive levels of the Bloom taxonomy. In addition, we were interested in the differences between the two active learning approaches: PjBL and PBL. These active methods were chosen because they have been used in our IL curriculum but have not been directly compared and statistically evaluated. We aimed to identify the IL topics and cognitive levels of IL in which each method was the most efficient, in order to make recommendations for adjusting the existing IL curriculum and methodologies so as to maximise the efficiency of learning.

Therefore, comparisons of the three teaching methods were carried out through the pre-test and post-test IL assessment analysis, taking into account the overall ILT result, as well as performance in groups of items according to subscale groupings of the ACRL standards and Bloom’s cognitive categories.

2.1. Working hypotheses

It was anticipated that the effectiveness of all of the teaching methods would be good, but some differences were expected between the methods in total and subscale analyses, based on the results of previous research [32, 33]. The following hypotheses were formulated according to the objectives of the study:

Total ILT score:

There are no statistically significant initial differences in the pre-test ILT total scores between the LBL, PjBL and PBL groups.

There is a significant improvement in the total ILT score (post-test versus pre-test) for all three teaching methods.

ILT total scores in the post-test are higher in the PjBL and PBL groups than in the LBL group (assumption based on [32]). There is a difference between the PBL and PjBL groups.

Subscale analysis – groupings according to the subscales of (1) five ACRL standards and (2) three cognitive levels of the Bloom taxonomy:

In the pre-test, there is no statistically significant difference between the LBL, PjBL and PBL groups.

There is a significant improvement in all subscales for all teaching methods.

There are some differences evident between the three teaching methods in the improvement of individual subscales (assumption based on [33], contrary to [3]).

3. Materials and methods

3.1. Test groups of students

A total of 245 of students from the University of Ljubljana, enrolled in life sciences and health study programmes, participated in the study in the period from February 2014 to June 2015. The students were selected on the basis of their enrolment in compulsory credit-evaluated courses in informatics, implemented by the authors of this study, at the Biotechnical Faculty, the Faculty of Health and the Faculty of Education of the University of Ljubljana, Slovenia. The IL content followed the ACRL standards for higher education [6], which were adopted and translated to the national language [41]. The study was performed with three experimental groups that were formed according to the three different teaching approaches applied in the courses:

LBL – 94 students, 38.4%;

PjBL – 101 students, 41.2%;

PBL – 50 students, 20.4%.

3.2. Teaching methods in the LBL, PjBL and PBL groups of students

The courses in informatics, with the IL content adapted to the field of study, were delivered by two university professors (co-authors of this article), both with a PhD in life sciences, with additional specialisation in information science, and with more than 20 years of university teaching experience. All of the courses were compulsory, bearing 3 credit points (45 contact hours), and covered IL topics according to the ACRL standards of higher education. All of the students had access to university computers and the Internet in the university computer rooms, and used the same professional databases with access according to university licenses. In the LBL group, the lectures were organised in a formal or traditional way, following the linear, predefined consecutive order of chapters from the curriculum. Presentations and explanations of practical examples were given by the professor during lectures, while the students’ practical work consisted of predesigned individual exercises in a computer room, with each student working with databases on a specific topic.

In the PjBL group, the lectures were organised mainly as structured explanations and instructions to support the project work. In most cases, the topics followed the predefined consecutive order of chapters from the curriculum, but at times adjustments were needed in order to proceed with the project work. Presentations and explanations of practical examples were given both by the professor during the lectures and by the students during shared discussions. The students’ practical work consisted of individual projects to be completed by the end of the course, with practical tasks that followed a general working protocol. The students could select the topics of their individual projects from a range of medicinal plants and fungi or from the recycling of various waste materials. Their main project task was to prepare a review article. In order to achieve this goal, they had to define the search terms, combine them into search queries, process the databases, evaluate and analyse the collected literature, and synthesise the information into a review article according to the practices of scientific writing and referencing.

In the PBL group, the students had to participate in solving a problem around the issue of drinking water. IL lectures were organised mainly as interventions, explanations, instructions or answering students’ questions in order to facilitate the solving of sub-problems or the completion of tasks in the PBL course. A non-linear approach was therefore applied, not following the common predefined consecutive order of IL topics from the curriculum. The main problem (Which water is better to drink – bottled, tap or spring?) was first discussed within the group and then broken down into individual tasks and sub-problems to be studied, presented and discussed by the students. The practical work followed the tasks and the group dynamics in solving the problem. During the learning process, the students performed similar practical tasks: definition of sub-problems and search terms, preparation of search queries, employment of queries in databases, evaluation and analysis of documents, and synthesis of information into a project report that was submitted in a written form as well as being presented orally within the group discussion.

3.3. Research instruments and testing procedure

The ILT [1] was applied as the main IL measuring instrument. The test consists of 40 multiple choice questions with four possible answers, only one of which is correct. The electronic version of the ILT was accessible through the open access survey system 1ka (http://www.1ka.si/). A unified introductory protocol was applied before testing, explaining the purpose of the study, ensuring voluntary participation and anonymity, providing detailed instructions and expressing gratitude for the students’ participation. Each student was tested twice: prior to taking the course (pre-test) and after taking the course (post-test). Both pre-testing and post-testing took place at university locations, in presence of a professor. There was no time limit for completing the test.

3.4. Subscales

The ILT questions were classified into two sets of subscales: five subscales according to the five ACRL standards (denoted by A1 through A5), and three subscales according to Bloom’s cognitive levels (Bloom1 through Bloom3). Due to the fact that the ILT is a multiple choice test, making it less suitable for assessing higher cognitive levels, application and higher levels were all included in the category Bloom3. In both groupings, the categorisation of the ILT questions was performed by a group of four specialists with experience in IL and education (for ACRL standards), and in science, pedagogy and psychology (for Bloom’s categories). The final decisions were agreed by consensus.

3.5. Statistical tools and analyses

Statistical analyses were carried out for comparisons of the three student groups based on the teaching methods LBL, PjBL and PBL. The analyses performed on the total ILT score and on the subscale scores (pre-test and post-test or pre-test versus post-test) included:

Descriptive statistics (mean and standard deviation; standard error of the mean and variance only for the total score) for each method group;

Paired samples t-test (pre-test versus post-test progress for each method group);

One-way ANOVA (differences between the three method groups);

Independent samples t-test (pairwise differences between the three method groups).

The Statistical Package for the Social Sciences (SPSS^®, IBM, version 22) was used in the data analyses.

4. Results

The pre-test and post-test results of the LBL, PjBL and PBL student groups were uniformly coded and compared; pre-test versus post-test analyses were carried out for total ILT scores and for ILT subscale scores, according to the two subscale item groupings: ACRL and Bloom.

4.1. Total ILT score

Descriptive statistics on ILT scores (Table 1) and score mean comparison (Figure 1) show that all of the groups of students improved in the post-test. While the three groups exhibited similar IL levels in the pre-test, a bigger separation of the group means in the post-test points to differences between the groups in terms of IL levels.

Table 1.

Descriptive statistics for ILT pre-test and post-test scores (%) by teaching methods

Group	ILT score	n	Mean	Std. error of mean	Std. deviation	Variance
LBL	Pre-test	94	68.22	1.26	12.22	149.41
	Post-test	94	77.39	1.17	11.34	128.62
PjBL	Pre-test	101	66.01	1.28	12.91	166.65
	Post-test	101	82.85	0.77	7.70	59.25
PBL	Pre-test	50	67.00	1.38	9.75	95.15
	Post-test	50	84.55	0.84	5.91	34.87

Figure 1.

ILT pre-test and post-test score means (%) by teaching methods (Error bars: +/– 2 std. error of mean).

The paired samples t-test (Table 2) showed that in all three teaching methods the improvement was statistically significant, with the biggest improvement achieved in the PjBL group (17.6%), followed by the PBL group (16.8%) and the LBL group (9.2%).

Table 2.

T-test on post-test improvement by teaching method

Group	Mean diff.	df	t	P <	d	Confidence interval
						Lower limit	Upper limit
LBL	9.2	93	11.1	0.001	0.79	7.5	10.8
PjBL	16.8	100	16.5	0.001	1.57	14.8	18.9
PBL	17.6	49	16.1	0.001	2.16	15.4	19.7

One-way ANOVA was used to measure group differences in the pre-test and post-test. No significant effect of the teaching method was found on the pre-test at the P <0.05 level (F(2, 242) = 0.813, P = 0.445, eta sq. = 0.007). In contrast, the post-test analysis revealed a significant effect of the teaching method (F(2, 242) = 13.551, P <0.001, eta sq. = 0.101). A subsequent t-test for independent samples (Table 3), used to compare pairs of groups, found a significant difference between the LBL and PjBL groups (5.5%) and between the LBL and PBL groups (7.2%). The difference between the PjBL and PBL groups (1.7%) was too small to be of statistical significance.

Table 3.

T-test on pairs of teaching methods in the post-test

Group pair	Mean diff.	df	t	P <	d	Confidence interval
						Lower limit	Upper limit
LBL-PjBL	5.5	162.1	3.9	0.001	0.54	2.7	8.2
LBL-PBL	7.2	142.0	5.0	0.001	0.78	4.3	10.0
PjBL-PBL	1.7	123.3	1.5	0.135	0.25	−0.5	3.9

Total ILT score analysis revealed that the project- and problem-based approaches achieved similar results and were both more effective than the classic lecture approach.

4.2. Subscale analysis – ACRL grouping

Item subscales were formed by mapping the 40-test items with the five ACRL standards [1]. The number of items in each subscale was 15, 10, 5, 3 and 7, respectively. Subscale scores were examined.

Paired sample t-tests, conducted for each subscale in all three student groups, reveal a significant improvement in all ACRL subscales for all teaching methods (Table 4). Effect sizes (d) are a magnitude higher in the PjBL and PBL groups than in the LBL group, and lower in the subscale A3 (critical evaluation of information) than in the other subscales.

Table 4.

Paired samples t-tests for grouping ACRL (pre-test vs. post-test)

Group	Subscale	Pre-test		Post-test		df	t	P <	d	Mean diff.	Confidence interval
		M	SD	M	SD						Lower limit	Upper limit
LBL	A1 – needs	73.1	14.6	82.1	10.9	93	6.6	0.001	0.68	8.9	6.2	11.6
	A2 – access	57.1	16.6	69.0	16.5	93	6.3	0.001	0.71	11.9	8.2	15.7
	A3 – evaluation	85.7	21.1	90.0	18.5	93	2.3	0.025	0.20	4.3	0.6	8.0
	A4 – use	81.2	24.7	91.8	17.4	93	4.5	0.001	0.66	10.6	5.9	15.4
	A5 – ethics	55.5	18.5	64.1	19.6	93	4.3	0.001	0.44	8.7	4.7	12.7
PjBL	A1 – needs	67.7	14.9	81.4	11.2	100	10.4	0.001	1.07	13.7	11.1	16.4
	A2 – access	57.8	18.2	82.5	11.7	100	13.8	0.001	1.57	24.7	21.1	28.2
	A3 – evaluation	83.6	21.2	95.2	9.9	100	5.8	0.001	0.70	11.7	7.7	15.7
	A4 – use	77.6	25.4	96.0	10.8	100	7.1	0.001	0.99	18.5	13.3	23.6
	A5 – ethics	56.7	18.5	72.0	15.1	100	7.6	0.001	0.83	15.3	11.3	19.3
PBL	A1 – needs	68.4	13.1	83.3	9.0	49	9.1	0.001	1.30	14.9	11.6	18.2
	A2 – access	61.6	15.4	86.2	12.4	49	10.3	0.001	1.77	24.6	19.8	29.4
	A3 – evaluation	81.6	20.9	93.6	11.0	49	4.7	0.001	0.73	12.0	6.9	17.1
	A4 – use	83.3	20.5	98.7	6.6	49	5.0	0.001	1.12	15.3	9.2	21.4
	A5 – ethics	54.3	16.1	72.3	10.6	49	7.7	0.001	1.41	18.0	13.3	22.7

Subscale score means are shown in Figure 2, where the pre-test and the post-test scores are superimposed. Across all groups, the lowest pre-test scores (between 54% and 62%) were observed in the subscales A2 (information access/searching skills) and A5 (intellectual property/ethics issues), and the highest (between 77% and 82%) in the subscales A3 (critical evaluation of information) and A4 (effective use of information).

Figure 2.

Pre-test and post-test score means (%) by ACRL subscales and teaching methods.

The pre-test–post-test mean differences (Table 4), also visible as improvement in Figure 2, imply that the PjBL and PBL groups achieved similar levels of improvement in the post-test, with the LBL group lagging behind in every subscale. In searching skills, information evaluation and intellectual property topics, the progress of the LBL group was only half that of the active groups. All of the groups achieved the most progress in searching skills, with nearly 25% in the PjBL and PBL groups, compared to 12% in the LBL group. A difference was expected, as the active learning groups had more authentic experience in database searching than the lecture-based group. The least progress was achieved in critical evaluation skills, with a less than 5% increase in the LBL group, compared to 12% in the active learning groups.

One-way ANOVA performed next found no significant differences between the methods in the pre-test in any of the subscales, with the exception of subscale A1 (information need identification), where the LBL group was better than the active groups. This subscale was excluded from the post-test comparisons.

Pairwise comparison of the methods in the post-test was conducted using a t-test for independent samples. The results shown in Table 5 demonstrate significant differences between the LBL and PjBL groups in every subscale and between the LBL and PBL groups in every subscale except for the evaluation of information (despite the difference in progress nearing 8%). No significant difference was found in any of the subscales between the methods PjBL and PBL. The difference between LBL and active approaches was most pronounced in searching skills.

Table 5.

T-test of independent samples – significance (P) and effect size (d) of pair differences on the post-test by ACRL subscales

Group pair	Subscale	Mean diff.	P <	d
LBL*-PjBL	A2 – access	13.4	0.001	0.95
	A3 – evaluation	5.2	0.016	0.35
	A4 – use	4.2	0.047	0.29
	A5 – ethics	7.9	0.002	0.45
LBL*-PBL	A2 – access	17.2	0.001	1.13
	A3 – evaluation	3.6	0.146	0.22
	A4 – use	6.8	0.001	0.47
	A5 – ethics	8.2	0.001	0.48
PjBL-PBL	A1 – needs	1.9	0.285	0.18
	A2 – access	3.7	0.073	0.31
	A3 – evaluation	−1.6	0.354	−0.16
	A4 – use	2.6	0.068	0.28
	A5 – ethics	0.3	0.903	0.02

Subscale A1 was excluded.

4.3. Subscale analysis – Bloom grouping

At this stage, subscales were created by mapping the ILT items with the first three cognitive levels according to Bloom’s taxonomy. The number of items in the subscales was 13, 16 and 11. The teaching methods were compared by analysis of subscale scores.

Paired sample t-tests on each subscale and teaching method (Table 6) show significant improvement in all three subscales – for all of the teaching methods. All effect sizes in the PjBL and PBL groups were large. In the LBL group, only the Bloom3 (knowledge application) subscale had a large effect size; Bloom1 (knowledge) had a moderate effect size and Bloom2 (comprehension) a small effect size.

Table 6.

Paired samples t-tests for Bloom grouping (pre-test vs. post-test)

Group	Subscale	Pre-test		Post-test		df	t	P <	d	Mean diff.	Confidence interval
		M	SD	M	SD						Lower limit	Upper limit
LBL	Bloom1 – knowledge	77.6	16.0	85.3	12.5	93	5.8	0.001	0.54	7.7	5.0	10.3
	Bloom2 – comprehension	66.6	12.9	71.7	13.8	93	3.9	0.001	0.42	5.1	2.5	7.7
	Bloom3 – application	59.6	17.2	76.4	15.6	93	10.1	0.001	1.00	16.8	13.5	20.1
PjBL	Bloom1 – knowledge	74.3	16.8	88.0	8.9	100	9.3	0.001	0.96	13.7	10.8	16.6
	Bloom2 – comprehension	67.6	14.2	81.1	10.9	100	10.2	0.001	1.09	13.5	10.9	16.1
	Bloom3 – application	54.0	17.8	79.4	10.4	100	16.1	0.001	1.73	25.4	22.3	28.5
PBL	Bloom1 – knowledge	72.8	14.4	90.2	6.8	49	8.3	0.001	1.48	17.4	13.2	21.6
	Bloom2 – comprehension	67.8	10.2	84.5	9.9	49	13.1	0.001	1.69	16.8	14.2	19.3
	Bloom3 – application	59.1	15.4	78.0	9.9	49	8.9	0.001	1.47	18.9	14.6	23.2

Superimposing the pre-test and post-test subscale score means (Figure 3) illustrates the mean differences (also in Table 6). The pre-test means reveal the highest test results in knowledge retrieval from memory (Bloom1, 72–78%) and the lowest in the knowledge application subscale (Bloom3, 54–60%). It is therefore not surprising that the most pronounced improvement was achieved in the subscale Bloom3 (17–25%), but the progress seems substantial in all three groups. In subscales Bloom1 and Bloom2, the LBL group lagged behind the PjBL and PBL groups, especially in the subscale of comprehension, where it only achieved approximately a third (5%) of the improvement of the two active-learning methods.

Figure 3.

Pre-test and post-test score means (%) by Bloom subscales and teaching methods.

Further investigation of group differences with ANOVA revealed no significant difference between the teaching methods in the pre-test in any of the subscales, while in the post-test, the groups differed in knowledge and comprehension, but not in knowledge application. The latter result implies that both passive and active methods were equally successful in conveying the skill of the use of information. Subscale Bloom3 (application) was therefore excluded from further comparisons.

A t-test for independent samples (Table 7) was used for pairwise post-test method comparisons. Only the LBL and PBL methods differed significantly in the knowledge subscale (with a small effect size), while in the comprehension subscale, the LBL group differed significantly from both of the other methods. In the lowest cognitive category (knowledge), the PBL group was the most efficient and the LBL the least (with a 10% difference in improvement).

Table 7.

T-test of independent samples – significance (P) with effect size (d) of pair differences in the post-test by Bloom subscales

Group pair	Subscale*	Mean diff.	P <	d
LBL-PjBL	Bloom1 – knowledge	2.7	0.083	0.25
	Bloom2 – comprehension	9.4	0.001	0.76
LBL-PBL	Bloom1 – knowledge	4.9	0.003	0.45
	Bloom2 – comprehension	12.8	0.001	1.02
PjBL-PL	Bloom1 – knowledge	2.2	0.097	0.27
	Bloom2 – comprehension	3.4	0.062	0.32

Subscale Bloom3 was excluded.

4.4. Discussion

The results suggest that while all three teaching methods significantly improved student information literacy, action-oriented learning methods such as PjBL and PBL were more successful than the traditional approach of LBL. This is in line with results of Hsieh and Knight [32], finding PBL more effective than LBL in IL instruction for engineering students.

Regarding the different topics of IL according to the ACRL standards, there was a significant improvement in the post-test in all subscales for all teaching methods. This suggests that any of the three teaching methods can successfully improve students’ IL in the whole range of standardised IL categories. However, students had the lowest initial knowledge in intellectual property and ethics issues and in searching skills, followed by slightly better results in the identification of information needs. In the teaching method comparison, the efficiency of PjBL and PBL did not differ much in any of the subscales, while both active-learning groups improved substantially more than the LBL group in four of the five ACRL standards. The improvement was the highest in searching skills, and in intellectual property/ethics, probably due to more authentic practical experiences with searching in the active-learning methods. Prior research [42] similarly finds PBL enabling better skill development. However, the results did not confirm the advantage of PBL in the identification of information needs, as found by Gurses [43].

Furthermore, the subscale analysis according to the Bloom’s taxonomy showed significant progress of students across all three cognitive level subscales, in all three teaching methods. The lowest initial IL level was recorded in the knowledge application category, and the highest in knowledge retrieval from memory. The comparison of teaching methods revealed no significant difference between PjBL and PBL in any of the subscales. In comparison to LBL, the active methods were significantly better in comprehension. This is in line with Keene et al. [12], who found comprehension to be stronger in student-oriented activities. In contradiction with Strobel and van Barneveld [42], PjBL and PBL were also more successful in knowledge recall, but no difference between the methods was found in knowledge application, where improvement was the highest. Although no important differences between PjBL and PBL were detected, the improvement in the cognitive category of knowledge application was higher in the PjBL group.

5. Conclusions

The presented study explored and compared the efficiency of lecture-based, project-based and problem-based teaching approaches in an IL course in higher education. According to the research objectives and working hypotheses, the results can be summarised as follows:

Total ILT score:

Initially the three student groups did not differ in their overall IL proficiency.

By the end of the IL course, all three groups had improved substantially.

The final IL level of the LBL group was lower than those of the PjBL and PBL groups. The latter two did not differ significantly.

Subscale analysis of the ACRL standard grouping:

With the exception of information need identification, the three method groups displayed similar initial IL levels, with the greatest deficiencies detected in intellectual property rights issues and in searching skills.

All of the teaching method groups improved notably, with the biggest improvement in searching skills.

At the end of the course, the PjBL and PBL groups performed better than the LBL group in all topics except for information need identification, with the biggest difference being observed in searching skills and intellectual property topics. Neither of the two active learning methods was superior to the other.

Subscale analysis of the Bloom taxonomy grouping:

Initially, the method groups did not differ significantly, with the highest scores in knowledge recall and the lowest in knowledge application.

All of the teaching method groups improved during the course, most notably in knowledge application, especially in the PjBL group.

Substantial final differences of the LBL group with the PjBL and PBL groups were recorded in comprehension. In knowledge recall, only the PBL group scored significantly better than the LBL group. Again, the PBL and PjBL groups were too close for meaningful distinction.

Based on the above findings, the study suggests the following conclusions:

A well-organised credit-bearing university course significantly improves the IL of students in all main IL topics.

Traditional and action-oriented teaching methods are all beneficial in reaching a higher level of IL knowledge, as well as of understanding and application of knowledge. Therefore, university educators may use and combine methods within the IL study courses.

In some aspects, different teaching methods result in different levels of IL knowledge and skills. Action-oriented teaching methods tend to promote IL through practice and experience, and are particularly better in searching skills and intellectual property topics. Project-based and problem-based approaches result in similar levels of IL improvement.

The value/originality of the presented research is the evaluation and comparison of three teaching methods, applied in higher education IL courses, from the viewpoint of their contribution to IL contents (as defined by ACRL standards) and cognitive levels (as defined by Bloom categories), using our own previously validated assessment tool (ILT).

The study strengthened some of the prior partial research results, citing the advantages of individual active learning methods versus the traditional approach, thus justifying and recommending their integration into IL curricula, particularly to offer students authentic practical experiences with searching tasks and intellectual property/ethics issues.

Based on the results and experiences of this study, the following observations and recommendations might be useful for IL educators:

Traditional teaching approaches seem to be most beneficial at the beginning of the IL course, as they provide a quick, solid, well-structured foundation, familiarising students with IL standards and with the purpose and objectives of the active learning tasks to follow.

In the traditional approach, the IL content follows a pre-defined order and is presented in a full lecture format. In contrast, short inserts of lectures on essential IL content seem to be more effective in the PjBL or PBL active learning sessions, presenting specific IL topics when they are needed in order to solve a PBL sub-task or to proceed with project work. However, this demands that the IL educator provide well-planned PjBL and PBL tasks, so that the whole spectrum of IL standards and outcomes is eventually covered, as defined in the curriculum. Not knowing the complete picture in advance, students might occasionally lapse into doubt, confusion, anxiety or frustration during the active learning process, particularly within the PBL format. The gain is sometimes recognised by students only at the end, when the tasks have been successfully completed, and when retrospection and reflection bring a clearer understanding and awareness of the acquired knowledge and skills.

Regarding IL content, PjBL and PBL seem to be preferable and are particularly effective for searching tasks (ACRL standard 2) and intellectual property/ethics issues (ACRL standard 5).

Within the active learning process focused on ACRL standard 2, the students gain personal experience and develop skills in a wide range of activities, such as: (1) identifying key concepts and search terms; (2) selecting search strategies and search queries with appropriate commands for information retrieval; (3) adapting and implementing search profiles in various databases, information systems and web search engines, and modifying/optimising search strategies to achieve a manageable amount of the most relevant information; (4) evaluating information by assessing the quantity, quality and relevance of the search results; (5) identifying potential gaps in the information retrieved, in order to repeat or upgrade searches using revised search queries; (6) examining and evaluating information from various sources regarding reliability, validity and accuracy; and (7) creating a personal information management system for organising, structuring, storing and retrieving information in order to achieve the final goal.

Through active learning approaches related to intellectual property/ethics issues (ACRL standard 5), students can gain first-hand experience in topics related to: (1) privacy, security and licensing (free versus fee-based access) of information sources; (2) authors’ rights, copyright issues, piracy and plagiarism; (3) referencing and citing sources, selecting and using appropriate referencing styles; (4) industrial intellectual property, including patenting, trademarks and protection of industrial designs; (5) issues related to censorship, freedom of speech, political, economic or social bias, unfair advertising, consumer deception, one-sided reporting and confusing facts with opinions.

Although these findings cannot yet be generalised, they can serve as preliminary information and provide ideas for the design of further similar studies in various educational and cultural environments. Further research could consider more cognitive levels, additional assessment methods and tools for active learning evaluation, incorporation of other active learning methods, other instruction delivery methods (e.g. e-learning, blended learning), and interpretation of results using the new ACRL Framework. For studying IL in the context of the higher cognitive categories (analysis, synthesis and evaluation), we suggest extending the ILT with a set of open-ended tasks, and/or assessing other student deliverables, such as seminar and project assignments, with adequate grading rubrics. Attempts to correlate the ACRL standards to the new ACRL Framework have been made [15], so a similar study would be possible by mapping the ILT items according to the Framework’s knowledge practices/abilities and dispositions, or more broadly according to the frames, instead of the standards, taking into account a set of new recommendations for assessment within the ACRL Framework [8].

Footnotes

Acknowledgements

The authors would like to thank Professors Cirila Peklaj and Saša Glažar for their contribution to ILT subscales according to the Bloom categories, all students who participated in the study and Neville Hall for proofreading the manuscript.

Funding

The study was financially supported by the Slovenian Research Agency, project J5-5535 ‘Development of information literacy of university students as a support for solving authentic science problems’.

References

Boh Podgornik

Dolničar

Šorgo

Bartol

. Development, testing, and validation of an information literacy test (ILT) for higher education. Journal of the Association for Information Science and Technology 2015. DOI: 10.1002/asi.23586.

Catalano

. The effect of a situated learning environment in a distance education information literacy course. Journal of Academic Librarianship 2015; 41(5): 653–659.

Ross

Furno

. Active learning in the library instruction environment: An exploratory study. portal: Libraries and the Academy 2011; 11(4): 953–970.

Hsieh

Dawson

Hofmann

Titus

Carlin

. Four pedagogical approaches in helping students learn information literacy skills. Journal of Academic Librarianship 2014; 40(3–4): 234–246.

Webber

Johnston

. Conceptions of information literacy: New perspectives and implications. Journal of Information Science 2000; 26(6): 381–397.

ACRL. Information Literacy Competency Standards for Higher Education. Chicago, IL: American Library Association, 2000. Available at: http://www.ala.org/acrl/standards/informationliteracycompetency (accessed 16 Feb 2016).

ACRL. Framework for Information Literacy for Higher Education. Chicago, IL: American Library Association, 2016. Available at: http://www.ala.org/acrl/standards/ilframework (accessed 16 Feb 2016).

Oakleaf

. A roadmap for assessing student learning using the new framework for information literacy for higher education. Journal of Academic Librarianship 2014; 40(5): 510–514.

Anderson

Sosniak

(eds) Bloom’s Taxonomy: A Forty-Year Retrospective. Chicago, IL: University of Chicago Press, 1994.

10.

Weiner

. Is there a difference between critical thinking and information literacy? Journal of Information Literacy 2011; 5(2): 81–92.

11.

Reece

. Critical thinking and cognitive transfer: Implications for the development of online information literacy tutorials. Research Strategies 2005; 20(4): 482–493.

12.

Keene

Colvin

Sissons

. Mapping student information literacy activity against Bloom’s taxonomy of cognitive skills. Journal of Information Literacy 2010; 4(1): 6–21.

13.

Walton

Hepworth

. A longitudinal study of changes in learners’ cognitive states during and following an information literacy teaching intervention. Journal of Documentation 2011; 67(3): 449–479.

14.

Rieh

Collins-Thompson

Hansen

Lee

H-J

. Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science 2016; 42(1): 19–34.

15.

Harmer

Lee

. Frameworks, Standards, and Benchmarks, Oh My! Georgia International Conference on Information Literacy. Paper 39. 2015. Available at: http://digitalcommons.georgiasouthern.edu/gaintlit/2015/2015/39 (accessed 16 Feb 2016).

16.

Barrows

. A specific problem-based, self-directed learning method designed to teach medical problem-solving skills and enhance knowledge retention. In: Schmidt

deVolder

(eds) Tutorials in Problem-Based Learning. Maastricht, Netherlands: Van Gorcum, 1984. p. 16–32.

17.

Savery

Duffy

. Problem-based learning: An instructional model and its constructivist framework. Educational Technology 1995; 35(5). Available at: http://works.bepress.com/john_savery/4 (accessed 16 Feb 2016).

18.

Boud

Feletti

. The Challenge of Problem-based Learning. Abingdon, UK: Psychology Press, 1998. 356 p.

19.

Savery

. Overview of problem-based learning: Deﬁnitions and distinctions. Interdisciplinary Journal of Problem-Based Learning 2006; 1(1): 3. Available at: http://docs.lib.purdue.edu/ijpbl/vol1/iss1/3 (accessed 16 Feb 2016).

20.

Blumenfeld

Soloway

Marx

Krajcik

Guzdial

Palincsar

. Motivating project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist 1991; 26(3–4): 369–398.

21.

Kenney

. Revitalizing the one-shot instruction session using problem-based learning. Reference & User Services Quarterly 2008; 47(4): 386–391.

22.

Hines

. Faculty and librarian collaboration on problem-based learning. Journal of Library Innovation 2012; 3(2): 18–32.

23.

Maybee

. Preparing today’s learners: The role of information literacy in the adoption of innovative pedagogies. Libraries Faculty and Staff Presentations 2015. Available at: http://docs.lib.purdue.edu/lib_fspres/92 (accessed 16 Feb 2016).

24.

Macpherson

. An information processing model of undergraduate electronic database information retrieval. Journal of the American Society for Information Science and Technology 2004; 55(4): 333–347.

25.

Snavely

. Making problem-based learning work: Institutional changes. portal: Libraries and the Academy 2004; 4(4): 521–531.

26.

Wenger

. Problem-based learning and information literacy: A natural partnership. Pennsylvania Libraries: Research & Practice 2014; 2(2): 142–154.

27.

Spence

. The usual doesn’t work: Why we need problem-based learning. portal: Libraries and the Academy 2004; 4(4): 485–493.

28.

Diekema

Holliday

Leary

. Re-framing information literacy: Problem-based learning as informed learning. Library & Information Science Research 2011; 33(4): 261–268.

29.

Cook

Klipfel

. How do our students learn? An outline of a cognitive psychological model for information literacy instruction. Reference & User Services Quarterly 2015; 55(1): 34–41.

30.

Dodd

. The impact of problem-based learning on the information behavior and literacy of veterinary medicine students at University College Dublin. Journal of Academic Librarianship 2007; 33(2): 206–216.

31.

Smith Macklin

. Integrating information literacy using problem-based learning. Reference Services Review 2001; 29(4): 306–314.

32.

Hsieh

Knight

. Problem-based learning for engineering students: An evidence-based comparative study. Journal of Academic Librarianship 2008; 34(1): 25–30.

33.

Chen

Y-H

W-I

. Effects of integrated information literacy on science learning and problem-solving among seventh-grade students. Malaysian Journal of Library & Information Science 2014; 19(2): 35–51.

34.

Detlor

Booker

Serenko

Julien

. Student perceptions of information literacy instruction: The importance of active learning. Education for Information 2012; 29(2): 147–161.

35.

Boustany

. Information literacy skills of students at Paris Descartes University. In: Kurbanoğlu

Grassian

Mizrachi

Catts

Špiranec

(eds) Worldwide Commonalities and Challenges in Information Literacy Research and Practice. Switzerland: Springer International Publishing, 2013. pp. 588–595.

36.

Maybee

. Undergraduate perceptions of information use: The basis for creating user-centered student information literacy instruction. Journal of Academic Librarianship 2006; 32(1): 79–85.

37.

Schroeter

Higgins

. The impact of guided vs. self-directed instruction on students’ information literacy skills. Journal for Advancement of Marketing Education 2015; 23(1): 1–10.

38.

Eskola

E-L

. Information literacy of medical students studying in the problem-based and traditional curriculum. Information Research 2005; 10(2): 10–12.

39.

Wopereis

Frerejean

Brand-Gruwel

. Information problem solving instruction in higher education: A case study on instructional design. In: Kurbanoğlu

Boustany

Špiranec

Grassian

Mizrachi

Roy

(eds) Information Literacy: Moving Toward Sustainability. Springer International Publishing, 2015. pp. 293–302.

40.

Uğur

Akkoyunlu

Kurbanoğlu

. Students’ opinions on blended learning and its implementation in terms of their learning styles. Education and Information Technologies 2011; 16(1): 5–23.

41.

ZDBS. Merila in kazalci informacijske pismenosti v visokem šolstvu. Chicago, IL: American Library Association, 2010. Available at: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/infolit_slovenian.pdf (accessed 16 Feb 2016).

42.

Strobel

van Barneveld

. When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning 2009; 3(1): 4. Available at: http://docs.lib.purdue.edu/ijpbl/vol3/iss1/4 (accessed 16 Feb 2016).

43.

Gürses

Açıkyıldız

Doğar

Sözbilir

. An investigation into the effectiveness of problem-based learning in a physical chemistry laboratory course. Research in Science & Technological Education 2007; 25(1): 99–113.