Abstract
This paper explores, on the basis of their cognitive states, the likely inclinations of a group of young people in England with regard to evaluating Web-based information. The work draws on data submitted via an online questionnaire by 82 16–18 year-olds. Frequency counts were generated and each participant received a ‘trust disposition’ score according to their questionnaire responses. The results present a mixed picture in terms of the probable tendencies of young people to evaluate information. The participants’ data in relation to some of the issues addressed in the questionnaire suggest that they may be inclined to accept much of what they see without challenging it, although few of their overall trust disposition scores are significantly high.
Introduction
Much has been written in library and information science (LIS) about the importance of people evaluating information. The issue has attracted particular attention from both academics studying information behaviour (IB) and practitioners seeking to promote information literacy (IL). Analysis and comment have come, too, from diverse fields far beyond the LIS discipline. In cognitive science, Lewandowsky has made significant contributions to this area in discussing how our brains process information (Lewandowsky, Ecker, Seifert, Schwartz and Cook, 2012; All in the Mind, 2012) and, in journalism, Marr (2005) has asked how far it is defensible to rely on the claims of a single source when making a serious allegation.
In LIS, an appreciation of the need to appraise information and, especially, information sources can be traced back over decades. It is evident, for example, in some of the earliest models for teaching information skills. More than 30 years ago, in a landmark framework, Marland (1981) cited a series of criteria that he stipulated should be borne in mind by young people when taking decisions as to whether to select or reject individual materials for use. Around the same time, in his article on ‘the use of the library’, Trigg (1981) presented a breakdown of evaluation skills consisting of nine elements.
Much more recently, the widespread availability of the Internet in homes, schools and libraries has raised very substantially the profile of the skill of evaluating information. Indeed, it is now a key aspect within several textbook definitions as to what IL involves. Harrod’s Librarians’ Glossary and Reference Book declares that IL should be regarded as the “ability to identify, locate, evaluate, organize and use information” (Prytherch, 2005: 351). Today, the evaluation aspect of IL is closely related to the notion of digital literacy (DL). According to Eisenberg, Lowe and Spitzer (2004), DL “considers the broad range of resources that are accessible online and underscores the importance of looking at each of these resources with a critical eye. Emphasis is placed on the format of the information presented and the special considerations that each type of resource presents” (p. 8).
The characteristics of the Internet that have increased the need for users to appraise information rigorously are now well accepted. The sheer volume of material that is available via the World Wide Web, the absence of gatekeepers who would be responsible for ensuring information quality and the fact that the search engines employed by so many users give no indication of the soundness of the material that has been retrieved have all led to major concerns. The controlled world of paper publishing is frequently compared favourably by commentators to the less rigorous and more uncertain environment of the Web. In the words of Madden, Bryson and Palimi (2006): “A sophisticated, well laid out Website may be the product of a professional publisher, or it may have been produced by a technically competent 15 year old with pirate software, who copied the design from another Website. Even the worst of books, in contrast, has been subject to some quality control” (p. 48). The way in which Internet users have progressed to their current status as active creators of content is also highlighted by Lewandowsky, Ecker, Seifert, Schwartz and Cook (2012). Pointing to the popularity of Twitter, YouTube and blogs, the authors affirm that whilst the Internet has revolutionised the availability of information, it has, in addition, facilitated the spread of misinformation. They note, too, the alarming trend that information on the Web “is progressively replacing expert advice” (Lewandowsky, Ecker, Seifert, Schwartz and Cook, 2012: 111).
Children and young adults are popularly portrayed as masters of digital technology, frequently because they have grown up in the electronic age and cannot remember a time before the widespread existence of the Internet. For them, we are often told, the world of keyboards, mice and monitors is as familiar as was that of books to their counterparts in previous eras. Yet, their youth can lead to problems as well. Nicholas suggests that whereas older information users have come to appreciate that traditional information sources are subject to vetting processes which lead to the stamp of quality by publishers and to organisational controls by intermediaries in libraries, many young people have no mental framework for understanding and measuring quality in the digital environment ( Analysis: Clever.com, 2009). Malik goes as far as to suggest, “The very people most at home on the web seem also the ones least able to make use of it” ( Analysis: Clever.com, 2009).
This paper explores the inclination of young people to evaluate the information they retrieve in the Web environment. The following section presents a review of the pertinent literature before attention is turned to a new study of the issue, which has involved the collection of data via an online questionnaire from students aged between 16 and 18 in a high school in north-east England.
Literature review
The tendency of young people to accept without question much of the material they encounter on the Web is a theme that recurs in the literature time and time again. In a significant analysis that scrutinised many significant projects, Williams and Rowlands (2007) report that all too often young people have been seen to pay little regard to the authority of the material with which they come into contact when they search. Reflecting on the findings of research studies of various kinds, the authors conclude that “evaluative skills are barely in evidence” (Williams and Rowlands, 2007: 11). Although there are individual papers that present a different picture, the conclusions drawn by Williams and Rowlands are difficult to refute when the relevant literature is viewed as a whole.
Readers will no doubt recognise for themselves that the key concern lies in identifying why the rigorous assessment of source material is observed so rarely by researchers and information professionals when working with young people. In Bloom’s Taxonomy of Learning Objectives, the skill of evaluation – defined as the making of decisions or exercising of judgement based on criteria or a rationale – is postulated as the most difficult within the cognitive domain (Bloom, Engelhart, Furst, Hill and Krathwohl, 1956). Even in the early 1980s, long before the proliferation of information associated with today’s Information and Communications Technology (ICT), Trigg (1981) acknowledged that the evaluation of information was “an altogether longer and more complex business” than that of finding material (p. 304).
For Lewandowsky, the “default assumption” when people receive information is to accept that the material presented is true. This mindset, he maintains, is partly the result of upbringing – generally, children are raised by parents who tell them the truth for the majority of the time ( All in the Mind, 2012). There is firm evidence in LIS literature to corroborate Lewandowsky’s stance that, for the most part, young people do not doubt what they are told. Even when children are working with paper sources, Wray and Lewis (1995) note that they often encounter material that is misleading, incorrect or biased, yet they will still “naturally tend to believe that everything they read in books written by adults who know a great deal more than them about a particular topic is bound to be true” (p. 8). The experiences of Brabazon (2007) suggest that the attitudes detected by Wray and Lewis (1995) remain prevalent in the 21st century and among older learners. According to Shenton and Dixon (2004), teenagers generally expect that material which is relevant to their area will be usable, unless, after the most cursory of explorations, “they consider there are good reasons to doubt it” (p. 38). Obvious inadequacy then leads to closer inspection. Clearly, the critical faculties of some young people become engaged at a reactive, rather than proactive, level.
Lewandowsky also maintains that, since recipients of information are inclined to believe the material presented to them, they must make a special effort to tag the information as false if it becomes obvious that it is inaccurate ( All in the Mind, 2012). Let us assume for a moment that the process of recognising that particular information is incorrect imposes a significant burden on the individual, as Lewandowsky claims. The person’s inclination to ignore the extra demand that is made may be viewed as consistent with the ‘principle of least effort’, which is frequently put forward in the literature to explain the nature of users’ interactions with information. On the basis of findings from a range of studies, Dresang (2005) suggests that young people have often been found to display “a propensity to take the easiest path possible” (p. 181) – a course of action which goes some way in explaining the observation that young people may well “simplify Web site evaluation tasks and make credibility judgments that rely heavily on design and presentation features rather than content” (Harris, 2008: 161).
Those who subscribe to Birkhead’s argument that “the spoon-feeding-and-teaching-to-the-test culture at school has drained [learners] of independent thought” (Birkhead, 2009), may lay some blame for the tendency of young people not to apply evaluative skills at the door of the education system. The American Library Association (1989) asserts quite unequivocally, “Information prepackaging in schools… encourages people to accept the opinions of others without much thought” (p. 90). This problem may reflect a more fundamental issue – the lack of priority given to IL in schools. Koltay (2010) notes the oddity that people’s lack of information literacy is frequently lamented by information practitioners; considerably less concern is expressed by professionals in education (whom we can understand to include classroom teachers), even though it is in their work with learners that the implications of such deficiencies may be most apparent.
The research of Heinström (2006) may give rise to the theory that, at least in some cases, a lack of thought as to the trustworthiness of information is indicative of insufficient real engagement on the individual’s part with either the assignment they have been given or the topic under investigation. Heinström discovered that, in a school situation, intrinsically motivated young people, who have a genuine desire to learn, are more ‘attentive to information quality’ than extrinsically motivated young people, whose priority is simply to gather enough material to meet the requirements of the set task.
There is also a growing debate as to how the evaluation of information is best taught and some of the criticisms of the kinds of instruction often provided may result in the reader questioning the quality of the teaching. Ostenson (2014) reports that certain scholars have felt that particular models for evaluating information are “inadequate or even inaccurate” (p. 36). They may “encourage shallow thinking” or concentration on “irrelevant features of the source”; prompts designed to stimulate appraisal may not be especially helpful and some criteria may lead to “false negatives” (p. 36). Shenton and Pickard (2012) have also questioned the value of the various models that are used to foster IL.
Undoubtedly, some of the individual strategies for evaluating information are problematic. Graef (2000) maintains that material is frequently selected by users after testing the content in front of them against their past experience. Typically, the information-seeker compares the encountered information with their own knowledge of the subject. Speaking on the BBC radio programme ‘The Wikipedia Story’ (2007), McHenry, however, argues that such an approach is futile. In his words, “you really have to know the subject you’re looking up to the point where you needn’t be looking it up.” An alternative avenue, and one that is consistent with Paterson’s principles, lies in using other sources to verify the accuracy of the information within the work under scrutiny (Paterson, 1981). This strategy is obviously slow to execute, though, and care is needed when making decisions as to which corroborating sources should be consulted. For convenience reasons, when assessing material on a Web page the temptation may well be to compare it with other information available through the Internet. This kind of evaluation is far from trouble-free, as questions have to be asked of the trustworthiness of the material used to make the comparison, too. The tasks of selecting appropriate sources for verification purposes and applying systematically each quality criterion in a lengthy series can seem unnecessary to an individual who can bring their work on an assignment to a swifter conclusion by merely accepting information which would seem adequate for their purposes.
It may be concluded, then, that the evaluation of information is a much more demanding task than educators tend to acknowledge and it is insufficiently valued in education. Some of the instructional models that are applied in schools have been subject to criticism and the issue of how source appraisal is best promoted remains unresolved. Undoubtedly, many young people spend at least some of their time at school tackling activities in which they feel uninvolved and these would seem especially likely to lead to an uncritical attitude to the use of information. As well as being cognitively challenging, if applied rigorously the evaluation of information is time consuming to the individual, who may not be inclined to expend the effort required to make such an assessment, even if they have taken the important first step of recognising the need to adopt a questioning attitude.
Origin of the research instrument
The questionnaire used to collect data from the teenagers who participated in the study was derived from research undertaken at Northumbria University by Alison J. Pickard, Pat Gannon-Leary and Lynne Coventry. In a project funded by the Joint Information Systems Committee (JISC), the team investigated “how users place their trust in digital information resources in the web environment” (Pickard, Gannon-Leary and Coventry, 2010: 4). The credibility judgements reached by people were found to be determined by three key sets of criteria. These pertained respectively to the individual’s cognitive state, internal cues relating to the material itself and external factors associated with the source but not intrinsically part of it. Shenton and Pickard recognised that it was possible to develop a document devoted to the first area and this could be used for both facilitating information literacy instruction and advancing research into information behaviour. In terms of the former, a carefully constructed questionnaire with prompts which, in their entirety, addressed all 16 of the cognitive state factors identified by Pickard, Gannon- Leary and Coventry (2010), could be used as an aid to stimulate reflection by older secondary school students (i.e. Sixth Formers) on their particular predispositions in relation to their tendency to trust material they find on the Web (Shenton and Pickard, 2014). In an era when the need to evaluate information has never been so vital, it was recognised that such self-awareness could form an important starting point before the educator seeks to instruct the young people in more specific matters with regard to the appraisal of material and sources. From an information behaviour (IB) perspective, the authors also knew that since the same questionnaire could afford a means of collecting data from ‘real’ students in relation to each of the personal characteristics that had already been identified as significant, further insight could be gained as to the likelihood that people will actually evaluate the material they access in the Web environment. It is the IB project that will be reported in the paragraphs to come.
Study questions and methodology
It was determined that the IB-focused project should address two key questions. Within a particular sample of young people, how prevalent are the cognitive traits revealed in research by Pickard, Gannon-Leary and Coventry (2010) to be factors in the level of trust individuals place in Web-based information sources? What can be learnt, via statistical measurement, about the likely inclination of members of the same sample to evaluate information on the Web on the basis of the cognitive state factors identified?
To some extent, the approach taken in the work may be considered typical of the ‘alternative’ paradigm for the investigation of information behaviour often associated with Dervin and Nilan (1986). In a seminal paper, the authors isolate a series of features characteristic of this perspective. One is a focus on internal cognitions, rather than external behaviour. This emphasis is especially pertinent to the evaluation of information since much of the assessment process takes place ‘in the head’ and little may be learnt from observing the individual’s actions. Certainly, the particular territory of interest in this paper – the information user’s cognitive state – is consistent with Dervin and Nilan’s concentration on the “internal” dimension of information behaviour.
The authors also explain that the “alternative” paradigm incorporates the use of qualitative approaches by the researcher (Dervin and Nilan, 1986). Yet, the study presented here is unequivocally quantitative; no provision was made for the students to record their ideas in their own words – they simply selected appropriate responses from a Likert scale. This stance is justifiable on the grounds that the project was never designed to identify the cognitive state factors that influence how far the individual trusts Web-based information; this has already been done, by Pickard, Gannon-Leary and Coventry (2010). As the questions above indicate, the study was devised with a view to identifying the prevalence, within a certain sample, of these criteria and then measuring, on the basis of the factors involved, the extent to which the participating young people are likely to evaluate the sources they find via the Internet.
Data collection
Having drafted the questionnaire on paper, initially for application in an information literacy context, the authors created, for the purposes of investigating information behaviour, an electronic version using the ‘survey’ facility provided by the Microsoft SharePoint software package, and the document ultimately created appeared as a Web page on the school intranet.
On 13th December 2013, all students in Years Twelve and Thirteen who would be invited to complete the questionnaire were briefed by the school’s Head of Sixth Form on what the task would involve. At the beginning of the following week, the students were asked to fill in a copy of the questionnaire, which could be accessed via their homepage. As a questionnaire was completed, the system created a record of the individual who had replied, together with their data. The questionnaire was not removed from the intranet until the end of the week, when the Christmas holiday provided a natural break in the academic year. At this point, the researchers felt they could be confident that no more responses would be received and the process of data analysis could begin.
Data analysis
The data were analysed in two different ways, each of which was designed to enable the researchers to answer one of the project questions. Question (a) was addressed through the use of SharePoint’s ‘graphical summary’ facility. This function generates simple counts and percentage proportions for each of the answers given in response to a particular question within a survey. The facility is invaluable when answers to multiple choice questions are involved as was the case in this research but it is much less useful when participants are required to submit their own text since the system is insufficiently sophisticated to recognise when different respondents present in different ways what is essentially the same answer.
All the data from the SharePoint survey were imported into a Microsoft Excel file. Personal scores, or ‘trust disposition’ values, were then calculated via the use of ‘if’ statements so as to answer research question (b). In some cases, a maximum mark (i.e. 5 points) in relation to a certain statement resulted from a response that indicated strong agreement; in other instances such a mark accrued from strong disagreement. In broad terms, when the personal characteristics of the individual are considered in relation to each statement, the higher the figure the greater the participant’s inclination to trust the information presented to them in the Web environment without evaluating it. An overall trust disposition score was calculated for each participant, based on the sum of their points for their 18 responses.
The school
The institution in which the study took place is a smaller than average secondary school located in northern England. At the time of the research, 560 young people ranging from 13 to 18 years of age were on roll. The Sixth Form, which consists of 156 teenagers aged between 16 and 18, is large and these learners comprise some 28 percent of the school’s total student population. Males significantly outnumber females at this level, mainly as a result of the attraction that the school’s Sixth Form football academy offers to boys who live in the locality.
At the time when data were collected, the school lacked a traditional library and many of the independent learning tasks that are included in lessons take place in the computer zones which are a feature of the building. Although most of these areas are used only for lessons and must be booked in advance, one – a large, ground floor computer zone that accommodates 72 workstations – is available for use by students during break, lunchtime and before and after school.
The sample
In all, 82 Sixth Formers completed questionnaires and the response rate was around 53 percent. The numbers of participants within the two year groups broadly reflect the wider balance within the Sixth Form. Many more Year Twelve students contributed data than did their Year Thirteen counterparts and this is indicative of the fact that the younger cohort is significantly larger in the school. Of the 82 students who participated, 63 percent were male and 37 percent female. This balance is virtually identical to the boy/girl proportions within the Sixth Form as a whole.
Results and discussion
In terms of question (a)
Table 1 presents a breakdown of the numbers of respondents and percentage proportions within the sample in relation to the frequency with which particular reactions were given to the questionnaire’s individual statements. It is illuminating to explore how the various patterns that emerge lead to both positive and negative pictures with regard to the likely inclination of the young people to accept without question the information that they find on the Web. In terms of ‘optimistic’ evidence, it can be seen that 88 percent of the sample considered that they like to be well informed and 82 percent claimed that they enjoyed exploring ideas. If we assume that people who exhibit these traits may well look for information from different viewpoints rather than simply trust any material they find, these figures are encouraging. Moreover, 80 percent believed that they were confident in their understanding of how the Internet and search engines work. Educators will hope, then, that these students appreciate the shortcomings of the provision of information on the Web. In view of the possibility that anyone who does not relish a particular task may well be inclined to bring it to a swift conclusion and lack the necessary rigour when undertaking it, the fact that just 16 percent reported that they did not enjoy using the Internet is also highly positive. Conversely, the discovery that 49 percent believed that the task of finding information was boring, unless they were pursuing material for personal interest, is disconcerting and we may wonder how many of the students are inclined to expedite the information-finding process and cut corners. Nevertheless, most of the participants seem well versed in at least one strategy for appraising information – 70 percent maintained that when they found new information, they usually assessed it against what they knew already. Finally, a high proportion of the sample recognised that, at least in some instances, it is unwise merely to accept material without challenging it – 69 percent were of the opinion that when the situation really mattered, they questioned the information they received.
Breakdown of student responses to the eighteen questionnaire statements.
Other discoveries, however, are more alarming. 80 percent regarded themselves as trusting individuals, although, curiously, only 50 percent felt they could rely on people to tell them the truth and be honest. Fifty-four percent thought of themselves as natural risk-takers in life. It is, of course, possible to equate risk-taking with a tendency to accept information without checking it thoroughly. If we accept Costa and Kallick’s principle that intelligent problem solvers ask questions of source material (Costa and Kallick, 2007), then the fact that only 23 percent rated their intelligence as higher than average may cause concern, unless of course the participants were simply being unduly modest. Clearly, if information-finding tasks are not tackled until near the deadline, little time is left for evaluation, so the acknowledgement by 44 percent that they tend to leave jobs to the last moment would seem to have serious consequences for information behaviour. Moreover, 53 percent admitted to a tendency to accept whatever information they could find and which seemed relevant to their needs when time was short. These revelations have major implications for time management on the part of students. 46 percent disagreed with the statement, “I’ve a low opinion of computer-based technology.” This would suggest that many of the teenagers are not especially sceptical of information that is offered by this means. We can certainly conclude from the discovery that 60 percent adjudged that they tended to accept any new information they encountered as potentially useful that a lot of these individuals do not submit to close scrutiny the material they access. It may also be the case that after a particular source has proved valuable to the student in the past, its new material is not closely evaluated; 59 percent stated that they were inclined to believe what they read on a Web site if they had often found the site useful before. This discovery that previous personal events were a significant factor in the students’ attitudes is underlined by other evidence – 53 percent considered that if they had a positive image of an author or their institution, this usually came from their past experience of them.
It is difficult to draw unequivocal conclusions from the data on one particular issue. Reactions to the statement, “I enjoy exploring information, even if the subject is less than interesting” failed to show any real consensus – 34 percent agreed, either strongly or slightly; 39 percent were neutral in their attitude; 27 percent reported some measure of disagreement.
In terms of question (b)
Once the trust disposition scores had been calculated, it became apparent that the overall range was considerable; 27 points separated the lowest (39 – recorded by a boy in Year Thirteen) and the highest (66 – registered by another male in the same year group). Yet, as can be seen from Table 2, there was much clustering of scores between 51 and 58. The mode was 52, with 10 students registering this score, and the median 54. The frequency of each of the values is shown in Figure 1. The overall pattern exhibits many of the features characteristic of a normal distribution. Since a high trust disposition score is by no means desirable, the fact that so many of the students’ figures are to be found around the mid-point of the overall 18–90 scale is encouraging. So, too, is the revelation that there were no values in the top third of the scale (i.e. the 67–90 region). Lewandowsky suggests that, ideally, information users should exercise a certain degree of scepticism when receiving information, as such an attitude helps them to differentiate between what is true and what is false ( All in the Mind, 2012). If we accept this argument, a low trust disposition score is obviously preferable.
Breakdown of student scores by year group and gender.
Frequency of the trust disposition scores.
The first time that the questionnaire was implemented was whilst one of the authors was teaching a small group of Year Thirteen students in a session broadly devoted to good academic practice. The class consisted of three learners, whose trust disposition scores were 35, 47 and 52 respectively. When these figures are compared with those of the participants in the research project, it can be seen that 47 and 52 lie in the bottom half of the scale, and no-one taking part in the study registered a score as low as 35. At this early stage, it is difficult to ascertain the point on the scale which may be considered to represent ‘high trust.’ In an article for a teaching magazine in which they presented the questionnaire, Shenton and Pickard (2014) suggest that, with 90 forming the maximum total and 18 the minimum, the term could apply to a figure of 60 or more. If this benchmark is applied to the statistics here, then only seven students (around 9 percent of the sample) would exhibit this characteristic.
Whilst the project reported in this paper was taking place, another was also being staged in the same school. This was a qualitative study that explored student perspectives as to why a traditional library might be needed in the institution. Here, too, findings gave considerable grounds for optimism, with several Sixth Formers making clear their lack of trust in information available via the Internet (Shenton, 2014). It must be admitted, however, that even if young people express misgivings of this kind when contributing data for a research project it cannot be assumed that they will demonstrate comparable scepticism when they actually use the Web for information.
Conclusions
In view of the pessimism inherent in much of the debate relating to young people and the evaluation of information, academics and practitioners can take considerable heart from the findings of this study. The most striking feature is that the trust disposition scores are not startlingly high. On an 18–90 scale, the biggest value was only 66 and a mere five students recorded scores in excess of 60. Educators can build, too, on some of the discoveries made by the researchers when answering question (a). Here it emerged that large numbers of the participants claimed they were keen to be well informed and they enjoy exploring ideas. It is not difficult to demonstrate that these are best furthered through the use of high quality information. Many of the students do appreciate the need to appraise material, and indeed maintain that they carry out such assessment if the situation is sufficiently important. A lot of the respondents also state that they exercise at least one strategy for evaluating information – they compare material that is new to them with what that they know already. There are, however, areas of concern, too; the willingness of some students to take risks and a lack of regard for time management can have unfortunate repercussions. There are instances where the results present mixed messages. The percentages of students who say that are trusting people are high, yet much lower proportions of the participants agreed with the statement, “I feel I can rely on people to tell me the truth and be honest.” It is difficult to explain this apparent anomaly and the situation can probably be clarified only with the collection and analysis of more data from these young people.
It should be acknowledged that some of the frequency figures relating to individual responses to a particular statement can also be interpreted in contrasting fashions. Many of the young people claimed to be confident in their understandings of how the Internet and search engines work. Ostensibly, this would seem heartening but some sceptical readers may feel that it is indicative of the widespread tendency of young people to overestimate their ICT/IL knowledge and skills. Latham and Gross (2013) point to the fact that research has shown that even after taking a test students with less than proficient IL skills “are unable to recalibrate their [inflated] self-views” (p. 432). Nevertheless, such a test would have been invaluable in the present study as it would at least have helped the researchers to determine whether the participants’ confidence was justified.
Readers studying the trust disposition scores recorded by the participants here – and teachers/school-based information specialists who use the questionnaire with their own students – may well seek to infer how far the young people in question are generally inclined to accept unquestioningly the information they find on the Web. It should be appreciated, however, that the trust issue is multi-faceted and, as Pickard, Gannon-Leary and Coventry (2010) demonstrate in their research, can be fully understood only if other influences are considered as well. In their overall model, the authors also highlight the importance of internal cues relating to the material itself and external factors associated with the source. Thus, whilst an analysis of the cognitive state of the individual gives us some insight into the person’s tendency to trust information found via the Internet, neither the teacher nor the student in question should believe that the data elicited by the questionnaire presented in this paper offer an entirely comprehensive picture of the situation.