How much do you need: a randomised experiment of whether readers can understand the key messages from summaries of Cochrane Reviews without reading the full review

Abstract

Objective

We explored whether readers can understand key messages without having to read the full review, and if there were differences in understanding between various types of summary.

Design

A randomised experiment of review summaries which compared understanding of a key outcome.

Participants

Members of university staff (n = 36).

Setting

Universities on the island of Ireland.

Method

The Cochrane Review chosen examines the health impacts of the use of electric fans during heat waves. Participants were asked their expectation of the effect these would have on mortality. They were then randomly assigned a summary of the review (i.e. abstract, plain language summary, podcast or podcast transcription) and asked to spend a short time reading/listening to the summary. After this they were again asked about the effects of electric fans on mortality and to indicate if they would want to read the full Review.

Main outcome measure

Correct identification of a key review outcome.

Results

Just over half (53%) of the participants identified its key message on mortality after engaging with their summary. The figures were 33% for the abstract group, 50% for both the plain language and transcript groups and 78% for the podcast group.

Conclusions

The differences between the groups were not statistically significant but suggest that the audio summary might improve knowledge transfer compared to written summaries. These findings should be explored further using a larger sample size and with other reviews.

Keywords

systematic reviews information retrieval dissemination computer-mediated communication academic communication

Introduction

Systematic reviews have been defined as ‘a high-level overview of primary research on a particular research question that tries to identify, select, synthesize and appraise all high quality research evidence relevant to that question in order to answer’.¹ And as such, systematic reviews synthesise a great deal of information from randomised trials making it possible for evidence to be accessible and usable by the busy clinician.

Several studies have found that Cochrane Reviews can be difficult to understand^2,3 due, at least in part, to the large amount of information contained in a review makes reading it a lengthy, time-consuming process that would be impractical for many.⁴ If the style of communication was thought to be inaccessible for readers, then the impact of systematic reviews would be limited⁵; to be accessible, it must be packaged in a format that will promote identification and encourage use.⁶ With this in mind, it has been acknowledged that successfully conveying scientific information in an engaging and understandable way is a real challenge.⁷ In order to combat this, the Cochrane Library requires that all reviews write a scientific abstract. In later years, other forms of summaries have been added such as the plain language summary, and for some reviews, an audio podcast is also offered. When accessing a review on The Cochrane Library, the reader is presented with both the abstract and plain language summary prior to clicking through to the full review.

Many readers will access the various summaries rather than the full review, so it is essential that these summaries are clear, understandable and accessible. The purpose of the abstract is to assist the reader in selecting and appraising a review.⁸ It has been found that abstracts are often the only part of the review that readers use.⁹ Indeed, physicians spent, on average, less than 2 min searching for an answer to a clinical question¹⁰ which highlights the need for summaries to be clear and concise; otherwise, errors in interpretation may be made which could impact on policy and medical practice.

Zhelev et al.⁵ reported that academic participants utilised abstracts as a ‘gateway’ to the review, used to ascertain the reviews relevance and whether or not they would go on to read the full review. Indeed, the non-academic clinicians, in their study, utilised the abstract only to understand the main findings and would not read the full review due to lack of time and disinterest in the complex methodology employed in reviews. The authors considered the problem posed by misunderstanding the results dictated by the abstract as going unchecked if the full review was not read; one participant stated that the abstract only made complete sense after they had read the full review.

Cochrane Library users can also read a plain language summary and/or listen to the authors in a podcast. The plain language summary has recently been introduced by Cochrane as a dissemination method to bridge the gap between academics, clinicians and laypeople and is a condensed version of the full review written in non-technical language devised to be read in isolation. Podcasting offers an opportunity to access the findings of the review by another route. It is an attempt to connect with busy readers who may not have the time to sit down and read summaries as they can be downloaded as audio recordings and listened to while on the move, or doing other tasks. Podcasts are well received, with students perceiving them as a more effective revision tool than textbooks.¹¹

It is thought that everyone has a style of learning which suits them best, and successful teaching and communication needs to encapsulate all styles to ensure maximum reach. This suggests that some readers of Cochrane Reviews may find audio podcasts more useful than other people might. In psychological and educational literature, the concept of individual learning preferences has widespread popularity despite mixed empirical support. While some research has not found any difference in outcomes,^12,13 other researchers have found positive outcomes from students using podcasts to supplement their learning.^14,15 There is an assumption that the written word will be read, understood and remembered as, being on paper, the reader can refer back anytime they wish. However, McKinney et al.¹⁴ demonstrated that podcast viewers also refer back to the material more than once, as in their study students who received a podcast lecture viewed it on average 2.56 times as well as having superior examination results than those who attended the original lecture.

As a method of communication, podcasting can in fact be considered a voice performance¹⁶ and the success of a performance may indeed influence learning. Prosody is the rhythm and intonation of speech which phonologists believe can have an effect on improving comprehension.¹⁷ Effective usage of prosody engages the listener and helps guide them through the argument the speaker is making and therefore helps to hold the listeners’ attention. Therefore, a podcast delivered by an effective orator will maximise the listeners’ experience.

There is a dearth of published research describing how systematic review summaries are experienced. One assumption that we could make is that academic staff who are exposed to abstracts and research summaries every day, would be able to distinguish key findings when reading abstracts, but is this the case? Surely all readers, regardless of their educational and professional background, should be able to connect with and comprehend the content of a plain language summary? In this study, we explored whether readers understand key messages without having to read the full review, and if there were differences in understanding between various summaries, including an audio podcast.

Methods

We carried out a randomised prospective experiment to determine reader’s effectiveness of retrieving information from systematic review summaries provided by the Cochrane Library. The study was conducted via email communication.

Cochrane Review

The Cochrane Review we focused on examines the health impacts of the use of electric fans during heat waves¹⁸ and was selected because of its topical nature, during the summer’s hot weather. It was also chosen as it is one of a growing number of Cochrane Reviews that offer an audio podcast in addition to traditional written summaries. The key message that we focused on concerns the review’s finding of mixed evidence on the effect on mortality of using electric fans in heat waves.

Participants

This study used a convenience sample of the authors’ colleagues. Approximately 200 people from three universities on the island of Ireland were offered the opportunity to take part via email. Participation was open to all grades of staff members and not exclusive to academic staff. All of the staff contacted work in schools interested in medicine and healthcare. Thirty-six members of staff agreed to participate. A precise estimate of response rate cannot be determined as the email may have been circulated more widely than the original list. However, based on our estimate that the email invitation was emailed to approximately 200 university staff members, this would represent a 19% response rate. Of those who agreed to participate, 28 were women and eight were men. Over three-quarters of participants came from an academic background (78%) and the remainder were employed by the various universities in administrative roles (22%).

Comparisons

Participants were randomised into four groups:

The abstract group

The plain language summary group

The podcast group

The podcast transcript group

Randomisation was achieved using a method of individual simple randomisation. Although a transcript of the podcast is not included on the Cochrane website, it was included in this study to assess if there are any differences in hearing the author speak, as opposed to reading the same words oneself.

Outcome measurement

Outcomes were measured, again via email, using multiple choice questions about their understanding of an outcome from the review. Participants were asked if they knew of the review and to select which of the following list of options matched their expectation of the effect on mortality of using electric fans during heat waves: (1) Fans increase mortality; (2) Fans decrease mortality; (3) Fans have no effect on mortality; (4) The research evidence is mixed; and (5) I am unsure of the effect. After being randomised, participants were asked to spend no more than 15 min reading or listening to the summary and then to answer again the question about the effect of electric fans on mortality and to indicate whether they would now want to read the full Cochrane Review. Responses were transferred into STATA© version 12 for data analysis.

Results

Of the 36 participants, nine were allocated to receive the abstract, eight were allocated to the plain language summary, eight were allocated to the transcript, and 11 were allocated to receive the link to the podcast. One participant was aware of the existence of the review at the outset, but they had not read it and were not familiar with its findings. All other participants answered that they were unaware that the review existed. Before the summaries were sent to them, 14% of the participants said that the research evidence would show that the effect is mixed, this changed to 53% after engaging with their allocated summary (Table 1).

Table 1.

Whole group pre- and post-test scores.

	Pre-test		Post-test
	Number	Percentage	Number	Percentage
Fans increase mortality	2	5.7	–	–
Fans decrease mortality	7	20	2	5.9
Fans have no effect on mortality	9	25.7	3	8.8
The research evidence is mixed	5	14.3	18	52.9
I am unsure of the effect	12	34.3	11	32.4

After considering their allocated summary, more than half of participants changed their view on the effects of electric fans in heat waves (57%). In total, just over half (53%) of the participants now said that the research evidence is mixed. The figures were 33% for the abstract group, 50% for both the plain language summary and the transcript groups, and 78% for the podcast group (see Table 2).

Table 2.

Percentage of participants who selected the correct outcome after reading the summary.

Selected correct outcome	Whole sample	Abstract	Plain language summary	Podcast	Transcript
Yes	52.9	33.3	50	77.8	50
No	47.1	66.7	50	22.2	50

Using Fisher’s Exact Test, the difference between these groups was not found to be statistically significant (p = 0.31). Looking at the difference between all three forms of written summary and the podcast, less than half of the people allocated written summaries correctly identified that research evidence is mixed (44%) compared with 78% of the podcast group. But this difference is not statistically significant using Fisher’s Exact Test (p = 0.09).

Thirteen (37%) of the respondents said that they would want to read the full review, after having considered their summary. Participants were also given the opportunity to comment on their answers. Comments given pre-test revealed that some participants felt that this would be a difficult thing to measure: ‘I think people may feel better when a fan is used but I think there would be such variability in fans, types, where they are placed, evidence of the effect would be difficult to measure’. One participant even commented that systematic reviews often have inconclusive results: ‘ … expect answer, as for most reviews, is more inconclusive’. As the majority of participants did not wish to read the full review, reasons given ranged from no interest in the area to: ‘as the summary suggests there is no definitive outcome from the review, there is no attraction to read it’. On the topic of the podcast, on participant said that while the podcast was ‘ … clear … the absence of pictures made me listen to the voice. Sounded like something being read out though – a bit boring’. Also two participants expressed confusion with the terminology used, with one stating: ‘the question relates directly to mortality, whereas the summary reports mixed “health” effects. This may be mixed findings in relation to mortality but this is unclear, therefore I have opted for the “unsure of effect” option’.

Discussion

The general aim of this research has been to illustrate the nature of the readers’ experience with systematic review summaries and the potential impact of type of summary on understanding. This study is original in that it has helped to shed light on the potential differences in reading and listening to review summaries both on the impact of understanding and also of generating interest in reading the full review when the abstract does not indicate a conclusive result. The findings from this study indicate that differences do exist in understanding key messages based on the type of summary that is provided. Although statistical analyses were not significant, the data demonstrated potential differences between the groups that should be explored further.

Key findings

In total, slightly over half of participants extracted the key message that the evidence of the use of electric fans in heat waves on mortality is mixed. The percentage getting this correct in the groups varied greatly. Abstracts appeared to be the least effective method for communicating and disseminating information. The abstract group achieved the lowest percentage of correct outcomes (33.3%) compared to the podcast group of whom almost 80% correctly identified the key message, which suggests that listening to a podcast is the most effective method of dissemination. Participants in both the transcript and plain language summary only chose the correct outcome 50% of the time. The discrepancy between the podcast and transcript group is interesting and suggestive of there being an added benefit of hearing words spoken as opposed to reading. This could perhaps be due to the added effect of listening and therefore picking up on phonetic cues, as opposed to simply reading something. It could be the case that participants concentrate intensively when listening but perhaps if they were reading, they may skim text which reduces capacity for information comprehension and retention. On the basis of this result, it does not appear that the content of the podcast is responsible for the increased understanding in this group. Plain language summaries exist to provide a simple, straightforward explanation of the review to all readers, regardless of their scientific or academic background and should be written at a layperson’s level. It may therefore come as a surprise that this summary was not the most successful in terms of being able to identify key messages. This leads to the questions: ‘Are academics good at writing plain language summaries?’ and ‘Should systematic review authors spend more time and care, perhaps even testing them with lay people before publishing them?’ None of the summary methods managed to successfully transfer the key message to everyone which is an interesting finding in this study. This implies that review authors must take more care in how their research is presented when in summary form. Not only has it been found that readers often will only engage in reading the summary⁹ but the impact of the summary will either attract or repel someone to read the full text. Less than half (37%) of participants actually wanted to read the full review on the basis of the summary. Some of the participants just would not be interested in the review subject area but others would not be interested given that the review did not have conclusive findings.

Study limitations

Some limitations are evident in this study. First, it should be noted that, although the study had a small sample size it does offer some evidence that further investigation of the effectiveness of the different Cochrane Review summaries in communicating key messages is warranted. Second, given the limited scope of this research as it was based on just one review, we do not believe that it can be generalised to all reviews. Indeed, by the very nature of systematic reviews, some authors may do a better job at communicating research results than others. One final limitation is the question that was posed to participants which asked specifically about mortality. This question was posed in order to simulate a real-life situation whereby a clinician or policy-maker would access a Cochrane review to answer a specific question. The research by Ely et al.¹⁰ found that clinicians spent on average just 2 min searching for information to answer their question. And indeed, in an emergency situation, information needs to be quickly accessible in a format that is clear and unambiguous. However, some participants in the study appeared to express confusion as the summaries stated that the health benefits of electric fans were mixed without specifically stating mortality. None of the summaries used the word mortality although all stated that the health impacts were mixed. Participants felt unable to make a connection between health impacts and mortality and possibly selected the unsure category on this basis. This also echoes the findings of Lai’s⁸ work, who found that clinicians were unable to make accurate conclusions on their own, in the absence of author-directed conclusions.

Suggestions for further research

On the basis of our findings, we argue that further research is required to confirm whether readers can understand key messages from Cochrane Reviews without having to read the full text. Further research should incorporate a larger sample size, summary of findings tables and utilise more than one review alongside some more in-depth qualitative research to understand how people read and comprehend review summaries. Future research should also consider determining individual learning styles and the effect on understanding summaries as well as recording how long participants spent on the summary and how often they accessed it. McKinney et al.¹⁴ work on university students accessing lecture podcasts has found that the effect of audio podcasts can be boosted by providing transcripts and notes for the listener to read along with and it would be interesting to see if this has an effect on Cochrane podcasts.

Conclusions

The current research demonstrates that there may be added benefits for scientific research to be presented in new ways to allow their full potential to be explored. It even more important to consider the reach, accessibility and knowledge required to interact with systematic reviews as it is known that clinicians and decision-makers have such a short amount of time to engage with them. With this knowledge, it is worth considering that a multimedia approach may achieve greater results than an abstract or plain language summary and that further research is needed in this area. More care needs to be taken by authors to ensure that summaries can quickly and accurately provide the reader with the key messages from the review. The findings suggest that people may better connect with listening to information as the tone and inflection in the author’s voice may help gain attention and increase information retention in memory and its subsequent retrieval.

Footnotes

Declarations

References

Cochrane AL. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust, 1972.

Law M and Baum C. Evidence-based occupational therapy. Can J Occup Ther 1998; 65: 131–135.

Gervais IS, Poirier A, Van Iterson L, et al. Attempting to use a Cochrane review: experience of three occupational therapists. Am J Occup Ther 2002; 56: 110–113.

Rosenbaum SE, Glenton C, Nylund HK, et al. User testing and stakeholder feedback contributed to the development of understandable and useful summary of findings tables for Cochrane reviews. J Clin Epidemiol 2010; 63: 607–619.

Zhelev Z, Garside R and Hyde C. A qualitative study into the difficulties experienced by healthcare decision makers when reading a Cochrane diagnostic test accuracy review. Syst Rev 2013; 14: 32.

Glanville J, Haines M and Auston I. Getting research into practice. Finding information of clinical effectiveness. Br Med J 1998; 317: 200–203.

Groffman PM, Stylinski C, Nisbet MC, et al. Restarting the conversation: challenges at the interface between ecology and society. Front Ecol Environ 2010; 8: 284–291.

Haynes RB, Mulrow CD, Huth EJ, et al. More informative abstracts revisited. Ann Intern Med 1990; 113: 69–76.

Novack L, Jotkowitz A, Knyazer B, et al. Evidence-based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgrad Med J 2006; 82: 817–822.

10.

Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. Br Med J 1999; 319: 358–361.

11.

Evans C. The effectiveness of m-learning in the form of podcast revision lectures in higher education. Computers & Education 2008; 50: 491–498.

12.

Lazzari M. Creative use of podcasting in higher education and its effect on competitive agency. Computers & Education 2009; 52: 27–34.

13.

Kemp PK, Myers CE, Campbell MR, et al. Student perceptions and the effectiveness of podcasting in an Associate Degree Nursing program. Teach Learn Nurs 2010; 5: 111–114.

14.

McKinney D, Dyck JL, Luber ES. iTunes University and the classroom: can podcasts replace professors? Computers & Education 2009; 52: 617–623.

15.

Vajoczki S, Watt S, Marquis N, et al. Podcasts: are they an effective tool to enhance student learning? A case study. J Educ Multimed Hypermedia 2010; 19: 349–362.

16.

Madsen V, Potts J. Voice-cast: the distribution of the voice via podcasting. In: Neumark N, Gibson R, van Leeuwen T (eds) Voice: Vocal Aesthetics in Digital Arts Media. Massachusetts: MIT Press, 2010, pp.34–60.

17.

Schafer A, Speer S, Warren P, et al. Intonational disambiguation in sentence production and comprehension. J Psycholinguist Res 2000; 29: 169–182.

18.

Gupta S, Carmichael C, Simpson C, et al. Electric fans for reducing adverse health impacts in heatwaves. Cochrane Db Syst Rev 2012; Art. No.: CD009888.