Abstract
This article presents the findings of a randomized controlled trial evaluation of the effects of a revised version of the volunteer mentoring programme, Time to Read. Participating children received two 30-minute mentoring sessions per week from volunteer mentors who carried out paired reading activities with the children. The current trial involved 512 children aged eight to nine years from 50 primary schools. The programme was found to be effective in improving decoding skills (d=+.15), reading rate (d=+.22) and reading fluency (d=+.14) and there was some evidence of a positive effect in relation to the children’s aspirations for the future (d=+.11). However, no evidence was found of the programme having an effect on reading comprehension or reading confidence and enjoyment of reading. The article concludes by suggesting that mentoring programmes using non-specialist volunteers can be effective in improving foundational reading skills but would appear to be less effective in terms of improving higher-order skills such as comprehension. The article also suggests that such programmes are likely to be most effective if concentrating on core reading activities rather than attempting to address reading outcomes indirectly through improving children’s confidence or wider enjoyment of reading.
Introduction
One-to-one mentoring is a popular and effective form of instruction employed by schools to prevent early reading failure and improve academic outcomes. Mentoring programmes can vary widely in terms of their scope, design and effectiveness (Shanahan, 1998; Wasik and Slavin, 1993) and programmes that have been found to be most effective tend to be those that: address multiple aspects of the reading process; have structured content; are delivered frequently; and have a strong emphasis on phonics (Slavin et al., 2009; Wasik and Slavin, 1993). Sound Partners is one example of a mentoring programme that has been found to be effective and is both structured and phonics-based. The programme consists of 100 scripted lessons that last for 30 minutes each and focus on phonological awareness, word recognition, text reading, and writing. Lessons take place four times a week for one school year and are delivered by trained volunteers. Sound Partners has been shown to considerably improve literacy skills for children aged six to seven years, with effect sizes ranging between +.28 and +.83 (Vadasy et al., 1997a, 1997b, 2000).
Programmes can be delivered by teachers, peers, parents and volunteers, and there is some evidence to indicate that teachers and college students tend to be particularly effective as tutors (Elbaum et al., 2000; Slavin et al., 2009) and that tutors work best under the supervision of qualified teachers or reading specialists (Elbaum et al., 2000). Reading Recovery is perhaps the most familiar example of a one-to-one mentoring programme that is led by teachers. It is aimed at young children aged five to six years who are struggling to learn to read and is delivered by qualified teachers for 30 minutes every day for approximately 20 weeks. In order to take account of emerging research evidence, the programme now incorporates a greater emphasis on phonics and phonological awareness (Brooks, 2007) although arguably not to an extent that is consistent with the National Literacy Strategy (Singleton, 2009).
Reading Recovery lessons involve a number of elements which include: reading and re-reading books that the child is already familiar with; teaching the child how to use letters and sounds to make words; composing, writing and reading stories; carrying out activities that will help the child to make connections between reading and writing; and finally, reading new and increasingly challenging books. Reading Recovery has been implemented in schools in a number of countries around the world including the UK, and children who have taken part in the programme have been shown to make substantially greater gains in reading and phonics skills than control children, with effect sizes ranging from +.49 to +.87 (Burroughs-Lange and Douëtil, 2007; Pinnell et al., 1994). These gains however, have been shown to diminish after only one year (Hurry and Sylva, 2007). Furthermore, while the programme is effective for many pupils, it is expensive and appears to be least effective for children most at risk of reading failure (Elbaum et al., 2000; Reynolds and Wheldall, 2007).
Overall, the types of one-to-one reading interventions described above have been found to significantly improve children’s reading skills (Elbaum et al., 2000; Ritter et al., 2009; Slavin et al., 2009). Indeed, mentoring appears to be particularly effective in terms of improving listening comprehension, reading fluency and decoding, with effect sizes ranging from +.30 to +.68. Smaller effects have been reported for improvements in reading comprehension, with effect sizes ranging between +.18 and +.28 (Elbaum et al., 2000; Ritter et al., 2009). However, beyond studies on the effectiveness of differing structured approaches by trained mentors and/or programmes delivered by specialist mentors, there is relatively little focusing on the effectiveness of programmes that are less structured and involve non-specialist volunteer mentors. The studies that do exist have tended to find improvements in oral fluency but little or no improvement in global reading skills (Erion, 1994; Miller, 1994). These studies have also tended to be small-scale and there has been no attempt to tease out the specific ways in which such non-specialist mentoring programmes are effective. The one exception to this is a recent large-scale evaluation of Experience Corps (Morrow-Howell et al., 2009), a programme that recruits older adults from the community to mentor struggling readers in local schools. Mentors receive some training in literacy and relationship building and are provided with materials to use during the mentoring sessions. Experience Corps is delivered to pupils on a one-to-one basis outside the classroom setting, and the evaluation found small effect sizes (on average d=+.11) for reading outcomes.
With this in mind, the purpose of the present article is to test the effects of volunteer mentoring programmes through reporting the findings of a large-scale trial of the Time to Read mentoring programme, which uses non-specialist volunteers recruited from local businesses.
The Time to Read programme
Time to Read is a mentoring programme aimed at struggling readers aged eight to nine years. This is a different programme to Time for Reading, which also uses volunteer mentors recruited from the community. In contrast to Time to Read, Time for Reading is aimed at four- to five-year-old children and is classroom and small groups based rather than a on-to-one intervention. An experimental evaluation of Time for Reading found no impact of the programme on children’s reading outcomes (Elliott et al., 2000). Time to Read is run by Business in the Community and is part of a wider suite of mentoring programmes involving people from local businesses. At the time of the evaluation, Time to Read involved 556 mentors, 688 pupils and 86 schools across Northern Ireland.
Trained volunteers are recruited from the local business community and deliver the programme to participating children who receive two 30-minute mentoring sessions per week for one school year. The programme is relatively unstructured, and each session involves the mentor and child engaging in paired reading activities together. During a two-hour orientation session mentors are encouraged to use a number of simple strategies while reading with the children, which include: using repeating and alternating reading methods; sounding out and looking up the meaning of unfamiliar words; talking about what is being read and stopping at various points during the text to ensure the child understands what they are reading. The mentoring sessions take place outside the classroom setting and schools taking part in the programme are supplied with books that the mentor and pupil can choose from for their session. However, pupils are also free to choose other books if they so wish. In addition to the mentoring sessions, mentors are encouraged to bring their mentees on workplace visits so that the pupils can learn about their mentor’s job, see where they work and meet some of their colleagues. The Time to Read programme sits alongside, rather than replacing, other reading improvement strategies that schools might employ.
A previous qualitative evaluation of the Time to Read pilot programme explored the views of teachers, principals and volunteers involved in delivering the programme (Deloitte, 2003). The findings of this study showed that there was considerable agreement in relation to the perspectives of schools and volunteers that Time to Read made a valid and important contribution to literacy. This improvement in literacy was perceived to be a consequence of positive behavioural changes in children, which included increased confidence and increased enjoyment in books and reading.
From 2006 to 2008 the programme was the subject of a first randomized controlled trial that is reported in detail elsewhere (Miller et al., 2009). This first trial found no evidence that the programme had any effect on reading or a range of other non-reading outcomes. The one exception was in relation to the children’s future aspirations where a small positive effect was found (d=+.17, p = .032). These findings led the programme developers to double the dosage of the programme from 30 minutes once a week to 30 minutes twice a week and to provide their mentors with more explicit instructions in relation to paired reading strategies to use during the mentoring session. The aim of this second trial is therefore to determine whether this revised and slightly more intensive Time to Read programme is effective in improving reading outcomes for struggling readers aged eight to nine years.
Method
Outcomes and measures
The outcomes were chosen in close consultation with the programme developers to accurately reflect the aims of the programme. The core outcomes relating to reading skills comprised: decoding; reading rate; reading accuracy; reading fluency; and reading comprehension. Decoding was measured using a non-word reading test while reading rate, accuracy, fluency and comprehension were measured using a test based on real words.
The other outcomes measured for this trial were attitudinal and comprised: enjoyment of reading; reading confidence (or efficacy); and aspirations for the future. Table 1 summarizes the outcomes and measures that were used together with details of their reliability.
Outcomes and measures
In addition to the outcome measures summarized in Table 1, data relating to children’s socio-economic status (via postcode), gender and age were also collected. Finally, mentors were required to keep a log of each mentoring session. This information included the number of sessions, the duration of each session and a checklist of the reading strategies used by the mentor during the session.
Sample
Fifty primary schools from across Northern Ireland agreed to take part in this second trial. Table 2 compares the characteristics of the schools in the sample to the population of primary schools in Northern Ireland. While the two are broadly comparable, the sample contains an over representation of schools from the Belfast Education and Library board and schools with higher levels of Free School Meal eligibility.
School sample characteristics compared to the Northern Ireland population
In total, 512 pupils took part in the trial. Teachers in participating schools were asked to identify pupils in their class who were below average in reading and lacked confidence in reading, and who the teacher felt would benefit from the programme. Children with a statement of special educational need were not eligible to participate because the training that mentors received was not sufficiently specialist to meet the needs of children who may have more complex reading difficulties. There were a limited number of mentoring places available in each school and eligible children were randomly assigned to either the intervention or control group by the research team using the random selection function in SPSS. In addition, mentors were randomly allocated to the children they would mentor over the year.
Table 3 shows the breakdown of the sample by gender and group allocation. Overall, 263 children were randomly allocated to the intervention groups and 249 to the control group. 59 percent of the sample was male and 41 percent female.
Pupil sample characteristics by gender and group allocation
Table 4 compares the intervention and control groups in relation to their scores at pre-test on the outcome measures used. For four out of five outcomes there were no statistically significant differences between intervention and control groups, demonstrating that the randomization process worked in producing two equivalent groups. Where differences did exist, these were controlled for in the statistical analysis used as described below.
Differences between groups on the outcomes at pre-test
These differences were calculated using multi-level linear regressions using the pre-test score as the dependent variable and group allocation (intervention or control) as the independent variable. This ensured that the clustered nature of the data was taken into account when calculating the statistical significance of the differences between the groups on the outcomes.
Procedure
Written parental consent was obtained for each child to: participate in the evaluation; be randomly allocated to the intervention or control group; and be tested on the outcome measures at two time points (pre- and post-test). Children’s direct informed consent was also sought prior to completing the outcome measures.
Outcomes were assessed pre- and post-intervention (October 2009 and June 2010) by researchers who were trained in the use of the above measures and who were blinded to the allocation of participants. The reading tests were administered on a one-to-one basis and the attitudinal (non-reading) measures were administered on a group basis. Figure 1 summarizes the participants’ route through the trial.
Flow of participants through the trial
Findings
Although pupils were individually randomly assigned to the intervention and control groups, they were nested within school classes and so, to address this, the main analysis took the form of the estimation of multilevel models with pupils (level one) nested within schools (level two). Full details of the main models for each outcome variable are provided in Table 5. The main findings, including the adjusted post-test means (controlling for any differences in pre tests) and the effect sizes are summarized in Table 6. Effect sizes were calculated using Cohen’s d, which is the difference between the intervention and control group post-test means (adjusted for pre-test scores) divided by the pooled standard deviation for the whole sample.
Main multi-level models for each outcome variable (standard errors in parentheses)
Summary of the effects of the Time to Read programme on child outcomes
As can be seen from Table 6, there is evidence that those who participated in the programme achieved higher scores, on average, than the control group in terms of decoding, reading rate and fluency, achieving effect sizes of +.15, +.22 and +.14 respectively. No evidence of any differences between the groups was found in relation to: reading accuracy; reading comprehension; enjoyment of reading; aspirations for the future; and reading efficacy (confidence).
The main models described in Table 5 were expanded using interaction terms to explore whether the programme had differential effects for: boys or girls; for children from more deprived areas compared to children from more affluent areas; for children who are initially poor readers; and, in relation to the intervention group, whether the programme worked better for those who received more of the programme.
A concern with such sub-analyses is that they notably increase the risk of committing a Type I error and can sometimes resemble what has been likened to a ‘fishing exercise’ (Shaffer, 1995). As such, any isolated significant effects found within these sub-analyses were ignored unless they were part of a consistent pattern. Using this approach no credible or consistent pattern of subgroup differences was found.
Discussion and conclusions
This trial has found evidence that the slightly revised and more intensive Time to Read programme was effective in improving certain reading outcomes for children, particularly in relation to decoding, reading rate and reading fluency. The first trial (Miller et al., 2009) found that Time to Read significantly improved aspirations for the future and some corroborating evidence for this has also been found in this second trial, although non-significant. However, there was no evidence to suggest that the programme improved the children’s reading accuracy, reading comprehension or that it improved their enjoyment of reading or reading confidence.
Only a small number of studies have evaluated the specific impact of programmes that are loosely structured and use non-specialist volunteers as mentors. While the findings from this trial are consistent with the impact of other similar volunteer mentoring programmes on reading outcomes (Elbaum et al., 2000; Morrow-Howell et al., 2009; Ritter et al., 2009; Slavin et al., 2009) it remains relatively unknown how or why volunteer mentoring programmes are effective. In addition to this, the body of evidence surrounding mentoring programmes arises from trials that are mostly small-scale.
This current Time to Read evaluation contributes to this evidence base as one of the largest trials conducted in the area of volunteer mentoring that has used such a robust design. Slavin and Smith (2009) argue that larger studies such as this one are likely to provide a more accurate representation of the true effects of volunteer mentoring programmes than smaller, underpowered trials that up until now have tended to comprise the body of evidence supporting the effectiveness of volunteer mentoring.
Time to Read is a relatively unstructured mentoring programme that uses non-specialist volunteers as mentors. To understand how Time to Read might work in terms of improving reading outcomes it is useful to compare it, and its effects, to other types of mentoring programmes. Mentoring programmes that use teachers and trained specialists, or a highly structured curriculum tend to achieve larger effect sizes, particularly for reading comprehension, than programmes that use non-specialist volunteers as mentors. This has been demonstrated in reviews of mentoring programmes by both Slavin et al. (2009) and Elbaum et al. (2000) and is apparent, for example, when we compare the effects achieved in evaluations of Sound Partners (d=+.28 to +.83), a highly structured programme, and Reading Recovery (d=+.49 to +.87) which is delivered by specialist teachers, to the smaller effects found in the trials from both Experience Corps and Time to Read.
This type of (non-specialist) volunteer mentoring programme, based on listening to children read, tends to have most effect on some of the core skills such as decoding, rate and fluency. To put it clearly, programmes like Time to Read that are based on listening to children read out loud would appear to have most effect on children’s ability to read out loud. These programmes are not as effective as others in terms of improving higher-level skills such as comprehension. However, and as is demonstrated by other programmes interventions need to be highly structured and delivered by specialist reading instructors in order to make the large gains in comprehension evident in the evaluations of these programmes. This may not be possible or feasible within the constraints on organizations such as Experience Corps and Time to Read, which depend upon the goodwill of individual volunteer mentors and/or their employers.
A recent practice guide aimed at improving comprehension for children aged five to eight years provides five evidence-based recommendations for improving comprehension (Shanahan et al., 2010). These include: teaching pupils how to use reading comprehension strategies; teaching pupils to identify and use the text’s organizational structure to comprehend, learn and remember content; guiding pupils through focused, high-quality discussion on the meaning of the text: asking follow-up questions to encourage and facilitate discussion; and having pupils lead structured, small-group discussions. It is evident therefore that improving reading comprehension requires specific knowledge and expertise on the part of the teacher or mentor, and this may well be beyond the remit or scope of many volunteer mentoring programmes. However, this is not a problem per se but it does suggest the need to be realistic in relation to the expectations that surround such volunteer programmes and what they are likely to achieve.
The findings from the current trial also show that reading skills improved independently of the non-reading outcomes that were measured, namely reading confidence and reading efficacy. The evidence therefore suggests that improvements in reading skills are not necessarily dependent upon and/or mediated by attitudinal, non-reading variables. The one exception to this is the finding that Time to Read increased children’s future aspirations; however, it is likely that this reflects the workplace visit component of the programme rather than being a product of being listened to reading. This finding tends to suggest that volunteer mentoring programmes such as Time to Read are therefore best focused on core reading activities if their goal is to improve children’s reading skills rather than attempting to do this through indirect means such as increasing their confidence or enjoyment of reading.
In conclusion, this trial has found robust evidence that the refined Time to Read programme is effective in improving some of the core skills (namely decoding and reading fluency) that children need in order to become effective readers. There was no evidence, however, of any impact on reading comprehension. The findings from this trial make an important contribution to the existing evidence base in this area that has consistently demonstrated that mentoring programmes are effective in terms of improving reading outcomes for children. In particular it has extended this evidence base by contributing to the smaller body of evidence that shows that mentoring programmes that use volunteers rather than specialized teachers or highly trained mentors are effective in improving key skills such as decoding and reading fluency but are less effective in terms of improving reading comprehension.