Abstract
Despite rapid growth in literacy-related programmes and evaluation in sub-Saharan Africa, little critical attention has been paid to the relevance of assumptions that underlie existing assessment methods. This study focuses on the issue of timing in the assessment of oral reading fluency, a critical component of successful reading (Chard, Vaughn, & Tyler, 2002; National Institute of Child Health and Human Development (NICHD), 2000; Pikulski & Chard, 2005). Within the context of the Primary Math and Reading Initiative, a randomized controlled trial of several instructional interventions in Kenya, timed and untimed Early Grade Reading Assessment (EGRA) oral reading fluency and reading comprehension tasks were administered to 4385 students in 95 government and 125 informal schools. Using the data from the EGRA – whose administration has expanded within sub-Saharan Africa recently – we found that students did not perform significantly better on the assessments when they had more time. This pattern largely held when we examined the effects disaggregated over student ability level. This suggests that timed assessments, which are faster to administer and logistically easier, are appropriate for use in Kenya.
The recent wave of national and large-scale reading programmes in sub-Saharan Africa and across the developing world has not been accompanied by rigorous analysis of some of the critical assumptions about how to measure reading outcomes in this region. In particular, there has been surprisingly little research on how to validly assess reading outside of the Western countries where many standardized assessments generally originate. This study aims to contribute to the technical research base on reading assessments in multilingual settings in sub-Saharan Africa by examining the issue of the stability of reading rates measured by oral reading fluency (ORF) assessments that differed in administration time.
Kenya is representative of many countries in sub-Saharan Africa that are struggling to improve literacy. Several studies implemented in Kenya since 2006 have shown very low reading outcomes in both formal and non-formal settings in the slums (Piper & Mugenda, 2012, 2013; Piper, Zuilkowski, & Mugenda, 2014), across urban and rural locales (National Assessment System for Monitoring Learner Achievement (NASMLA), 2010), and in mother tongue (Piper, Schroeder, & Trudell, 2015), as well as in Kiswahili and English (National Assessment System for Monitoring Learner Achievement, 2010; Piper & Mugenda, 2012; Piper & Zuilkowski, 2015; Uwezo, 2012). The measures employed in the Kenyan studies above have similar characteristics. The NASMLA measures literacy competencies utilizing spelling tasks and inferring meaning from short passages. The Early Grade Reading Assessment (EGRA) employs timed oral reading fluency measures with comprehension items, among other literacy assessment tools. Kenya is using a range of methodological approaches to measure literacy, including some assessments that are timed and others that are not. As interest in and use of these tools expand rapidly in Kenya and in the region more broadly, a critical examination of the appropriateness of timed and untimed assessments is required. This study examines issues related to timing in the measurement of oral reading fluency and reading comprehension in the context of the Primary Math and Reading (PRIMR) Initiative, a series of nested randomized controlled trials in government and non-formal primary schools.
Background and context
Oral reading fluency as an indicator of reading competency
Over the past two decades, there has been a great deal of convergence among researchers regarding the process of learning to read (Hulme & Snowling, 2013; Snow, Burns, & Griffin, 1998). Pikulski and Chard (2005) identified fluency as ‘part of a developmental process of building decoding skills that will form a bridge to reading comprehension and that will have a reciprocal, causal relationship with reading comprehension’ (p. 512). When examining fluency, researchers generally consider three components: accuracy, speed or automaticity, and prosody, meaning reading with expression (Hudson, Lane, & Pullen, 2005; Kuhn, Schwanenflugel, Meisinger, Levy, & Rasinski, 2010; Kuhn & Stahl, 2003). As posited by LaBerge and Samuels (1974), fluency – and automaticity in particular – is an important factor in the development of reading comprehension skills. A beginning reader who spends most of his or her time laboriously decoding words cannot devote much cognitive space to comprehension of the material. A more fluent reader will be more capable of understanding and thinking critically about what he or she is reading (National Institute of Child Health and Human Development, 2000).
In the United States, the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) (Good & Kaminski, 2002) is one of the most common tools for measuring oral reading fluency, and the EGRA oral reading fluency assessment is modelled after it. In DIBELS, a student reads a selected passage out loud for a specified period of time, generally 1 minute (Rasinski, 2004). The assessor notes how many words the student read, and how many of these words were correct. The most commonly used oral reading fluency statistic is words correct per minute (WCPM). The oral reading fluency subtasks utilized in EGRA therefore measure accuracy and speed, but not prosody, a limitation noted by critics (Deeney, 2010). Even so, given that it offers a free and flexible framework for assessment, the EGRA is likely the most widely used literacy measurement tool in the developing world.
The validity of oral reading fluency assessments has long been a topic of research in the United States (Marston, 1989; Shinn, 1998). Researchers have focused on criterion validity, comparing the results of the assessments with existing normed tests. For example, Schilling, Carlisle, Scott, and Zeng (2007) compared an oral reading fluency assessment with the Iowa Test of Basic Skills and found correlations with the total reading score and reading comprehension subtests between .63 and .75. Correlations between .49 and .83 have been found between oral reading fluency assessments and state assessments in Colorado (Shaw & Shaw, 2002), North Carolina (Barger, 2003), Florida (Roehrig, Petscher, Nettles, Hudson, & Torgesen, 2008), Arizona (Wilson, 2005), Ohio (Vander Meer, Lentz, & Stollar, 2005), and Minnesota (Silberglitt, Burns, & Madyun, 2006). Studies have also found that DIBELS has predictive validity regarding the development of reading skills over the early elementary years, with kindergarten DIBELS results predicting first- and second-grade ORF scores (Burke, Hagan-Burke, Kwok, & Parker, 2009). The bulk of the evidence thus indicates that the DIBELS approach to measuring oral reading fluency is valid for the assessment of reading in the United States (Goffreda & DiPerna, 2010). However, some researchers have questioned whether some reported correlations are overestimations due to methodological issues (Valencia, Smith, Reece, Li, Wixson, & Newman, 2010).
Despite this extensive body of research in the United States, to our knowledge, only one criterion validity examination has been undertaken in the developing world comparing the EGRA oral reading fluency assessment to other assessments. Vagh (2012) compared the fluency measures typically used in EGRA with the Annual Status of Education Report (ASER) reading and mathematics tests, along with a battery of other assessments. The results showed that the EGRA assessments were highly correlated with the ASER reading test, with concurrent validity coefficients between .90 and .94 (Vagh, 2012). Despite this gap in the literature, research conducted with English language learner (ELL) populations in the United States may offer insights as to the assessment’s validity with students learning and being tested in a language in which they are not yet fluent, as is the case for many Kenyan children. Crosson and Lesaux (2010), in their analysis of fifth-grade Spanish-speaking English language learners in the south-western United States, found that oral proficiency in English mediated the relationship between text reading fluency and reading comprehension. This suggests that the relationship between ORF and reading comprehension may not be as close in Kenya – where children are learning in English and Kiswahili rather than in their mother tongues. However, in comparing the relationship between the DIBELS ORF scores and the state third-grade reading comprehension assessment for ELLs and non-ELLs, a study conducted in Colorado found that the correlations were similar across the two groups (Scheffel, Lefly, & Houser, 2012). Among current and former ELLs in California, a reading fluency assessment correlated with comprehension scores at a level similar to that found in other studies (Quirk & Beem, 2012). The authors noted, however, that many of the students in the sample exhibited a large gap between their fluency and comprehension scores, suggesting that some children may have been able to successfully decode words, but not grasp the words’ meanings.
A second aspect of this method to critically consider, and the focus of the following analyses, is the sensitivity of student rate scores across different time periods that students are allowed for the assessment. The 1-minute rule of thumb has been accepted among practitioners in the field in sub-Saharan Africa without much question. However, Valencia et al. (2010) found, when comparing 1-minute and 3-minute scores for second- and fourth-graders in the United States, that students read more slowly and with less accuracy as time passed. Similar results were found on the 2002 National Assessment of Educational Progress (NAEP) Oral Reading Study (Daane, Campbell, Grigg, Goodman, & Oranje, 2005). Alternatively, Deeney (2010) argued that struggling readers may perform better over a longer period of time on a passage task, due to the assistance of contextual information. While there has been some discussion in developed countries about the appropriate length of the timed period, from 30 seconds to 3 minutes, no peer-reviewed research has looked at comparable timing issues in the developing world, or whether these assessments should be timed at all.
EGRA and the assessment of oral reading fluency in developing countries
Vast sums of international aid have been invested in literacy improvement programmes in developing countries. The United States Agency for International Development (USAID) alone intends to provide technical assistance to improve 100 million children’s reading skills by 2015 (USAID, 2013), with the Global Partnership for Education (Global Partnership for Education, 2012) and the United Kingdom’s Department for International Development (DFID, 2010) having a similar focus on improving literacy. In order to quantify the returns on these expenditures, education systems and interventions need valid measures of reading. As text-based assessment measures can be developed using texts adapted in local languages, oral reading fluency assessment is an appealing, inexpensive option that avoids many of the problems with cross-cultural translation. However, the quick expansion of oral reading fluency assessment – and particularly EGRA – in developing countries requires the careful examination of the technical issues discussed in the previous section.
The EGRA development process began in 2006 with work by Helen Abadzi, Luis Crouch, and colleagues (Abadzi, 2006; Abadzi, Crouch, Echegaray, Pasco, & Sampe, 2005), funded by a USAID contract to RTI International and by work within the World Bank. EGRA has since been used in more than 65 countries and adapted into more than 100 languages (Dubeck & Gove, 2015; Gove & Wetterberg, 2011). As indicated earlier, the EGRA oral reading fluency task is similar to the DIBELS oral reading fluency task. EGRA does, however, include comprehension questions as part of the assessment, so comprehension can be assessed directly as well as inferred from the WCPM score.
Despite EGRA’s popularity, it has been the target of criticism from researchers and other stakeholders who feel it presents an incomplete picture of literacy learning and may lead teachers to emphasize speed and decoding over other important aspects of reading, such as comprehension and prosody (Goodman, 2014). Similar arguments have been made against the use of DIBELS in the United States (Samuels, 2007). In addition, some have argued that this assessment approach is inappropriate for use in sub-Saharan Africa, where early reading levels are so low that the majority of children may have scores of zero (Hoffman, 2012). In this paper, however, our goal is to examine a critical technical issue of immediate importance to programme managers and evaluators rather than engaging in the broader philosophical debate over the role of testing. While the developers of the EGRA made extensive efforts to ensure the validity of the basic assessment before it went into widespread use (Dubeck & Gove, 2015; Gove & Wetterberg, 2011), no peer-reviewed publications have used the resultant data to examine technical assessment issues such as the most accurate and reliable time span for oral reading fluency assessment in non-Western countries. This study aims to begin to fill that gap.
Literacy instruction and outcomes in Kenya
In Kenya, recent expansions in access to primary education have not been paralleled by improvements in school quality (Sifuna, 2007). While enrolment in the early primary grades is now almost universal, the official primary pupil–teacher ratio has increased from 28:1 in 1998 to 47:1 in 2009 (UNESCO, 2014). As of 2012, the average class size in PRIMR government schools was slightly smaller, with 43.2 pupils, but there was a great deal of variability in size, with a standard deviation (SD) of 20.1. It would be difficult for even the most skilled educators to teach a classroom of 43 children to read, but the problem is compounded by the fact that most children in Kenya are learning to read simultaneously in Kiswahili and English, neither of which generally is their mother tongue (Dubeck, Jukes, & Okello, 2012). Moreover, classroom practice does not align with current recommendations regarding literacy instruction. In an observational study in Kenya’s Coast province, Dubeck and colleagues (2012) found that most teachers were instead using whole-class oral repetition as their primary instructional method, particularly in English. Piper and Mugenda (2012) found that in PRIMR schools, only 2.4 minutes were spent per class period with pupils reading books at all.
These challenges in the classroom contributed to the Kenyan education system’s poor literacy outcomes on myriad assessments. Data from an assessment in 2000 showed that, on average, Kenyan children did not reach intended benchmarks (Onsomu, Nzomo, & Obiero, 2005). For instance, just 21% of sixth-graders had a ‘desirable level’ of reading, indicating the ability to handle the reading requirements of the following academic year. In addition, the percentage of students achieving ‘minimum’ mastery decreased between 1998 and 2000. Outcomes have not improved in more recent surveys. One recent national survey implemented by the Kenya National Examinations Council (KNEC) (Wasanga, Ogle, & Wambua, 2010) found that more than half of third-graders were still at the emergent-reader stage; while a second study, implemented by Uwezo from household samples, concluded that seven out of 10 children in third grade could not read a second-grade level passage (Mugo, Kaburu, Limboro, & Kimutai, 2011; Uwezo, 2012). Concerns about these poor literacy outcomes led to the design of the PRIMR Initiative, which aimed to improve student achievement in reading.
The Primary Math and Reading Initiative
The PRIMR Initiative was developed in response to the growing body of evidence that Kenyan children were not learning to read. PRIMR was a research programme organized to improve reading and mathematics achievement in Kenya. Funded by USAID and DFID, PRIMR was a partnership among USAID/Kenya, DFID/Kenya, and the Kenyan Ministry of Education, Science and Technology (MOEST), with staff from the research firm RTI International serving as technical experts supporting the MOEST in the Initiative. Initial empirical analyses after one year of implementation showed that pupils in randomly assigned PRIMR zones were twice as likely to read with fluency and comprehension as those in control schools (Piper, Zuilkowski, & Mugenda, 2014).
The PRIMR baseline evaluation (administered in January 2012) incorporated two tools that allowed researchers to investigate the issue of timing in oral reading fluency assessment in Kenya. The availability of these data allowed the authors to investigate the findings more fully to determine whether the EGRA results were robust to different choices about the amount of time allowed for fluency assessment subtasks.
The PRIMR January 2012 baseline assessment was substantial, assessing 4385 pupils in three subjects: Kiswahili, English, and mathematics. The baseline study presented pupils with two separate reading passages in English and another two in Kiswahili. For each language, one of the assessments was a story approximately 60 words long accompanied by five comprehension questions, including factual, textual inferential (i.e., answers could be inferred from information in the text), and inferential questions (i.e., pupils had to infer meaning from outside of the text). For reference, the MOEST’s fluency benchmarks were established by KNEC in August 2012, utilizing analysis of the PRIMR baseline results. Particular attention was paid by KNEC to the fluency rates associated with high levels of comprehension, and that relationship was used to create benchmarks for fluent and emergent readers. Fluent readers are identified at 65 WCPM for English and 45 WCPM for Kiswahili. Emergent readers are identified at 30 WCPM for English and 17 WCPM for Kiswahili. The number of comprehension questions administered to students depended on how far in the passage the child read. This assessment was timed to 1 minute, as is standard in DIBELS-type measures (Good & Kaminski, 2002). We will refer to this assessment as the time-delimited passage.
The second assessment was a passage approximately 40 words long accompanied by four comprehension questions, again including factual, textual inferential, and inferential questions. This assessment was timed to 3 minutes, but given that assessors were trained to move a pupil to the next word if they were unable to read an individual word after 3 seconds, this assessment was essentially untimed for the vast majority of pupils in the sample. We will refer to this passage as the passage-delimited story. Both the time-delimited and passage-delimited assessments had an early stop rule, which meant that if pupils read 10 consecutive words incorrectly, the assessment was stopped. It should be noted that the time-delimited and passage-delimited stories differed somewhat, including the number of words, the number of comprehension questions and the stories themselves.
The data set analysed in this paper had individual-level data for both the time-delimited and passage-delimited story results and their associated comprehension questions. This design allowed the authors to investigate the stability of early reading assessment outcomes in relationship to the length of time that the pupil was given to read. Specifically, we were interested in the following research questions (RQs):
Were there differences in the fluency and reading comprehension scores in English and Kiswahili, depending on whether the assessment was time delimited or passage delimited?
Were there differences in the fluency and reading comprehension scores for fast and slow readers, depending on whether the assessment was time delimited or passage delimited?
Were there differences in total and attempted reading comprehension scores, depending on assessment timing?
Research design
Site
With an estimated population of 43 million, Kenya is the second-largest country in population and geographic size in the East African Community, of which it has been a member since 1999 alongside Uganda, Tanzania, Burundi, and Rwanda (World Bank, 2013b). Kenya’s population is growing at 2.7% annually and 24% of the population lives in urban areas (World Bank, 2013a). Kenya’s gross domestic product (GDP) is growing at 4% annually and is the largest in East and Central Africa (World Bank, 2013b). In the education sector, Kenya’s performance has been encouraging with respect to the Millennium Development Goals, and it remains on track to meet its targets for enrolment and primary completion by 2015. However, as discussed above, literacy outcomes remain poor despite the improvement in access to primary education.
The USAID-funded portion of the PRIMR Initiative (August 2011–August 2014) 1 operated in peri-urban and rural settings in Nairobi, Kiambu, Murang’a, Nakuru, and Kisumu counties, in both formal government-funded schools and non-formal private schools in slum settlements. The counties were selected by the MOEST and USAID, and the selection of zones (groups of between eight and 20 schools) within the counties was random, as was the assignment of zones to treatment groups. The PRIMR January 2012 sample of formal schools (N = 220) was more urban and wealthy than the average formal school in Kenya (Piper & Mugenda, 2013 ). However, USAID PRIMR also included 225 non-formal schools in Nairobi’s slums at the baseline. For the January 2012 PRIMR study, the PRIMR team randomly selected 25 formal and non-formal zones of schools from the list of eligible zones and then randomly selected schools from within those zones (Piper & Mugenda, 2013). The January 2012 sample served as the baseline assessment for the USAID PRIMR Initiative intervention.
Measures
The measures used to collect the fluency and comprehension data presented in this paper were based on earlier versions of EGRA literacy assessments in Kenya. In particular, the Kiswahili and English tools used in this study built on previous versions used in Malindi district in 2007–2008 (Crouch, Korda, & Mumo, 2009) and Central and Nyanza provinces in 2009 (Piper, Schroeder, & Trudell, 2015). The tools were created at an adaptation workshop in September 2011 with representatives from the MOEST, Kenyan universities, the Kenya National Examinations Council and various other education stakeholders. These tools were piloted in October 2011 and Rasch models were fit to determine which items were not performing as anticipated. The overall Cronbach’s alphas for the final English and Kiswahili instruments were .90 and .91, respectively. Due to the size of the tools, an extracted sample passage can be found in Appendix A. The full instrument is presented and discussed in detail in the PRIMR baseline findings report (Piper & Mugenda, 2012). The difficulty levels followed the expectations of the Kenya Institute of Curriculum Development for Class (grade) 1 pupils in literacy. The vocabulary was simple, the sentences were short, and words that would not be understood by pupils in all parts of Kenya were discarded. The validation process included vetting the stories with a team of experts as well as classroom teachers, and for English, undertaking Flesch-Kincaid assessments of text difficulty to ensure that the texts were appropriate for Class 1 learners and of similar difficulty levels between the passages. The English time-delimited and passage-delimited stories were at 0.9 and 1.0 grade levels, respectively, and the two passages had reading ease of 98% or higher. For Kiswahili, text complexity was analysed by professors of Kiswahili and KNEC assessment experts, as standardized text difficulty options were unavailable.
The tools discussed in this paper are the time-delimited oral reading fluency passages of approximately 60 words for both English and Kiswahili, with five reading comprehension questions; and the passage-delimited oral reading fluency stories of approximately 40 words for both English and Kiswahili, with four reading comprehension questions. Assessors administered the time-delimited story to the pupils before the passage-delimited story. The two languages were assessed in random order alongside of mathematics assessments, meaning that a child could undertake the English or Kiswahili assessment first, second or third. When we analysed whether there was any effect of the order of assessment, looking at 36 comparisons for oral reading fluency and reading comprehension, we found three comparisons with statistically significant differences in outcomes. 2 Given that only three of the comparisons were statistically significant, and that none of the differences occurred during the third assessment, when children would more likely be fatigued, these findings do not support the notion that the order of the time-delimited story and the passage-delimited story had a very large or systematic relationship with learning outcomes.
Assessment procedures
As noted, the oral reading fluency assessments were conducted as part of a large-scale Early Grade Reading Assessment that served as the baseline assessment for the PRIMR initiative. The EGRA was a battery of early literacy tasks given to Class 1 and 2 pupils in the sampled schools. Assessors were trained by PRIMR monitoring and evaluation experts and RTI International literacy experts with experience in training literacy assessors in the region. The 68 assessors – many of whom had undertaken literacy assessments in Kenya in previous studies – were trained for six days and were provided with a fieldwork manual and assessment tools. Interrater-reliability scores for the entire EGRA, including the oral reading fluency subtask, were 95.3% in Kiswahili and 96.1% in English. Simple random sampling at the school level was undertaken to ensure class and gender balance. Operationally, that meant that the assessors lined up all of the pupils in the school by grade and gender, and, using a sampling interval derived by the number of pupils in the grade, selected five male and five female pupils per grade, for a total of 20 pupils per school. A team of MOEST and PRIMR staff members supervised the assessors. The order of the Kiswahili, English, and mathematics sections was randomized to avoid systematic effects of test fatigue. Prior to undertaking both types of literacy assessments, the assessors prompted pupils to read the short passage aloud, quickly but carefully. They also told the pupils that they would be asked about what they had read when they were finished, and that they should read the passage ‘as best as they can’. Assessors were trained not to react to correct or incorrect responses, and gave prompts to the students to encourage them and keep them moving forward rather than stopping to focus on incorrect or slow responses.
Sample
For the PRIMR baseline study, the 4385 pupils were assessed from 220 schools, 95 of which were formal schools and 125 of which were non-formal schools in slum areas. In all, 2186 girls and 2199 boys were sampled, as were 2192 Class 1 pupils and 2193 Class 2 pupils (see Table 1 for a description of the sample). The table indicates that the average pupil age at the assessment was 7.2 years old, and that nearly 50% of the sample was female. It also shows that on an index of 10 household items that many Kenyans have at home (including a toilet, a mobile phone, and a vehicle), the average pupil reported having five. Fifty-two per cent of the sample reported having books other than the textbook at home, and 43% and 44% of the sampled pupils had access to an English and Kiswahili textbook, respectively. It is interesting to note that 91% and 92% of the sampled pupils reported that their mother and father (or guardian), respectively, could read or write. As mentioned earlier, the results on this survey suggest that the sample had higher socioeconomic status than average in Kenya (Piper, Jepkemei, & Kibukho, 2015). Given the relatively low achievement of Kenyan students on literacy outcomes, the inclusion of a population of slightly higher socioeconomic status likely resulted in fewer floor effects and greater variation in student scores than would have been the case otherwise.
Demographic description of the sample at baseline, by school type (standard errors in parentheses).
Data-analytic plan
Owing to the sampling procedure, all calculations utilized the svy suite of commands in Stata. This fixed the weights to each of the estimates and the standard errors, which in turn allowed the findings to be externally valid to the population from which the sample was derived – namely, the eligible peri-urban and rural schools in Nairobi, Kiambu, Murang’a and Nakuru counties and the universe of non-formal settlements that RTI had identified in 2011. Because the same pupils took both the time-delimited and passage-delimited assessments, we focused on mean differences between the two assessment types for the entire sample and undertook subgroup analysis that differentiated the sample by their reading rate, as explained below.
Findings
Consistent with the literature from the United States presented above, we found strong correlations between oral reading fluency and reading comprehension on both time-delimited and passage-delimited tasks (see Table 2). For English, the correlations were .77 and .69, respectively. For the Kiswahili tasks, the correlations were .88 and .80. Students’ fluency scores on the two assessment approaches were strongly correlated – .90 for English and .89 for Kiswahili. We found lower, but still strong, correlations between pupils’ passage-delimited and time-delimited reading comprehension scores – .75 for English and .73 for Kiswahili.
Pearson correlations for time-delimited and untimed passage-delimited oral reading fluency and reading comprehension.
p < .001.
Our first research question focused on whether fluency rates and reading comprehension scores differed according to the amount of testing time available. Table 3 presents the mean fluency rates and the associated standard errors (with corrections for nested sampling inherent in student assessments at the school level). The data show that students did not read fluently in either English or Kiswahili, with mean fluency rates ranging from 9.8 words correct per minute (passage-delimited Kiswahili) to 14.3 words correct per minute (passage-delimited English). Results were similarly low in reading comprehension, with overall mean comprehension scores ranging from 9.8% correct (time-delimited English) to 15.1% correct (passage-delimited Kiswahili). These low percentage rates indicated that on average, these students were not answering even one comprehension item correctly. Results were somewhat better on the percentage of attempted reading comprehension questions answered correctly; this portion of the test took into account that most pupils did not attempt all of the reading comprehension questions since, as indicated above, they did not read enough of the passage for the latter questions to be asked of them.
Descriptive statistics for oral reading fluency and reading comprehension, by language and time-delimited and untimed passage-delimited reading assessments.
Table 3 also presents the results of post-hoc linear hypothesis tests fit to determine whether there were any differences in fluency rates as measured by the time-delimited and passage-delimited tasks. The tests showed that students read more fluently in English when untimed: The mean fluency rates for English time-delimited and passage-delimited assessments differed by 0.4 words per minute (p < .01). We did not find any statistically significant differences between fluency rates on time-delimited and passage-delimited stories in Kiswahili (p = .238). The table also presents the reading comprehension rates associated with the fluency stories. Pupils performed nearly two percentage points better on the reading comprehension questions accompanying the English passage when the exam was untimed (p < .001). There was no statistically significant difference in the comprehension rates in Kiswahili across the two methods (p = .196).
In summary, we found that the English fluency and overall reading comprehension scores were slightly higher on the passage-delimited story. Substantively, however, the differences were rather small – 0.4 WCPM in comparison to the standard deviation of the fluency scores, between 22.0 and 23.9 WCPM. Similarly, the 1.9 percentage point difference in overall reading comprehension scores in English was small compared to the standard deviations (23.6 and 21.9 percentage points). Given that each comprehension item was worth 20 percentage points, a 1.9 percentage point difference is just less than 10% of one item. Effect sizes were between .02 and .09 SD for the differences between time-delimited and passage-delimited in English, and the differences were between .01 and .02 SD in Kiswahili – although as we discuss below, there were large differences between timed and untimed attempted reading comprehension scores in Kiswahili.
As discussed above, one of the reasons to provide students with a longer time to read was if their reading speed was expected to vary over the period of the assessment. If this were the case, then a very brief testing period might not capture those changes. In order to ascertain whether pupils’ oral reading fluency rates changed during the assessment, we created a variable that differentiated pupils by how long it took them to finish the passage-delimited oral reading story. Just over 68% of pupils finished the passage-delimited English story in the first minute, while 18% finished in the second minute and the remaining 14% finished in the third minute. For parsimony we grouped those pupils who finished in the second minute and the third minute together. We created a similar variable for the Kiswahili passage; 58% finished in the first minute, 22% in the second minute, and 21% in the third minute. These variables allowed us to distinguish those who read the passage in 1 minute or less from those who read the passage more slowly. We then were able to compare their overall fluency rates on the time-delimited and passage-delimited stories. The results are presented in Figures 1a and 1b.
English oral reading fluency rates for time-delimited and untimed passage-delimited stories for pupils who read the untimed passage-delimited story fast (< 1 minute), and slow (> 1 minute).
Kiswahili oral reading fluency rates for time-delimited and untimed passage-delimited stories for pupils who read the untimed passage-delimited story fast (< 1 minute), and slow (> 1 minute).
In this analysis we were most interested in comparing whether fluency rates differed dramatically within particular groups of pupils, depending on their rates on the untimed passage-delimited story. We posited that this would give us insight into whether the length of assessment time was related to the outcomes for pupils, and whether the rates at which pupils read changed during the duration of the assessment. We found that, for English (Figure 1a), the mean WCPM scores were 1.3 WCPM (.06 SD) higher for fast readers when the assessment was passage-delimited (p < .001). Conversely, slow readers performed more poorly in terms of WCPM when the assessment was passage-delimited – the mean WCPM rate was 1.4 (.07 SD) lower (p < .001). Slow readers actually read less fluently when given more time than when given less time, which is initially counterintuitive. We discuss this finding and present our hypothesized explanation below. For Kiswahili (Figure 1b), the differences were 0.6 WCPM more (.04 SD) for the fast readers (p < .05) and 1.4 WCPM less (.09 SD) for the slow readers (p < .001). 3
This rich and detailed data set also allowed us to compare the reading comprehension scores for pupils, differentiated by their reading rate on the passage-delimited story (comparing fast and slow readers as above). The findings were somewhat less consistent for fast and slow readers for reading comprehension than for fluency. Figures 2a and 2b present the reading comprehension rates (percentage correct). We were particularly interested in whether the rates would be similar for the time-delimited and passage-delimited measures. For English (Figure 2a), we found that comprehension rates were slightly higher following the passage-delimited story for the fast readers (0.9 percentage points, .04 SD, p < .001) and the slow readers (4.1 percentage points, .20 SD, p < .001). For Kiswahili (Figure 2b), the comprehension rates were slightly higher for the time-delimited story for fast readers (2.2 percentage points, .10 SD, p < .001) and higher for the passage-delimited story slow readers (4.3 percentage points, .19 SD, p < .001).
English reading comprehension rates for time-delimited and untimed passage-delimited stories for pupils who read the untimed passage-delimited story fast (< 1 minute), and slow (> 1 minute).
Kiswahili reading comprehension rates for time-delimited and untimed passage-delimited stories for pupils who read the untimed passage-delimited story fast (< 1 minute), and slow (> 1 minute).
The EGRA administration rules allowed assessors to make specific decisions on whether or not pupils should be asked particular comprehension questions, based on how much of the passage they completed. Therefore, regardless of their reading speed or fluency rates, we were able to compare the comprehension rates for pupils on the portion of the passage they read (comprehension on attempted) and the entire passage (comprehension on total). From this definition, then, for the entire sample, attempted comprehension scores would be equal to or higher than total comprehension scores. For the purposes of this analysis of the validity of both the time-delimited and passage-delimited measures, we were interested in examining whether the scores on the timed total and attempted comprehension subtasks would be similar, as for almost all of the pupils in the sample, for the passage-delimited story, there were no differences between total and attempted comprehension rates. In Figure 3, the two black bars in each language pair were compared to determine whether there were differences between the rates. Passage-delimited total comprehension was 1.9 percentage points higher (.09 SD) than time-delimited total comprehension in English (p < .001) and there was no statistically significant difference in Kiswahili (p = .25). The differences in attempted comprehension scores were larger – 2.6 percentage points higher (.09 SD) on the time-delimited assessment in English (p < .001) and 13.9 percentage points higher (.33 SD) in Kiswahili (p < .001). This difference likely is because the final questions on the comprehension measures were inferential and therefore more difficult than the other questions.
Reading comprehension scores by language for time-delimited and untimed passage-delimited stories, comparing scores for total comprehension and comprehension on questions attempted.
Discussion
In this section we discuss the findings related to each of the research questions. First, we investigated whether timing the subtasks in individual assessments resulted in differences in fluency and comprehension scores in the two languages of interest. The similarities between student outcomes on the time-delimited and passage-delimited assessments were substantial for oral reading fluency as well as for reading comprehension. We found that in English, pupils read 1.4 WCPM more of the passage-delimited story than of the time-delimited story, and their reading comprehension scores were 1.9 percentage points higher on the time-delimited than the passage-delimited story. There were no differences for Kiswahili on either comparison. The English differences we identified were less than .1 SD. Given the small differences in English and the similarities in Kiswahili, the length of time allotted to pupils did not lead to meaningfully different outcomes.
With respect to our second research question – related to whether the reading assessment was time-delimited or passage-delimited would result in differences between the scores of fast and slow readers – the analyses associated with Figures 1a and 1b and Figures 2a and 2b showed minimal differences whether pupils scored as fast or slow readers, for both English and Kiswahili. The largest differences existed among the slow readers, for whom the passage-delimited fluency scores were 1.4 WCPM (.07 SD) lower for English and 1.4 WCPM (.06 SD) lower for Kiswahili. This suggests some slowing down as these pupils struggled through the story, although the differences are very small. For them, therefore, fluency rates on a time-delimited story were a somewhat more accurate assessment of their ability to decode. For comprehension, however, slow readers performed better on the passage-delimited story than on the time-delimited story. This suggests, interestingly, that although struggling readers had more difficulty reading as time progressed, there was a difference in their comprehension outcomes with the increased time to better understand the passage. This comparison comes with some caveats, however: The comprehension scores cannot be equated using linear equating methods because they emanate from fundamentally different tasks; and the improvements in comprehension outcomes for slow readers resulting from the additional time were 4.1 percentage (.14 SD) points for English and 4.3 percentage points (.10 SD) for Kiswahili. The effect of additional time on comprehension scores therefore was statistically significant and should be considered when evaluators are deciding between timed and untimed assessments.
The richness of the EGRA data set and the ability to compare comprehension scores on the total number of questions and the number of attempted questions are important contributions to the literature, as this work may inform the development and refinement of future assessments for use in international contexts. For example, as discussed above, Figure 3 shows a difference of only 0.3 percentage points in comprehension scores between total and attempted for English and a 1.2 percentage-point difference for Kiswahili. This consistency is logical given that, for all but a few pupils, the 3-minute time period was sufficient for the pupils to finish their attempt to read the passage-delimited story. The findings show that the differences between time-delimited and passage-delimited assessments were much smaller on the total comprehension questions than on the attempted comprehension questions.
We offer the following suggestions to explain the difference in attempted and total comprehension scores for Kiswahili. First, the inferential questions that were placed at the end of the list of comprehension items likely were more difficult than the recall questions. Second, pupils who were scored on the attempted comprehension results were not tested on their ability to answer those more difficult inferential items. Third, the findings also show that the largest differences between total comprehension scores (which scored comprehension items that were not attempted as incorrect) and attempted comprehension scores (which did not score comprehension items that were not attempted) were found in Kiswahili. Given that these students were more likely to speak Kiswahili at home than English, this suggests that valid measurement in Kiswahili depends more on the relative difficulty of the set of items asked than in English, given how poorly children perform on English comprehension.
Therefore, given that the attempted comprehension scores assume a linear progression of scores, ignoring the relative difficulty of the items that were not attempted, this finding suggests that using the attempted comprehension rates alone would inflate pupils’ real comprehension scores. Therefore, this also implies that the total comprehension rates are more appropriate for evaluation purposes, even for time-delimited assessments. In the case of Kiswahili, using the attempted scores for a 1-minute passage task would have inflated the comprehension rates by 13.9 percentage points or .33 SD, which is a large difference.
Implications
Comparing timed and untimed assessment results is not a new area of research. However, our findings are unique in sub-Saharan Africa and particularly in Kenya, and are important given the recent burgeoning interest in assessing literacy outcomes in sub-Saharan Africa. It is essential for research to evaluate whether the assumptions utilized in Western contexts to design literacy assessments are applicable in Kenya. In addition to filling in that important gap in the literature, our findings have three main implications for programme managers and evaluators working on literacy programmes in sub-Saharan Africa. First and foremost, our findings indicate that students did not appear to perform at a meaningfully higher level in either oral reading fluency or reading comprehension when an assessment was untimed. There was no apparent bias for or against fast or slow readers in terms of time-delimited versus passage-delimited performance, in English or Kiswahili, although comprehension rates differed, suggesting some benefits to untimed assessments. Timed tests are advantageous with respect to the required assessment time for each pupil, and as a result, the cost of the assessment. Therefore, we recommend that researchers consider the potential increase in precision on reading comprehension subtasks using untimed assessments against the logistical considerations that favour timed assessments. Second, we recommend that evaluators consider using reading comprehension outcomes from all possible items rather than simply reporting the percentage correct on attempted items. Doing so appears to inflate reading comprehension scores and may render them less useful for evaluation purposes. Third, the low scores on both fluency and comprehension identified in this assessment mean that the vast majority of pupils in this sample struggled to decode and comprehend at meaningful levels in either English or Kiswahili. This suggests a need for immediate interventions in the area of literacy to ensure that outcomes improve in the short term. It might be the case that when student outcomes increase as a result of literacy improvement programmes, larger differences between timed and untimed assessments could emerge, and the analyses in this paper could be revisited.
Limitations
This section presents a set of limitations to our research that are important to take into account in assessing the validity of the conclusions. First, for both English and Kiswahili, the passage order was not randomized; that is, the time-delimited story always came first and the untimed passage-delimited story second. Therefore, if practice effects indeed occurred, as Petscher and Kim (2011) found on multiple-passage assessments in the United States, then student scores on the untimed assessment may be inflated. That said, a previous study in Kenya using EGRA gave pupils three passages for three different languages. That analysis, of 4000 pupils, showed that the order of assessment had no effect on oral reading fluency or comprehension (Piper, 2010). Analyses from the data set examined for this paper suggest that pupils did not do significantly better or worse on either the time-delimited or passage-delimited assessment when the language assessed (English or Kiswahili) was assessed first, second or third (after the mathematics assessment), although children did somewhat worse when the Kiswahili assessment was given second (but not third). This suggests, therefore, that learning and fatigue effects had a modest influence on the results.
Second, unfortunately, our analysis did not correct for any difficulty differences that might have existed across the (timed or untimed) passage or comprehension questions, as the fundamental differences in the passages made it impossible to equate difficulty scores as the PRIMR Initiative has done with other fluency and comprehension passages of similar lengths and administration timings (Albano & Rodriguez, 2012; Piper & Mugenda, 2013). The science of equating timed assessments is nascent and the literature does not discuss methods of equating passages of different lengths and sets of comprehension questions containing varying numbers of items.
Third, the overall low achievement levels of the students in this study suggest that the small differences identified in the paper could become larger differences if there were more variation in student outcomes. As Kenya implements literacy improvement programmes, this analysis should be revisited to determine whether larger differences between outcomes on timed and untimed assessments would be identified when overall achievement increased.
Footnotes
Appendix A: Sample English passage and comprehension questions from the PRIMR EGRA instruments
Acknowledgements
We are grateful to the Kenyan Ministry of Education, Science and Technology, and the committed teachers and educators in Kenya. We appreciate the USAID/Kenya and DFID/Kenya teams for their financial and technical support for PRIMR. We are thankful for the excellent editorial work of Erin Newton.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The lead author served as the Chief of Party (project director) of the PRIMR Initiative.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/ or publication of this article: This work was supported by the United States Agency for International Development through the Kenya PRIMR Initiative, carried out under the USAID Education Data for Decision Making (EdData II) project [Task Order Contract No. AID-623-M-11-00001] and by a Professional Development Award from RTI International [award number 0190453.300.001].