A Multisite Randomized Controlled Trial of the Effectiveness of Descubriendo la Lectura

Individualized Instruction

DLL reading instruction also is based on the principles of Marie Clay’s research on literacy processing. For example, the program’s distinctive individualized instructional component spawns from Clay’s assertion that children’s processing systems are ever-changing, and thus, careful teacher observation is essential to directing effective instructional decisions. Furthermore, understanding children’s strengths allows teachers to maximize students’ reading skill set. Clay suggested that literacy learning includes more than isolated instruction on letters, sounds, and words; rather, quality learning involves reading continuous text (Reading Recovery Council of North America, 2014a). This principle is reflected in DLL lessons that move beyond such isolated instruction into fluent reading practice. Clay’s (2001) assertion that children must actively construct their own understandings aligns with the program’s emphasis on independent learning.

Self-Directed Learning

DLL focuses on sustainable literacy success by offering opportunities for self-directed learning through writing, motivation, and independence strategies (May et al., 2013). As opposed to passive activities, DLL facilitates “active construction” learning based on the notion that carrying out activities, such as reading books and writing messages, bolsters self-monitoring and problem-solving abilities (DeFord, 2007). Teachers foster these skills by allowing students to work independently while offering guidance as needed.

High-Quality Instruction

The DLL model is also based on the assumption that high-quality instruction is central to improving the literacy of low-achieving students (May et al., 2013). The program model supports quality instruction through stringent teacher selection standards, extensive training, and job-embedded ongoing professional development. University-trained teacher leaders provide a year of intensive training in Spanish or a year of English training and a second year of “bridging” to Spanish. Following this training, teacher leaders provide close supervision during the teacher’s field year and, in future years, multiple professional development sessions and continuous technical assistance including one-on-one coaching. These high standards and continuous, supportive mentoring opportunities combine to produce teacher expertise, and ultimately reflect the underlying principle that instructional quality is paramount (May et al., 2013).

Systemic Support for Reform

Moreover, the program is guided by the assumption that interventions and reforms must be supported by school systems and, therefore, implementation oversight is required (May et al., 2013). As Bryk, Gomez, and Grunow (2010, p. 130) contended, “The history of educational innovation is replete with stories that show how innovations work in the hands of a few, but lose effectiveness in the hands of the many.” DLL strives for a sustainable and scalable system that values implementation fidelity. It accomplishes this fidelity of implementation through continuous, district-level monitoring, national standards and guidelines, and feedback provided by teacher leaders to key DLL staff (e.g., administrators, teachers; May et al., 2013).

Early Intervention

While much research supports DLL’s specific approach to reading instruction, it is also clear that intervening early is essential to future academic achievement. Many studies have shown that students who fall behind in reading in first grade fail to catch up to their peers (Juel, 1988; Lyon & Fletcher, 2001; Shaywitz et al., 1999). An exploration of research in economics, developmental psychology, and neurobiology suggests that these early deficits have lasting consequences, as early learning has a particularly strong influence on cognitive skills and brain architecture (Knudsen, Heckman, Cameron, & Shonkoff, 2006). The authors assert that skill development is a “hierarchical process in which higher level functions depend on, and build on, lower level functions, and that the capacity for change in the foundations of human skill development and neural circuitry is highest early in life and decreases over time” (Knudsen et al., 2006, p. 10155). Thus, early learning programs, like DLL, represent the highest potential return on investment.

Native Language Learning

Finally, DLL is predicated on the notion that learning in one’s native language can accelerate literacy development in both the short term and, even more important, in the long term. From a practical standpoint, DLL provides a coordinated approach to supplement and support the development of quality literacy programs, in Spanish, for Spanish-speaking children. While there is evidence that ELs in programs offering varying amounts of English instruction perform equally well initially, there is substantial evidence to suggest that literacy programs offering more initial learning in the students’ native language lead to larger academic gains in the long term (Ashdown & Simic, 2000; Richards-Tutor, Baker, Gersten, Baker, & Smith, 2016). In fact, a meta-analysis on EL literacy development from the National Research Council concludes that literacy programs taught in a student’s second language heighten the risk of later reading difficulties, which “may compound the risks associated with poverty, low levels of parental education, poor schooling, and other such factors” (National Research Council, 1998, p. 234). DLL is specifically designed to support high-quality programs that emphasize early instruction in students’ first language of Spanish.

Sample

Two school districts, one in the Southwest and one in the Rocky Mountain region of the United States, were included in this study. These two districts had schools participating in the i3 grant program with trained or in-training DLL teachers participating in a scale-up study of Reading Recovery. From these two districts, 15 schools, 27 teachers, and 187 (89 control and 98 treatment) first-grade students participated in the study. All sampled schools offered transitional bilingual programs, which provided Spanish language regular-classroom reading and language arts instruction to their first-grade ELs. DLL teachers provided supplemental one-on-one Spanish-language literacy tutoring to four students each semester and each child progressed through the program at an individualized pace, generally lasting 12 to 20 weeks. All students were native Spanish speakers, and all but one of the participating students were free-lunch program participants.

The teachers participating in this study were previously trained DLL teachers or educators who had qualified to participate in training under the strict requirements of the i3 Grant, “Reading Recovery: Scaling Up What Works.” Newly selected teachers were undergoing training (graduate level coursework during the fall and spring semesters) while concurrently teaching students. Overall, 92% of the participating teachers were female, 81% were of Hispanic ethnicity, 85% spoke languages other than English at their homes, 56% had beyond a bachelor’s degree, and on average, the teachers had 16 years of teaching experience.

Study Design

For this study, we employed an RCT design in which the lowest performing first-grade native Spanish-speaking students were randomly assigned to immediate treatment or a delayed, poststudy treatment condition. Our study focuses on the fall semester outcomes for the students and contrasts the pre-post outcomes for the immediately treated students to those of the students whose treatment was delayed until after the study was completed. Following typical DLL practices, during the fall of 2011, DLL teachers administered a Spanish literacy assessment, IdO (described below), to all at-risk Spanish-speaking students who were identified by their classroom teachers as performing below grade level in literacy in their school. Based on the results of the IdO, the lowest performing students at each campus were identified and randomized to an immediate or a poststudy, delayed DLL treatment condition.

Six campuses had one DLL teacher available, six had two DLL teachers, and three had three DLL teachers available. If the school had one DLL teacher, then eight students were randomized. If the school had two or three DLL teachers, then a two-stage randomization process first randomized students to teachers and then randomized students to immediate or delayed treatment. Those students randomized to the immediately treated condition received DLL services at the beginning of the fall of the 2011–2012 academic year, and those in the delayed condition received services 12–20 weeks later—as the immediately served students exited the program.

The researchers achieved this randomized design by generating a random number for each eligible child (i.e., the lowest performing EL students) at the beginning of the school year. Teachers served those with lower randomly assigned numbers with first priority, and those students with higher numbers remained on a waiting list and were served later, after the conclusion of the study period. As each of the immediately treated students completed DLL, the teachers served the next child on the waiting list with the lowest randomly assigned number. As students assigned to immediate service during the fall completed DLL, they were tested using two instruments, the IdO and Logramos, which we describe below. At the same time, prior to entry into DLL, the next child on the randomly determined waiting list also was administered the two instruments. Because the priority of service was randomly determined, a comparison of the exiting and entering students’ test scores provides an unbiased estimate of the differential achievement impacts between, respectively, immediate assignment to DLL services and assignment to “business as usual” literacy instruction in the absence of DLL services. The time line for program delivery and testing for the immediate- and delayed-treatment students is summarized in Figure 1.

OPEN IN VIEWER

Figure 1.

Time line of Descubriendo la Lectura (DLL) program delivery and administration of tests.

Measures

The two student assessments that were used were the IdO (Escamilla, Andrade, Basurto, & Ruiz, 1996) and the Logramos (2006) reading achievement test. The IdO is the Spanish-language version of the Observation Survey of Early Literacy Achievement (OS; Clay, 1993). The OS and the IdO measure student performance on the following six subtests: Letter Identification, Ohio Word Test, Concepts about Print, Writing Vocabulary, Dictation, and Text Reading (Clay, 2005). Both the OS, for English-language students, and IdO, for Spanish-language students, are administered as a regular component of the, respectively, Reading Recovery and DLL programs. The OS and the IdO tests are administered at the beginning of the first-grade school year as a screener to determine eligibility for, respectively, Reading Recovery and DLL services and after students complete services in order to measure academic progress. The IdO also has been used to examine student outcomes in prior research studies, including Escamilla et al. (1996).

IdO is not a literal translation of the English OS but rather a conceptual recreation, which measures literacy skill acquisition for young native Spanish speakers. Both the OS and IdO are tools designed for the systematic observation of young children’s early literacy competencies. The National Center on Response to Intervention reviewed OS, giving it its highest rating of “convincing evidence” in each category (see http://www.rti4success.org/observation-survey-early-literacy-achievement-reading). Construct validity for the IdO was established, using data from approximately 500 first-grade students, yielding Cronbach’s alphas ranging from α = .51 to α = .82 (Escamilla et al., 1996). Concurrent validity was established by comparing the six subtests of IdO with a norm-referenced test, Aprenda: La Prueba de Logros en Español (Pearson, 3rd ed., see http://www.pearsonassessments.com/learningassessments/products/100000585/aprenda-3-aprenda-la-prueba-de-logros-en-espanol-tercera-edicion.html). Content validity was established through a series of translations and back translations (Escamilla et al., 1996). Furthermore, as described by Escamilla et al. (1996), IdO was assessed to determine whether it performed equally as well with individuals from a variety of Spanish-speaking backgrounds. This study included 60 first-grade students from three different cultural backgrounds: Mexican American, Puerto Rican, and Cuban American. The results suggested no differential understanding of the survey questions across the three groups.

The Logramos, second edition, is the Spanish-language version of the Iowa Test of Basic Skills achievement test battery (Logramos, 2006). The items in Logramos follow the scope and sequence of Form E of the English-language Iowa Assessments and have been translated, when appropriate, from the original English items in Form E of the Iowa Assessments. Though the vast majority of items were direct translations of the English test into Spanish, some items required adaptation and replacement of English items in the Spanish version in order to target the same skills and maintain the underlying psychometrics of the test items.

For this study, the following sections of Logramos Level 7 were administered: Vocabulary (Vocabulario), Word Analysis (Análisis de Palabras), and Reading (Lectura). The Logramos is a standardized, norm-referenced test that was administered as an additional assessment for the purposes of the study and is not typically given to DLL students under routine, nonstudy conditions. Both the individually administered IdO and group-administered Logramos posttests were proctored by a DLL teacher leader or by another teacher who did not provide DLL or classroom instruction to the tested students.

During the fall of 2011, the IdO was administered to first-grade Spanish-speaking ELs performing below grade level in literacy who were identified by their teachers as most in need of supplemental literacy support. Based on the empirical results of the IdO, the lowest scoring students at each campus were identified and randomized to the immediate or delayed DLL treatment conditions. On completion of the DLL program (after 12–20 weeks), each immediate-treatment student was given the IdO test again along with the Logramos battery. At the same time, the next delayed-treatment child on the waiting list with the lowest randomly assigned number was also given the IdO and the Logramos prior to beginning their series of DLL lessons. Thus, all students—both immediate and delayed treatment—were administered a fall IdO baseline test during August or September, prior to being assigned to DLL, and all students were again tested, using the IdO and Logramos instruments, after 12 to 20 weeks during November 2011 through February of 2012. For the purposes of the study, the IdO fall test serves as our preintervention baseline pretest, and the later IdO and Logramos assessments served as our posttests. We use the fall pretest IdO score as a covariate in analyses of both the IdO and Logramos outcomes.

Beyond the test scores, we collected demographic data for each teacher and student through the International Data Evaluation Center (2012) at the Ohio State University, which gathers data on all Reading Recovery and DLL sites and students each year. The teacher characteristics are summarized in the previous Sample section, and the student background information is presented in Table 1, which appears in the Results section.

OPEN IN VIEWER

Table 1

Summary of Background and Pretest Data for DLL and Control Students

Variables		IdO Outcome			Logramos Outcome
		Control (N = 82), %	Treatment (N = 94), %		Control (N = 59), %	Treatment (N = 51), %
Student characteristics	Male	60	66		61	69
	Latino	100	100		100	100
	Multiple race/ethnicities	22	18		9	10
	Other languages spoken within home	100	100		100	100
	Free lunch	100	99		100	100
	Special education	10	2		9	3
	ESL	80	78		93	85
		Control (N = 82), M (SD)	Treatment (N = 94), M (SD)	d	Control (N = 59), M (SD)	Treatment (N = 51), M (SD)	d
Pretests	Letter-Identification	−0.06 (1.01)	0.03 (1.00)	0.09	0.01 (0.98)	−0.03 (1.03)	−0.04
	Ohio Word Test	−0.14 (0.96)	0.12 (1.04)	0.26	−0.14 (0.91)	0.08 (1.07)	0.22
	Concepts about Print	−0.05 (1.06)	0.04 (0.96)	0.09	−0.09 (1.10)	0.06 (0.93)	0.15
	Writing Vocabulary	−0.11 (0.93)	0.10 (1.07)	0.21	0.01 (0.95)	−0.02 (1.05)	−0.03
	Hearing/Recording Sounds in Words	−0.12 (1.01)	0.09 (0.99)	0.21	−0.08 (0.98)	0.02 (1.02)	0.10

Note. DLL = Descubriendo la Lectura; IdO = Instrumento de Observación; ESL = English as a second language.

Program Implementation

For this study, the program was implemented under routine conditions that would support any typical DLL implementation. The services provided one-on-one tutoring in Spanish for first-grade Spanish-speaking students receiving regular classroom literacy instruction in Spanish and experiencing substantial challenges with reading and writing. The program offered daily 30-minute instruction for 12 to 20 weeks for students performing in the bottom 20% to 25% of their class, as specified by DLL protocol (Reading Recovery Council of North America, 2014f). Trained literacy teachers provided intensive, individualized support as a complement to students’ regular-classroom instruction, with the goal of raising their literacy skills to the level of their peers (Reading Recovery Council of North America, 2014e). DLL teachers worked with the classroom teachers to avoid providing this supplemental literacy instruction during the regular classroom literacy block. Typically, DLL services were offered at times that did not interrupt students’ core academic instruction, during a time when recess or physical education was scheduled, for instance.

As the delayed-treatment students awaited DLL services, they experienced “business as usual” literacy instruction, which consisted of Spanish language regular classroom literacy instruction and, in many cases, small-group, supplemental reading instruction or, in a smaller number of cases, special education services. The DLL teachers monitored the academic progress of all of their assigned delayed- and immediate-treatment students throughout the school year, often providing supplemental small-group instruction to their delayed-treatment students, but only immediate-treatment students received the one-on-one DLL programming during the first 12–20 weeks of the school year.

There are several defining features of the DLL program, which we describe below, which were implemented as part of this study and are implemented under any circumstance in which schools or districts adopt the DLL model. Specifically, these program components include the following: daily lessons tailored to students’ specific literacy needs, a rigorous program of teacher training, ongoing professional development and support for participating DLL teachers, and a collaborative infrastructure of teacher leaders, district-site coordinators, and University Training Centers to oversee high-quality implementation of the program in the study sites. Though this was strictly an impact evaluation, and we conducted no formal implementation study, anecdotal information collected from the DLL teacher leaders and University trainers suggested that fidelity to these program components was adequate to high. Given the structured and “manualized” nature of the DLL program, this general observation concerning the quality of implementation is not surprising.

Daily Lessons

While teachers had flexibility to adapt lessons based on individual student needs, the program required that several activities occur during each student’s daily 30-minute lesson. These activities included rereading familiar books, reading a recently assigned book while teachers took a running record, working with letters or words using magnetic letters, writing a story, assembling the child’s cut-up story, and reading a new, strategically selected book (Reading Recovery Council of North America, 2014e). Lessons provided teaching on letter and sound relationships, spelling patterns, comprehension, producing oral language, and increasing vocabularies (Clay, 2001; Reading Recovery Council of North America, 2014e). Rather than a prescribed curriculum, however, teachers had the flexibility to adjust lessons based on each child’s strengths and needs, offering support and strategic guidance as needed. Furthermore, teachers provided extensive opportunities for student problem solving and self-correction with limited interruption (Clay, 2001).

Individualized Instruction

Individually tailored instruction is one of the defining features of the program. The training provided to teachers who participated in the study offered moment-to-moment responses to student behaviors and to diagnose and respond to individual student needs (Reading Recovery Council of North America, 2014b). As is the case with typical implementations of DLL, the teaching decisions were dependent on careful observation and an understanding of student progress, and thus, program implementation relied on and demanded extensive record keeping. Teachers maintained records of each student—using data to inform their practice—by employing a tool that captures literacy progress before entering the program, on leaving the program, and on completion of the school year (the IdO; Reading Recovery Council of North America, 2014a). Teachers also were required to use a variety of tools to routinely assess student progress, including a planning report, daily monitoring tools, and daily assessment forms. The running records form is an example of a crucial assessment tool that allows for responsive instruction. Using this form, DLL teachers calculated error ratios, accuracy rates and self-correction ratios, and determined which sources of information (meaning, syntax, visual) children used when they made errors or self-corrections. They also made notes about fluency, observed strategic processing, and high frequency words. DLL teachers used the running record analysis to determine the focus of the next day’s lesson.

Teacher Training

In order to provide this high-quality, individualized instruction, the program is designed around rigorous teacher training and standards. To be eligible to become a DLL instructor, teachers must hold state-required certification, have at least 3 years of overall teaching experience, and have demonstrated, through teacher evaluation, successful bilingual teaching (Reading Recovery Council of North America, 2012, p. 9). On entry to the program, teachers are trained by teacher leaders in a yearlong master’s level Reading Recovery class and then “bridge” to DLL in the second year, during which they must teach lessons and implement all aspects of DLL. Alternatively, if there are enough teachers training in DLL at once, a teacher leader may choose to train teachers directly in DLL. The present study evaluates the program as run by already trained teachers, who must teach at least four students per day, keep complete records on each student as a basis for instruction, use the Spanish version (IdO) of the OS, communicate with parents and teachers, submit data to the DLL teacher leader, and monitor the progress of students who have exited the program (Reading Recovery Council of North America, 2012).

Ongoing Support

Ongoing professional development and support is another cornerstone of the program—a requirement that ensures the infrastructure and expertise needed to provide high-quality instruction (Reading Recovery Council of North America, 2012). Teachers must regularly consult with teacher leaders about those students who are not making satisfactory progress, participate in a minimum of four professional development sessions each year, receive at least one school visit from a teacher leader annually, and make and receive school visits from colleagues annually (Reading Recovery Council of North America, 2012). This annual professional development regimen is required for as long as a teacher is implementing DLL. The teacher experience is, therefore, one of continuous collaboration among colleagues, classroom teachers, administrators, and mentors.

Collaborative Infrastructure

In addition to their training role, teacher leaders provide program implementation oversight in schools by monitoring the progress of teachers and students. Teacher leaders accomplish this oversight by regularly collecting program data on student achievement, providing professional development to teachers (at least four sessions per year, serving as specialists to support teachers facing struggling readers, supporting the development of school teams, and creating annual site reports for the university training center; Reading Recovery Council of North America, 2012). Teacher leaders themselves receive continuous support through professional development activities, attending three sessions throughout the year: once in the fall and once in the spring at their respective university training centers, and once in the summer at an annual program–sponsored summer institute.

Teacher leaders also work with district-site coordinators to establish policies and procedures that promote quality districtwide implementation, especially with regard to program structure, budgeting, and personnel processes. Finally, to complete the collaborative approach to program implementation, University trainers connect national DLL organizations to district services, provide ongoing professional development for teacher leaders, and offer technical assistance and networks of communication (Reading Recovery Council of North America, 2012, p. 29). They also monitor and evaluate the implementation of the program in affiliated sites and conduct research on DLL (Reading Recovery Council of North America, 2012). This collaborative infrastructure strives to produce well-trained, supported teachers who can effectively teach students using real-time data to implement literacy instruction that is responsive to individual student needs.

Ultimately, DLL extends the successful Reading Recovery approach to ELs struggling with English literacy by first addressing literacy in their native, Spanish language. The program aims to improve early reading and writing outcomes for the lowest achieving ELs through a focused, short-term approach that offers the needed boost to catch up to their classmates and that, ultimately, aims to improve long-term academic achievement (Reading Recovery Council of North America, 2014e).

Analytical Methods

We estimated impacts for students randomized to immediate DLL services (hereinafter called “treatment” students) relative to students randomly assigned to the delayed-treatment condition (hereinafter called “control” students). Specifically, using the lmer function of the R package lme4 we estimated the impact of being assigned to receive DLL services, or the “intent to treat,” on students’ performance on the various literacy outcomes using the following two-level—students nested within teachers—regression model:

Y ij = α + β ( DL L ij ) + γ ( PRETES T ij ) + ∑ φ X ij + u j + ε ij

In this model, Y ij represents the test score of students within schools, α represents the model intercept, β is the coefficient representing the impact of DLL, γ represents the average difference for the outcome associated with a one-unit change in the pretest measure, ∑ φ X ij includes a vector of indicator variables representing those schools in the sample in which more than one DLL teacher was nested, u j is the school-specific error, and ε ij is the student-specific error term. Thus, most important, this analytical model not only estimates the student-level impact of receiving DLL services but also estimates the random variation in the DLL impact across teachers. The level one, or student-level, impact speaks to the overall effectiveness of DLL services, and the random level two, or teacher-level, component of our model addresses the extent to which the DLL impacts may vary across different teachers. Though our estimation of treatment-effect heterogeneity—the random variation in DLL impacts across teachers—is relatively underpowered (Bloom & Spybrook, 2017), these estimates provide some insight into the degree to which DLL impacts can be replicated across teachers.

Attrition Analysis

Attrition in a RCT occurs when the final analytical sample is different from the initial sample that was randomly assigned at baseline. Both the overall attrition and the differential attrition, between treatment and control groups, can affect the statistical equivalence of the initial randomized sample and can potentially create bias in outcome estimates (What Works Clearinghouse [WWC], 2017). In this section, we report our observed attrition outcomes relative to attrition standards developed by the WWC which describe guidelines for determining low, moderate, and high attrition, based on a combination of overall and differential attrition rates. Attrition was calculated for both standardized tests: IdO and Logramos. Examining attrition for both tests at the teacher level shows no teachers attrited from the IdO sample; however, for the Logramos sample, six teachers attrited by way of missing posttest data. Below, we document the teacher and student attrition from our analytical sample.

Baseline Equivalence

Our preliminary analyses assessed the extent to which data attrition may have compromised the baseline equivalence of the treatment and control groups. Guidance from the WWC’s most recent English Language Learners Evidence Review Protocol indicates: “. . . for the groups to be considered equivalent, the groups must be similar for each outcome on a pretest or a reasonable predictor of posttest performance” (WWC, 2013, p. 5). Thus, we assessed the equivalence of the final analytical samples in terms of the treatment and control students’IdO and Logramos pretest scores. Table 1 shows the results of these analyses for both the IdO and Logramos outcomes. In addition, Table 1 provides descriptive data for the treatment and control samples, including information such as gender, race/ethnicity, and other student background data. For the pretest scores, we report the standardized differences between treatment and control groups as an effect size difference, d. The results suggest that, for the IdO outcomes, with the exception of the Ohio Word Test, the differences between the treatment and control groups’ baseline characteristics were less than 0.25 standard deviation units. Since the IdO sample is categorized as having low attrition, it meets WWC group design standards regardless of the differences between the treatment and control groups. For the Logramos sample, all pretest differences between the treatment and control groups were less than 0.25. That is, the Logramos sample meets WWC group design standards with statistical adjustment for the pretest.

Regarding the student background characteristics, the overall sample was highly homogeneous with regard to race/ethnicity, free lunch participation (a proxy for poverty), and whether languages other than English were spoken in the students’ homes, and no notable treatment-control differences were found. The control students did participate in special education services at a somewhat higher rate than treatment students and a slightly higher percentage of male students were found in the treatment than control group, but these differences were limited to 6 to 8 percentage points. In summary, both the IdO and Logramos samples meet the WWC Group Design Standards, respectively, without and with pretest covariate adjustment.

Impact Estimates

The analyses of DLL effects on both IdO and Logramos outcomes include pre- and posttest scores, which we standardized to a mean of 0 and a standard deviation of 1. Thus, each of the coefficients can be interpreted as an effect size, which is expressed in standard deviation units. We included no student background characteristics, such as race/ethnicity, as additional covariates because there was little variability found for these measures and their inclusion did not improve the precision of our impact estimates.

Instrumento de Observacion

Table 2 shows the average DLL effects on six IdO outcomes: Letter Identification; Ohio Word Test; Concepts about Print; Writing Vocabulary; Hearing and Recording Sounds and Text Reading Sounds; and Text Reading Level. We found statistically significant student-level impacts of assignment to DLL services on all IdO posttest outcomes, ranging from an effect size of 0.49 for the Letter Identification outcome to d = 1.24 for the Text Reading Level subtest. Because we tested the statistical significance of six outcomes within the literacy domain tested by the IdO, we used the Benjamini-Hochberg method to adjust for multiple comparisons by controlling the false discovery rate. Specifically, we rank ordered the six statistically significant findings within the domain in ascending order of the p values, such that p₁≤p₂≤p₃…≤p₆. For each p value, p_1–6, we computed,

p i = i α / M ,

where i is the rank for p_i, with i = 1, 2, …6; M is the total number of findings, M = 6, within the domain; and α is the target level of statistical significance, α = .05. Even after adjusting for multiple comparisons across the six IdO tests, by using the Benjamini-Hochberg correction, these statistically significant results held. In other words, our analyses indicated that the treated students served by DLL outperformed the control students on all IdO measures.

OPEN IN VIEWER

Table 2

Hierarchical Linear Model Results for IdO Outcomes

	IdO: Letter Identification		IdO: Ohio Word Test		IdO: Concepts about Print		IdO: Writing Vocabulary		IdO: Hearing and Recording Sounds		IdO: Text Reading Level
	β	SE	β	SE	β	SE	β	SE	β	SE	β	SE
Fixed effects
β0	−0.17	0.16	−0.22	0.16	−0.49***	0.14	−0.59***	0.14	−0.31	0.16	−0.76***	0.13
β1 (DLL)	0.49***	0.14	0.51***	0.13	0.92***	0.17	1.07***	0.14	0.48**	0.16	1.24***	0.13
β2 (pretest)	0.61***	0.07	0.52***	0.07	0.41***	0.06	0.38***	0.07	0.50***	0.07	0.37***	0.06
Variance components
Between-teacher variance for intercept ( τ 0 2 )	0.09		0.03		0.04		0.00		0.09		0.00
Between-teacher variance for slope ( τ 1 2 )	0.11		0.00		0.41***		0.14*		0.19		0.10
Covariation in intercept- slope ( τ 01 )	−0.10		−0.01		−0.12		−0.00		−0.13		0.00
Residual (σ2)	0.64		0.75		0.53		0.58		0.73		0.54

Note. IdO = Instrumento de Observación; SE = standard error; DLL = Descubriendo la Lectura. All models include school fixed effects representing those schools with more than one DLL teacher.

*

p < .05. **p < .01. ***p < .001.

In addition to examining the student-level effects of DLL, we tested whether there were random teacher-level effects, as indicated by between-teacher variability in the outcomes, using a random intercepts and slopes model at the teacher-level. First, using a random intercept (u_0j), we tested whether the intercepts varied across teachers. Table 2 shows that the between-teacher variation for the intercept ( τ 0 2 ) ranges from 0 to 0.09 across the six IdO outcomes; however, none of these estimates is statistically significant. This finding suggests that for each of the IdO outcome measures there is no statistically significant variation in the teacher-level intercept. Next, modeling a random slope (u_1j), we tested whether there was random variation across teachers for the DLL impact. In other words, this random slope addresses the question: Do teacher effects moderate the overall DLL treatment effects on IdO outcomes? Results show teacher-to-teacher differences do not moderate DLL effects on four of six IdO outcomes (i.e., Letter Identification, Ohio Word test, Hearing and Recording Sounds and Text Reading Sounds, and Test Reading Level). For Concepts about Print ( τ 1 2 = 0.41, p < .001) and Writing Vocabulary ( τ 1 2 = 0.14, p < .05), the overall DLL effects were statistically significantly different across teachers.

In sum, assignment to the DLL program produced impacts across all six IdO posttest measures, and these effects were relatively consistent from teacher to teacher (with the exception of one outcome, Concepts about Print).

Logramos

Table 3 shows the average DLL effects for the three Logramos outcomes: Vocabulary, Comprehension, and Word Analysis. The DLL effect sizes for the Logramos posttests ranged from d = 0.36 for the Word Analysis subtest to d = 0.48 for Vocabulary. All three Logramos outcomes, including Vocabulary (β = 0.8, p < 0.01), Comprehension (β = 0.40, p < .05), and Word Analysis (β = 0.36, p < .01), revealed the statistically significant impacts of DLL. After applying the Benjamini-Hochberg correction for multiple hypothesis tests, using the same procedure as outlined above for the three Logramos outcomes, these impacts remained statistically significant at their respective adjusted critical p values.

OPEN IN VIEWER

Table 3

Hierarchical Linear Model Results for Logramos Outcomes

	Logramos: Vocabulary		Logramos: Comprehension		Logramos: Word Analysis
	β	SE	β	SE	β	SE
Fixed effects
β0	−0.16	0.2	−0.06	0.25	−0.17	0.26
β1 (DLL)	0.48**	0.17	0.40*	0.18	0.36**	0.14
β2 (pretest)	0.44***	0.09	0.41***	0.07	0.50***	0.08
Variance components
Between-teacher variance for intercept ( τ 0 2 )	0.08		0.32***		0.31***
Between-teacher variance for slope ( τ 1 2 )	0.00		0.28*		0.01
Covariation in intercept-slope ( τ 01 )	0.01		−0.05		0.05
Residual (σ2)	0.76		0.40		0.49

Note. SE = standard error; DLL = Descubriendo la Lectura. All models include school fixed effects representing those schools with more than one DLL teacher.

*

p < .05. **p < .01. ***p < .001.

Additionally, we tested whether there was between-teacher variability for the outcomes. Using a random intercepts model, we tested whether the intercept varied across teachers. The results suggested that across the three Logramos outcomes, two of the random intercepts were statistically significant for Comprehension ( τ 0 2 = 0.32, p < .001), and Word Analysis ( τ 0 2 = 0.31, p < .001). That is, there was between-teacher variability for these outcomes. Next, using a random slope model, we tested whether teacher effects moderated the overall DLL effects on the Logramos outcomes. The results showed that teacher effects do not moderate DLL effects for the Vocabulary and Comprehension IdO outcomes, but for the Comprehension outcome, the overall DLL effects were statistically significantly different across teachers ( τ 1 2 = 0.28, p < .05).