Evaluating contextually adapted reading interventions with third-grade,Costa Rican students experiencing significant reading difficulties

Abstract

Scholarship in school psychology has continued to document the need and importance of contextually relevant intervention and prevention research, but this type of research remains relatively scarce. Also problematic, this type of research is even more limited in low- and middle-income countries (LMIC) compared to high-income countries. This situation within school psychology scholarship not only has negative implications for research and practice, it also limits internationalization within the discipline. The geographical context for the present study was in Costa Rica, currently a Latin American LMIC. Given the global importance of literacy, this article describes an experimental evaluation comparing two time- and resource-efficient reading interventions that differed only by instructional grouping: A one-on-one intervention, and an even more resource-efficient small-group intervention. Participants included third-graders experiencing significant reading difficulties. Analyses showed that all students benefitted from intervention, but some students responded somewhat more favorably to one intervention versus the other. Limitations, implications, and future research directions are discussed, particularly within the context of international school psychology and how professionals in the discipline can benefit from more intervention research in otherwise underrepresented global regions.

Keywords

dual-language programs intervention Latin America LMIC reading fluency single-case design research small-group instruction Spanish reading

The overall purpose of this article is to describe an experimental study that evaluated two evidence-based interventions targeting reading fluency—both implemented with third-grade, Costa Rican students with significant reading difficulties. The evidence-based interventions were instructionally similar (i.e., included the same instructional strategies that were implemented in the same sequence and over the same amount of time), but they differed by instructional grouping (i.e., the study compared a one-on-one versus a small-group version of the intervention). Each key aspect of the study rationale is summarized next.

Importance of reading and reading fluency

Few would argue the numerous benefits and opportunities that come from an individual developing proficient literacy skills and the need for children to develop such skills early in their schooling (e.g., Kakarmath, Denis, Encinas-Martin, Borgonovi, & Subramanian, 2018; Mullis, Martin, Foy, & Hooper, 2017). This is true around the world, and yet evidence continues to reflect that many nations have large percentages of individuals with inadequate literacy skills (e.g., Lions & Pena, 2016; Mullis et al., 2017). In high-income countries (HIC) like the USA, for example, national data reveal that 66% of fourth-grade students read below a proficient level and 24% read below a basic level (National Center for Education Statistics, 2015). In low- and middle-income countries (LMIC), when data are available, literacy statistics are usually poorer compared to most HIC (Center for Global Development, 2016; Lions & Pena, 2016; Mullis et al., 2017).

In most if not all languages, reading fluency arguably helps to improve reading comprehension. Research examining English and Spanish, for example, suggests that reading fluency is important during early grades because it is strongly correlated with reading comprehension (e.g., Álvarez-Cañizo, Suárez-Coalla, & Cuetos, 2015; Daane, Campbell, Grigg, Goodman, & Oranje, 2005) and is considered a major component of effective reading instruction (Armbuster, Lehr, & Osborn, 2001; Lee & Yoon, 2017). Yet, a large percentage of early readers do not acquire adequate reading fluency. Data in the USA reveal that approximately 40% of fourth-grade students are ‘nonfluent’ readers (Daane et al., 2005) and international data suggest similar patterns (Mullis et al., 2017).

Intervention research: Essential for evidence-based practice but relatively scarce

For school psychologists and other educators around the globe, context- and culturally-relevant scholarship about intervention and prevention strategies is needed to advance research and provide practitioners with evidence of practices that are likely (or not likely) to support particular intervention approaches (Begeny, Levy, Hida, & Norwalk, 2018b; Nastasi, 2017; Strein, Cramer, & Lawser, 2003). Of course, this is true in HIC (Bliss, Skinner, Hautau, & Carroli, 2008; Villarreal, Castro, Umana, & Sullivan, 2017) and LMIC (Arnett, 2008; Dang, Weiss, Trung, & Ho, 2018; Leon, Campagnaro, & Matos, 2007). Despite the importance of intervention and prevention research in school psychology and related fields, numerous studies have shown that experimental intervention or prevention studies only account for approximately 10% of the scholarship, and is far less than most other types of scholarship (e.g., conceptual papers, correlational studies) even though many contend that experimental work most directly informs and supports evidence-based practice (Begeny et al., 2018b; Bliss et al., 2008; Burns, Klingbeil, Ysseldyke, & Petersen-Brown, 2012; Hseih et al., 2005; Seethaler & Fuchs, 2005; Strein et al., 2003; Villarreal et al., 2017). As such, there is a continued and expressed need for school psychologists and other educators to have more access to culturally relevant experimental studies that can advance research and support practice (Dang et al., 2018; Leon et al., 2007; van de Vijver, 2013; Villarreal et al., 2017).

An even greater scarcity of intervention research in LMIC

The need for additional intervention and prevention research is true regardless of the country a professional works within, but it is important to highlight the overwhelmingly consistent data showing that most of the available scholarship pertains to the Western world and HIC (e.g., participants living in North America or Western Europe). Very little research reflects the majority of the world population—i.e., participants residing outside of these global regions (e.g., Arnett, 2008; Begeny et al., 2018c; O’Gorman, Shum, Halford, Ogilvie, 2012; Piocuda et al., 2015). Most pertinent to this study, Begeny et al. (2018b) found that across eight school psychology journals from 2002–2016, only 0.4% of the experimental studies published in those journals included participants from a country within Latin America, whereas 95.1% of the studies included participants from North America or Europe. Some may quickly dismiss this lack of research representing much of the world’s population by suggesting there are a sufficient or similar number of studies with participants from the non-Western world reflected in other journals published around the globe. However, those who have carefully examined that possibility do not find that to be the case (e.g., Arnett, 2008; Begeny et al., 2018c; Begeny, Wang, Hida, Oluokun, & Jones, 2018d; van de Vijver, 2013).

Furthermore, the above evidence comes predominantly from journals that specifically aim to publish international research. School Psychology Quarterly (SPQ) is a good example because for many years the mission of the journal has been to publish international scholarship. The brief description of the journal on its official website currently states, ‘We welcome manuscripts from scholars throughout the world,… [such as] work that has the potential to be adapted to and implemented around the globe to address the needs of diverse populations, cultures, and communities’. From an international perspective this is, of course, a laudable mission of the journal and is consistent with the majority of school and educational psychology journals published around the world (Begeny et al., 2018d). However, a critical look at its ability to fulfil this mission suggests there is substantial room for improvement. For instance, Begeny et al. (2018b) found that SPQ published only one causal-experimental study with participants from outside of the United States from 2002–2016, and Begeny et al. (2018c) found that across a similar time period, the total percentage of articles that included participants from outside of the Western world (including but not limited to experimental studies) was only 1.6%—all from the Asia-Pacific region. These findings are generally consistent with several other psychology and school psychology journals that state a mission of publishing international scholarship (Arnett, 2008; Begeny et al., 2018c). As another example, compared to seven other journals, School Psychology International published more experimental studies with participants residing outside of the USA, but the number was still relatively low (37 total articles across 15 years) and participants from some global regions (e.g., Latin America, Africa) were represented in two or fewer articles across those 15 years (Begeny et al., 2018b). These examples are presented simply to evidence the need for additional experimental research and other types of intervention studies to reach the pages of internationally-oriented, discipline-relevant journals.

A necessity for evaluating intervention strategies in different cultural contexts

When it comes to advancing research and supporting practice, there are excellent reasons for evaluating indigenously developed practices and strategies as well as considering how practices developed in one cultural context might be adapted for use in another culture or context (Nastasi, 2017; van de Vijver, 2013). For the purposes of this study, discussion will only focus on the latter. The following highlights some central ideas about contextual or cross-cultural research; interested readers might also consider work by others who have discussed this topic in greater detail (e.g., Begeny, 2018; Dang et al., 2018; Nastasi, 2017; van de Vijver, 2013).

Benefits of cross-cultural work include the following: It helps to (a) make the discipline more inclusive (e.g., as noted in the prior section, scholarship should not reflect only a small portion of the world’s population); (b) promote better practice by offering greater knowledge and insight about strategies that may or may not work in particular contexts; and (c) reinforce the fact that theories or practices developed in one cultural context cannot be assumed to apply to all cultural contexts (van de Vijver, 2013). For example, if a particular behavioral intervention program is found to be effective within several types of French schools and classrooms, one cannot assume that the same program would be effective in schools outside of France because there are many variables (e.g., cultural norms and expectations, language variables, access to resources) that could diminish the effectiveness of that program when implemented outside of French classrooms. Most pertinent to the present study, these same types of variables can also influence the effects of academic interventions.

Geographically diverse scholarship can also help to inform new ideas that may lead others to develop or evaluate similar strategies or practices within another cultural context (Arnett, 2008; Begeny et al., 2018b). Furthermore, evaluating professional practices in cross-cultural contexts and sharing scholarship about such work is a critical element within the internationalization of a discipline or sub-discipline, such as school psychology (Begeny, 2018; van de Vijver, 2013).

Context for the present study

Broadly speaking and true for many educators around the globe, school psychologists in Latin America have articulated a need for intervention and prevention research that reflects country-level and community-level realities (e.g., Leon et al., 2007; Soto, Casapia, Ponce, & Morales, 2007)—realities that are often not reflected in the large percentage of studies taking place in relatively ‘research rich’ countries. For example, Leon and colleagues (2007), describing their experiences with education in Venezuela, wrote:
Applied research is considered fundamental … the most important research-related issues include the development and evaluation of intervention strategies to determine their effectiveness in the Venezuelan reality [and] … applied practice cannot be based only on theory, because [educators] must validate the methods and tools that they commonly use in their daily professional work. (p. 433)

Unfortunately, as noted earlier, scholarship in school psychology highlights (a) the scarcity of empirical work that includes participants from global regions such as Latin America (Begeny et al., 2018b) and (b) an overall lack of scholarly outlets produced in and for Latin America (Begeny et al., 2018d). More specific to the present study, but potentially applicable to other LMIC schools, a more detailed context about the study setting is presented next—provided here rather than the Method section because this context influenced the study purpose, rationale, questions, and design.

Educators at the participating school expressed the need and interest in evaluating options of highly resource-efficient strategies to assist their most struggling third-grade readers (e.g., practices that have potential for effectiveness but could be used in less than 10 minutes, and potentially, with multiple students at once). The brevity of the intervention was particularly important because primary grade students only attended school approximately 4 hours per weekday and there were no full-time educational support staff designated to support students with academic difficulties.

Educators in the school were likewise interested in examining practices that could reasonably be used by non-credentialed teachers (e.g., individuals from volunteer groups or possibly non-instructional school staff) and would not require costs for potentially expensive materials (e.g., books, white-boards, lamination tools). Certainly there are many educators in HIC who also desire time-efficient, low-cost strategies to support struggling learners (a context our team often works within), but the time- and cost-efficiency desired for this specific project was more unique to a low-resource school in a LMIC. Finally, educators at the school desired strategies that would be highly engaging for the students because extra space for supplemental instruction was extremely limited, and large class sizes combined with a mainly outdoor school setting could sometimes present a noisier climate that could distract students during supplemental instruction, especially those who struggle with reading.

At the inception of project communication, work was done to explore whether educators within the school had any preferences for the types of literacy practices they wanted to evaluate and would fit the contextual needs described above. We also searched for relevant scholarship (in English and Spanish) that could inform decisions about strategies to evaluate. For various reasons it was collectively decided that we should attempt to evaluate two versions of an intervention (a one-on-one and small-group version) that could specifically target students’ reading fluency and would adhere to previously described contextual factors. This evaluation also served to advance the small but growing body of work aiming to compare the relative effectiveness of similarly designed one-on-one versus small-group reading interventions (e.g., Klubnik & Ardoin, 2010; Ross & Begeny, 2015; Vaughn et al., 2003).

The focus of intervention was also influenced by the fact that the students most in need were struggling with reading fluency, and their teachers felt it would make less sense to specifically target areas such as comprehension or vocabulary. Because there is currently a lack of reading fluency intervention research with participants residing in Latin America, the vast majority of scholarship pertaining to the selected intervention strategies were based on studies conducted in the USA (see reviews by Begeny et al., 2018a; Lee & Yoon, 2017; Morgan & Sideridis, 2006; Richards-Tutor, Baker, Gersten, Baker, & Smith, 2016; Therrien, 2004). This study therefore integrated and adapted evidence-based strategies described within that literature-base, ultimately creating two versions of a brief intervention that was culturally and contextually appropriate.

Research questions

Three main research questions guided this work. First, was the one-on-one and/or small-group version of the intervention more effective than a control condition? Second, were the two interventions more or less effective compared to each other? Third, did student participants report positive attitudes about each intervention? This third question was important not only because it is generally important to examine intervention acceptability, but also because international data highlight the value of students having a positive attitude toward reading and how positive attitudes are correlated with higher reading achievement (Mullis et al., 2017).

Method

Participants and setting

This study included six third-grade students (three male and three female) from one school located in Costa Rica (a country currently classified as a LMIC by the World Bank). All third-grade students in the school (N = 56) were assessed during the middle of the school year using Curriculum-Based Measurement of reading (CBM-R) procedures and AIMSweb Spanish passages (Pearson Education, 2011). All assessment procedures were conducted in a location free from noise and distractions (i.e., a small school library).

We sought to work with students demonstrating the most difficulty and ultimately selected six students reading at approximately the 10th percentile according to the AIMSweb Spanish CBM-R Growth Table for words read correctly per minute (WCPM) during a winter benchmark assessment of Spanish speaking third-grade students. These students were also identified by their teachers as being in greatest need of a reading intervention. All students’ nationality was Costa Rican and the average age of the students was 9.6 years. One student had been previously retained, but none of the students received special education services because, due to the school’s limited resources, only students with relatively more severe disabilities (e.g., autism) were eligible for special education services. Based on the grade-wide reading assessment, the six participants read the following median number of WCPM and (noted in parentheses) words incorrect per minute (WIPM): Rolando: 50 (1); Sofia: 41 (2); Arturo: 43 (1); Alejandra: 46 (2); Ligia: 42 (2); Gerardo: 36 (0). Across all 56 third-grade students evaluated in the grade-wide assessment, the 50th percentile score was 74 WCPM, which was highly consistent with the AIMSweb Spanish CBM-R Growth Table that reported the 50th percentile as 76 WCPM and the 10th percentile at 43 WCPM. Participants’ screening data also confirmed teachers’ observations that students could decode relatively accurately but lacked text reading fluency.

Materials

Oral Reading Fluency (ORF) passages

ORF passages used for screening were described in the Participants section. ORF passages used during the two intervention and control conditions in the current study came from the first grade Spanish AIMSweb progress monitoring materials (Pearson Education, 2011). These materials were used because (a) they represented a set of published materials that were developed in Spanish for the purposes of monitoring students’ text reading fluency; (b) they were the only standardized Spanish reading materials available to the interventionists—mainly due to the limited options for Spanish progress monitoring materials; and (c) first grade materials represented the most appropriate instructional match for the students (based on brief screenings with CBM-R materials from grades one, two, and three).

Acceptability

To evaluate students’ attitudes about the two different intervention conditions, each student independently completed a three-item acceptability scale developed for the purposes of this project. Using a five-point Likert-scale (1 = strongly disagree; 3 = neither agree or disagree; 5 = strongly agree), students responded to the following items for each intervention condition: (a) I liked doing this reading activity, (b) I think this reading activity helped me become a better reader, and (c) I think this reading activity would be good to use with other third-grade students.

Experimental design and procedures

Consistent with the majority of studies described in a recent review of small-group interventions that target reading fluency (Begeny et al., 2018a), an alternating-treatments design (ATD) was used to evaluate the effectiveness of the two interventions, with a brief control (no-intervention) condition used throughout the study to evaluate the effectiveness of each intervention package versus a no treatment condition. To avoid possible carry-over effects of the conditions, all conditions were randomly counterbalanced. The total duration of this project spanned 10 weeks with each intervention condition and control condition scheduled for implementation seven times and four times, respectively. All study procedures were implemented by two adults who volunteered within the school. These interventionists were trained according to following sequence of procedures: (a) they reviewed the protocol of procedures and student directions associated with each study condition and discussed the procedures with the lead researcher; (b) they practiced implementing the procedures with second grade students in the school who did not participate in the study, and during these practice sessions they received implementation feedback and guidance from the lead researcher; and (c) they were deemed ready to implement the conditions for purposes of the study after implementing all procedures with 100% accuracy in two consecutive sessions with the second grade students.

Control

During control (CL) sessions, students read a Spanish passage two times, with the second reading (i.e., post-test of immediate gains) immediately following the first reading (i.e., pre-test). Students had never been exposed to the CL passages prior.

Intervention conditions and components

Two interventions were evaluated in this study and they only differed by how many students received intervention at the same time. Specifically, the study included a one-on-one (1:1) intervention (i.e., the interventionist worked individually with one student), and a small-group (SG) intervention where the interventionist worked with three students at once. Both interventions included the same components, the same sequence of components, and required 7 to 8 minutes to implement. As stated earlier, each intervention integrated strategies found to be effective in studies involving participants with similar reading difficulties (e.g., Begeny et al., 2018a; Lee & Yoon, 2017; Richards-Tutor et al., 2016; Therrien, 2004), but almost all of those studies were conducted in the USA and it could not be assumed such strategies could be successfully adapted and used in non-USA (and non-English) contexts (van de Vijver, 2013). The integrated evidence-based strategies were: repeated reading, modeling, systematic error-correction, performance feedback, motivational/reward procedures, and brief story retell.

Table 1 describes each procedure for both the 1:1 and SG conditions. As shown in Table 1, the procedures were nearly identical but implemented according to the 1:1 or SG context. Regarding Step 7 in the table, throughout the study there were only two occasions when an interventionist did not provide the maximum number of stars on the student’s Star Chart—once during one SG session and once during one student’s 1:1 session. Thus, student behavior, engagement, and effort was good throughout the study. Additional details about this type of Star Chart are described elsewhere (e.g., Ross & Begeny, 2015).
Table 1.
Summary of procedures during one-on-one and small-group intervention conditions.

Step Procedure Description of procedure during One-on-One Condition Description of procedure during Small-Group Condition

1 Modeling • Interventionist modeled fluent oral reading by reading with good expression and at a pace only slightly faster than the student could read the story. • Student read along silently from his or her copy of the passage while interventionist read aloud. • 5–7 times, interventionist paused from reading the story and at those moments asked student to say next word in the story (which helped to ensure student was following along). • Same procedures as 1:1 condition except that each time the interventionist paused from reading, one student was randomly selected to read the next word.

2 Oral reading • Student read the story aloud while the interventionist recorded WIPM. • One student from group was randomly selected to be the ‘group leader’. This student read the passage aloud while other students read aloud, but with a softer voice. • If group leader came to a word he or she did not know, interventionist provided the word aloud for the group.

3 Retell • Student was asked to retell what he or she remembered about the story. • Each student in group was randomly selected to say one thing remembered about the story.

4 Error-correction • Interventionist demonstrated for the student how to correctly read a misread word. • Student then read aloud a brief 3–6 word phrase that included the error word (and repeated 3 times). • Procedure was replicated for each misread word. • Same procedure as 1:1 condition except all students were instructed to read the phrase aloud and in unison.

5 Oral Reading • Student read the story aloud a second time while the interventionist recorded WIPM. • Same as small-group condition Step #2(described above).

6 Behavior Specific Praise • Interventionist provided praise for specific reading behaviors (e.g., improving accurate reading, reading with good expression). • Same procedure as 1:1 condition except interventionist provided praise to all students in the group.

7 Motivation-based Star Chart • 1-2 stars were recorded on student’s Star Chart according to student’s effort during session; after receiving 15 stars, student could select a small reward from a sample of inexpensive items (e.g., pencils, stickers) the student rated as desirable. • Same procedure as 1:1 condition except interventionist provided stars on each student’s Start Chart (all students received same number, which was determined from the group’s overall effort during the session).

Assessment procedures

Immediately before and again immediately after each intervention session, each student read aloud the intervention passage for that session (i.e., the passage practiced only on that specific day) while the interventionist recorded the student’s number of WCPM and WIPM for session-by-session assessment and progress monitoring purposes. The reading of a given passage that occurred immediately prior to an intervention session will be referred to as the pre-test assessment, and the reading immediately following an intervention session will be called the post-test assessment. To measure a student’s retention of WCPM and WIPM improvements, the student read the same passage again approximately four days later (i.e., she received the retention assessment). This retention assessment occurred just prior to administering the pre-test assessment for a given passage. For both the 1:1 and SG intervention conditions, the average latency from intervention session to retention assessment was 4.0 days (range = 3–5 days). Retention assessments were not administered as part of the CL condition because unlike the intervention conditions, students did not practice the passages during the CL condition. All assessments, including those for the the SG condition, were completed in a setting that would not allow other students to hear a student reading aloud.

Dependent variables

To examine WCPM gains for each student across conditions, difference scores were determined between the student’s initial reading of the passage (pre-test) and his or her reading of the passage immediately following intervention (i.e., post-test or immediate gain). Difference scores were also determined between the student’s pre-test score and his or her reading of the passage approximately four days after the pre-test reading (i.e., retention gain). This procedure and use of immediate and retention WCPM gains is consistent with several past studies examining reading fluency interventions (e.g., Klubnik & Ardoin, 2010; Ross & Begeny, 2015).

Data analysis

Data within and across conditions were first analysed visually. Visual analysis involved examining level, trend, variability, immediacy of effect, overlap, and consistency of data patterns within and between phases (Kratochwill et al., 2012). However, as has been noted by others, a measure of nonoverlapping data often serves as the primary criterion for determining differences between conditions in an ATD because level and trend are sometimes less relevant or difficult to observe in ATDs (Barlow, Nock, & Hersen, 2009; Manalov & Onghena, 2017). Ma’s (2006) Percentage of Data Points Exceeding the Median (PEM) method was used to calculate the percentage of 1:1 and SG data points that exceeded the median score for the control condition. PEM was also used to evaluate possible differences between the 1:1 and SG conditions. Compared to other statistical methods of examining intervention effects with ATD data, such as Percentage of Non-Overlapping Data (PND) and Percentage of All Non-Overlapping Data (PAND), the PEM method was used in this study because it corrects for possible ceiling effects that may be present with the PND and PAND methods, and it minimizes a potential for Type 2 error that may result from PND methods that capture outlier data—which is particularly plausible when using the WCPM dependent variable, which is known to have relatively high levels of variability (Christ & Silberglitt, 2007). Other possible effect size options, such as Tau or Tau-U, were not used in this study due to known limitations when used with ORF data (e.g., ceiling effects) and, at least at this time, a lack of clarity about their appropriate use with ATDs (e.g., Klingbeil, Van Norman, McLendon, Ross, & Begeny, 2018; Manalov & Onghena, 2017; Ross & Begeny, 2014). Evidence for the validity and relative benefits of PEM is reported by Ma (2006), Chen and Ma (2007), and Manolov, Solanas, and Leiva (2010). Consistent with Ma’s recommendations, when at least 70% of the data points from one condition exceed the median data point of the comparison condition, this shows that one condition is more effective (70% to 89% of the data points) or highly more effective (≥ 90%) than the other.

As a tertiary approach to data interpretation, mean difference data across conditions were calculated as a measure of central tendency. Mean difference data have strengths and limitations for interpreting ATD data, but according to Manalov and Onghena (2017), they are the most common form of quantification in ATDs and can be particularly useful when data patterns (even nonlinear patterns) are similar across conditions. By calculating means across conditions, this also allowed for a relatively straightforward method of examining whether mean differences between conditions exceeded what might be expected simply from measurement error, which is increasingly being used in single-case experimental design (SCD) research (e.g., Burns et al., 2017; Ross & Begeny, 2015). Although there are no known standard error of measurement (SEM) data available for the specific ORF passages used in this study, tertiary analysis of mean differences relied upon known estimates of SEM for third-grade ORF passages. Christ and Silberglitt (2007) found that under assessment conditions similar to those in the present study, 6–7 WCPM is a good estimate of the SEM. From this, if a difference between conditions was greater than 7 WCPM, this offered supplemental evidence of differences between the conditions.

Procedural integrity and interscorer agreement

Each step of the assessment and intervention procedures had a brief, scripted direction the interventionist read prior to the step to ensure the student(s) understood the instructions. In addition, all students learned the directions and proper sequence of intervention steps during a practice session of each intervention condition. To measure procedural integrity, each interventionist’s implementation was observed by the other interventionist. The protocols developed for training also served to measure procedural integrity. Across interventionists, 75% of the CL sessions, 66% of the 1:1 sessions, and 100% of the SG sessions were observed. All procedures for each condition were followed with 100% accuracy with the exception of two total instances when an interventionist implemented an intervention step in the wrong sequence.

Inter-scorer agreement was calculated (using an on independent observer) on 42% of all screening passages administered and 35% of the assessments that preceded or followed the intervention and control conditions. Agreement was calculated by dividing the total number of agreements by the total number of agreements plus disagreements, and then multiplying by 100. The average inter-scorer agreement was 98.6%.

Results

Figures 1 and 2 show each student’s scores across the conditions for both immediate gains (Figure 1) and retention gains (Figure 2). The y-axis of the graphs in Figure 1 are labeled as a ‘WCPM change’ instead of ‘WCPM gains’ because in some instances a student’s WCPM was less than the pre-test WCPM, resulting in a negative change-score and not a gain. Due to space considerations within this manuscript, an abbreviated visual analysis revealed the following. For each student’s immediate WCPM scores (Figure 1), data were relatively variable across conditions and there were no obvious trends. Known measurement error associated with ORF passages (Christ & Silberglitt, 2007; Klingbeil et al., 2018) helps to explain relatively variable WCPM scores and is consistent with other ATD studies that use ORF as a dependent variable (Begeny et al., 2018a). Level and lack of overlap in performance for the two intervention conditions (compared to CL) was most pronounced with Sofia, Arturo (especially the 1:1 condition), Alejandra, and Ligia. Figure 2 shows that for each student there was a fair amount of variability and overlap in retention scores between the 1:1 and SG conditions. Alejandra’s data reveal one exception to this; the SG data were generally higher than the 1:1 data, with minimal overlap.
Figure 1.
Each participant’s immediate words read correct per minute (WCPM) score.

Figure 2.
Each participant’s retention words read correct per minute (WCPM) score.

As stated previously, nonoverlapping data is often a central criterion for establishing differences between conditions when using an ATD. PEM analyses are summarized in Table 2 (see superscripts) and reveal that for all but one student (Rolando) at least one of the intervention conditions was considered more effective than the control condition. For Arturo, Ligia, and Gerardo the 1:1 condition was considered highly more effective compared to the control, and for Sofia and Alejandra it was considered more effective. The SG condition was highly more effective than the control for Ligia and Gerardo, and more effective for Arturo and Alejandra. In comparing the 1:1 to the SG condition, the 1:1 intervention was more effective than SG for Arturo (when comparing immediate gains), but there were no differences for the other students. When comparing retention gains, the SG condition was more effective than the 1:1 intervention for Alejandra and Gerardo, but there were no differences for the other students.
Table 2.
Students’ average immediate and retention gains in words read correctly per minute (WCPM) across conditions and summary data from analyses of the percentage of data points exceeding the median (PEM).

Student Immediate WCPM Gains
Retention WCPM Gains

CL 1:1 SG 1:1 SG

Rolando 3.3 (15.1) 14.0 (11.2) 20.0 (16.2) 22.0 (11.8) 26.0 (14.1)

Sofia 10.0 (4.3) 23.5^a (10.6) 18.2 (17.2) 25.8 (12.9) 32.6 (13.0)

Arturo 7.5 (8.8) 26.8^{b
f} (8.7) 14.5^c (6.8) 27.0 (7.6) 23.2 (4.8)

Alejandra 13.0 (1.0) 28.7^a (13.9) 26.5^c (16.6) 24.7 (7.2) 37.8^h (8.9)

Ligia 5.5 (14.1) 30.4^b (6.6) 33.3^d (11.9) 35.0 (11.6) 38.2 (11.1)

Gerardo 2.3 (6.1) 8.0^b (8.2) 7.5^d (6.0) 12.0 (2.5) 15.0^g (4.1)

Averages 6.9 21.9 20.0 24.4 28.8

Notes. Standard deviations are indicated in parentheses. CL = Control; 1:1 = One-on-one Intervention (interventionist and student); SG = Small Group Intervention (interventionist and three students). Underlined values indicate mean score exceeds CL mean score by more than 7 WCPM (i.e., the SEM).

a
80% to 90% of 1:1 data points exceeded the median data point of the CL condition.

b
>90% of 1:1 data points exceeded the median data point of the CL condition.

c
80% to 90% of SG data points exceeded the median data point of the CL condition.

d
>90% of SG data points exceeded the median data point of the CL condition.

e
80% to 90% of 1:1 data points exceeded the median data point of the SG condition.

f
>90% of 1:1 data points exceeded the median data point of the SG condition and mean 1:1 score exceeds mean SG score by more than 7 WCPM.

g
80% to 90% of SG data points exceeded the median data point of the 1:1 condition.

h
>90% of SG data points exceeded the median data point of the 1:1 condition and mean SG score exceeds mean 1:1 score by more than 7 WCPM.

Table 2 also shows the means and Standard Deviations across students and conditions. Using the previously stated SEM criterion of 7 WCPM to examine whether mean differences between conditions are within that range of measurement error, Table 2 (underlined values) show that for all students except Gerardo, the mean of both the 1:1 and SG conditions exceed that criterion. Only in two circumstances did an intervention condition mean exceed that criterion compared to the other intervention (see Arturo and Alejandra). Collectively across the three methods of analysis, the data are mostly consistent and offer at least some evidence that one or both interventions resulted in meaningfully higher WCPM scores compared to the CL condition. For most participants there were minimal differences between the two intervention conditions, but evidence of positive intervention effects were somewhat less robust for Rolando and Gerardo, compared to all other students.

Student acceptability data suggested each student found both interventions acceptable because all students rated each item with a 4 (agree) or 5 (strongly agree) on the 5-point Likert scale. In fact, the average score for each acceptability item, for both interventions, was 4.8. Although these data only represent a small number of students, they indicate that the students liked both the 1:1 and SG interventions and felt they both helped their reading skills.

Discussion

Functional literacy is important for individuals around the globe, but far too many children lack adequate reading skills, particularly students in relatively lower-resource schools (Mullis et al., 2017). Not surprisingly, educators desire access to scholarship about intervention and prevention strategies that will inform their practice, including but not limited to context-relevant interventions designed to support students with reading difficulties (e.g., Leon et al., 2007; Soto et al., 2007; Strein et al., 2003; Villarreal et al., 2017). However, intervention and prevention research accounts for only a small fraction of the scholarship in school psychology and related disciplines (Bliss et al., 2008; Burns et al., 2012; Hseih et al., 2005; Seethaler & Fuchs, 2005)—and intervention research that includes participants from LMIC is even more limited or non-existent (Arnett, 2008; Begeny et al., 2018b; Dang et al., 2018). This is worth highlighting because, although the most recent international assessment of children’s literacy found that proficient readers tend to be in well-resourced schools (Mullis et al., 2017), such well-resourced schools are not the reality for the majority of school-aged children around the world—making intervention research in lower-resource educational contexts particularly important.

Within the context of a lower-resource school in a Latin American LMIC, this study examined two reading interventions that were implemented with third-grade students who demonstrated significant reading difficulties. Overall findings showed at least some evidence that each student benefitted from at least one intervention, but across the three approaches to data analysis, the evidence was strongest for Sofia, Arturo, Alejandra, and Ligia, and less robust for Rolando and Gerardo. Findings also showed that although both interventions were more effective than the CL condition, neither intervention was consistently more effective than the other. Only Arturo, and to some extent Sofia, seemed to benefit slightly more from the 1:1 intervention.

For the two students who responded only somewhat favorably to the interventions, there are possible reasons for this. For Rolando, his somewhat irregular school attendance (i.e., five absences throughout the time of the study) resulted in less overall intervention and fewer sessions to generate comparative data—in fact, Rolando is the only participant where the number of data points did not meet recommended SCD standards (Kratochwill et al., 2012). Combined with the fact that Rolando experienced significant reading difficulties, his lack of attendance may have impacted his ability to demonstrate strong reading improvements as a result of either intervention. For Gerardo, although he benefitted somewhat from the interventions in comparison to the CL condition, he experienced the most significant reading difficulties of all the participants (as determined from the screening assessment and from conversations with teachers). For this reason, it is logical to hypothesize that Gerardo especially needed more intensive intervention in order to demonstrate greater improvements. Keeping in mind that the study used interventions lasting only 7 to 8 minutes, most educators would recognize that students with significant reading difficulties need relatively more intensive interventions to make sizable and sustainable reading improvements.

Collectively, findings from this study were highly consistent with a recent review of the existing literature about SG interventions that target reading fluency (Begeny et al., 2018a). That study and the present study showed, for example, that (a) 1:1 and SG reading fluency interventions appear to be effective for most but not all students, and (b) some students may benefit more from 1:1 instruction compared to SG, but the majority seem to benefit equally. However, the present study extends this body of research by adapting and evaluating previously identified evidence-based intervention strategies (a) within a context, country, and language rarely discussed in the school psychology literature, and (b) with students experiencing significant reading difficulties—as opposed to most studies in this literature-base that have involved students performing above the 15th percentile on reading assessments.

Limitations, future research directions, and potential implications

As a preliminary study aiming to evaluate fluency-based intervention strategies within a Latin American context, there are certainly limitations of this study and future research is needed to determine whether the findings would generalize to other students or similar educational contexts (e.g., other schools in Latin America that have similar levels of resources, needs, and length of school-day). Future research should also address limitations of this study by using, for example, additional reading measures to evaluate the effectiveness of the interventions (e.g., to examine whether fluency gains generalize over time and to other text, and whether reading comprehension might improve from these interventions). Future research might also (a) more closely examine teacher and student acceptability of the intervention; and (b) attempt to use other types of experimental designs, such a between-group design.

Such group-design research would certainly benefit the existing research, but it is also worth noting that by using a SCD in the present study, this may spotlight an opportunity for other researchers when considering how to increase experimental research internationally. SCDs offer rigorous evaluations allowing for causal inference (Horner et al., 2005) and represent at least half or more of the experimental studies published in school psychology journals (Bliss et al., 2008; Burns et al., 2012). In some cases, they could also offer an advantage over using group-based designs because they require smaller samples and allow for better evaluation of students at the individual level. However, it does not appear that SCDs are used much in school psychology research outside of the USA. In the study by Begeny and colleagues (2018b) that examined experimental research internationally across 15 years and eight journals, of the 79 experimental studies that included participants from outside of the USA, only seven used a SCD (with none occurring in Latin America). Thus, in future efforts, if SCDs are perceived as culturally appropriate in other settings, such designs hold promise for evaluating interventions and could strengthen both research and practice.

The findings in this study may also have important implications for school psychologists who work in predominantly Spanish speaking countries and want to support struggling readers. For example, the findings suggest that educators could use (or assist other educators with using) either the 1:1 or SG intervention described in this study—or some adaptation of it. Because both interventions can be implemented in 7 to 8 minute per session, these interventions are likely to be feasible for the many educators working to improve students’ literacy skills. However, educators with particularly scarce personnel resources may opt to choose the SG intervention, and this study adds to the small but growing literature suggesting that many students (including Spanish-speaking students with significant reading difficulties) can benefit as much from SG instruction than they do from a 1:1 intervention.

As another implication for school psychologists, their involvement with implementing brief, small-group interventions may help classroom teachers improve their perceptions about school psychologists’ roles and contributions—a need summarized by Soto and colleagues (2007). For instance, through direct intervention or consolation, school psychologists can use interventions like those examined in this study to assist more students and possibly counter the misperception that psychologists only offer individualized assistance for students.

Although the context of this study was specific to a Latin American LMIC, schools in HIC also struggle to identify time- and cost-efficient reading interventions (Ross & Begeny, 2015); and particularly relevant to this study, it is worth highlighting that thousands of educators outside of Latin America teach in schools with English-Spanish dual-language programs or teach English-language learners who immigrated from Latin America. This study may therefore have relevance in these contexts as well, especially when considering the very little reading intervention research that has been conducted with English-language learners. In a review of all reading intervention research for English learners, Richards-Tutor et al. (2016) identified only 12 total studies, and only five that involved students from grades 2–6. Overall, the present study offers implications for many researchers and practitioners involved with supporting students’ literacy, and in particular helps to advance aspects of internationalization within the discipline by examining resource-efficient interventions in the context of a Latin American LMIC.

Step	Procedure	Description of procedure during One-on-One Condition	Description of procedure during Small-Group Condition
1	Modeling	• Interventionist modeled fluent oral reading by reading with good expression and at a pace only slightly faster than the student could read the story. • Student read along silently from his or her copy of the passage while interventionist read aloud. • 5–7 times, interventionist paused from reading the story and at those moments asked student to say next word in the story (which helped to ensure student was following along).	• Same procedures as 1:1 condition except that each time the interventionist paused from reading, one student was randomly selected to read the next word.
2	Oral reading	• Student read the story aloud while the interventionist recorded WIPM.	• One student from group was randomly selected to be the ‘group leader’. This student read the passage aloud while other students read aloud, but with a softer voice. • If group leader came to a word he or she did not know, interventionist provided the word aloud for the group.
3	Retell	• Student was asked to retell what he or she remembered about the story.	• Each student in group was randomly selected to say one thing remembered about the story.
4	Error-correction	• Interventionist demonstrated for the student how to correctly read a misread word. • Student then read aloud a brief 3–6 word phrase that included the error word (and repeated 3 times). • Procedure was replicated for each misread word.	• Same procedure as 1:1 condition except all students were instructed to read the phrase aloud and in unison.
5	Oral Reading	• Student read the story aloud a second time while the interventionist recorded WIPM.	• Same as small-group condition Step #2(described above).
6	Behavior Specific Praise	• Interventionist provided praise for specific reading behaviors (e.g., improving accurate reading, reading with good expression).	• Same procedure as 1:1 condition except interventionist provided praise to all students in the group.
7	Motivation-based Star Chart	• 1-2 stars were recorded on student’s Star Chart according to student’s effort during session; after receiving 15 stars, student could select a small reward from a sample of inexpensive items (e.g., pencils, stickers) the student rated as desirable.	• Same procedure as 1:1 condition except interventionist provided stars on each student’s Start Chart (all students received same number, which was determined from the group’s overall effort during the session).

Student	Immediate WCPM Gains	Retention WCPM Gains
Rolando	3.3 (15.1)	14.0 (11.2)	20.0 (16.2)	22.0 (11.8)	26.0 (14.1)
Sofia	10.0 (4.3)	23.5^a (10.6)	18.2 (17.2)	25.8 (12.9)	32.6 (13.0)
Arturo	7.5 (8.8)	26.8^{b f} (8.7)	14.5^c (6.8)	27.0 (7.6)	23.2 (4.8)
Alejandra	13.0 (1.0)	28.7^a (13.9)	26.5^c (16.6)	24.7 (7.2)	37.8^h (8.9)
Ligia	5.5 (14.1)	30.4^b (6.6)	33.3^d (11.9)	35.0 (11.6)	38.2 (11.1)
Gerardo	2.3 (6.1)	8.0^b (8.2)	7.5^d (6.0)	12.0 (2.5)	15.0^g (4.1)
Averages	6.9	21.9	20.0	24.4	28.8

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author biography

John C. Begeny, PhD, is an Associate Professor of School Psychology at North Carolina State University. He is also the founder of Helps Education Fund, a 501(c)3 nonprofit organization that offers various forms of free and research-based support for educators and parents around the globe. His primary interests relate to issues of educational equity, such as trying to improve educational equity through academic intervention, effective community–university partnerships, enhancing school-based systems, and internationalization.

References

Álvarez-Cañizo

Suárez-Coalla

Cuetos

(2015) The role of reading fluency in children’s text comprehension. Frontiers of Psychology 6: 1–8. doi: 10.3389/fpsyg.2015.01810.

Armbuster, B., Lehr, F., & Osborn, J. (2001). Put reading first: The research building blocks for teaching children to read (Kindergarten through Grade 3). Washington, DC: National Institute for Literacy.

Arnett

J. J

(2008) The neglected 95%: Why American psychology needs to become less American. American Psychologist 63: 602–614. doi: 10.1037/0003-066X.63.7.602.

Barlow, D. H., Nock, M. K., & Hersen, M. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Boston, MA: Pearson.

Begeny

J. C.

(2018) A working definition and conceptual model of internationalization for school and educational psychology. Psychology in the Schools 55: 924–940. doi: 10.1002/pits.22157.

Begeny

J. C.

Levy

R. A.

Field

S. A.

(2018a) Using small-group instruction to improve students’ reading fluency: An evaluation of the existing research. Journal of Applied School Psychology 34: 36–64. doi: 10.1080/15377903.2017.1328628.

Begeny

J. C.

Levy

R. A.

Hida

R. M.

Norwalk

(2018b) Experimental research in school psychology internationally: An assessment of journal publications and implications for internationalization. Psychology in the Schools 55: 120–136. doi: 10.1002/pits.22070.

Begeny, J. C., Levy, R. A., Hida, R. M., Norwalk, K., Field, S., Suzuki, H., et al. (2018c). Geographically representative scholarship and internationalization in school and educational psychology: A bibliometric analysis of eight journals from 2002–2016. Journal of School Psychology, 70, 44–63. doi: 10.1016/j.jsp.2018.07.001.

Begeny

J. C.

Wang

Hida

R. M.

Oluokun

H. O.

Jones

R. A.

(2018d) A global examination of peer-reviewed, scholarly journal outlets devoted to school and educational psychology. Manuscript submitted for publication .

10.

Bliss

S. L.

Skinner

C. H.

Hautau

Carroli

E. E.

(2008) Articles published in four school psychology journals from 2000 to 2005: An analysis of experimental/intervention research. Psychology in the Schools 45: 483–498. doi: 10.1002/pits.20318.

11.

Burns

M. K.

Klingbeil

D. A.

Ysseldyke

J. E.

Petersen-Brown

(2012) Trends in methodological rigor in intervention research published in school psychology journals. Psychology in the Schools 49: 843–851. doi: 10.1002/pits.21637.

12.

Burns

Taylor

Warmbold-Brann

Preast

Hosp

Ford

(2017) Empirical synthesis of the effect of standard error of measurement on decisions made within brief experimental analyses of reading fluency. Psychology in the Schools 54: 640–654. doi: 10.1002/pits.22022.

13.

Center for Global Development. (2016). Literacy of adults in developing countries: New data from a skills survey. Retrieved from https://www.cgdev.org/blog/literacy-adults-developing-countries-new-data-skills-survey.

14.

Chen

(2007) Effects of treatment on disruptive behaviors: A quantitative synthesis of single-subject researches using the PEM approach. The Behavior Analyst Today 8: 380–397. doi: 10.1037/h0100629.

15.

Christ

Silberglitt

(2007) Estimates of the standard error of measurement for curriculum based measures of oral reading fluency. School Psychology Review 36: 130–146.

16.

Daane

M. C.

Campbell

J. R.

Grigg

W. S.

Goodman

M. J.

Oranje

(2005) Fourth-grade students reading aloud: NAEP 2002 special study of oral reading. US Department of Education, National Center for Education Statistics. Washington, DC.

17.

Dang

H. M.

Weiss

Trung

(2018) Mental health literacy and intervention program adaptation in the internationalization of school psychology for Vietnam. Psychology in the Schools 55: 941–954. doi: 10.1002/pits.22156.

18.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005) The use of single-subject research to identify evidence-based practice in special education. Exceptional Children 7: 165–179. doi: 10.1177/001440290507100203.

19.

Hseih

Acee

Chungh

Hseih

Kim

Thomas

et al. (2005) Is educational research on the decline. Journal of Educational Psychology 97: 523–529. doi: 10.1037/0022-0663.97.4.523.

20.

Kakarmath

Denis

Encinas-Martin

Borgonovi

Subramanian

S. V.

(2018) Association between literacy and self-rated poor health in 33 high- and upper middle-income countries. International Journal of Public Health 63: 213–222. doi: 10.1787/19939019.

21.

Klingbeil, D. A., Van Norman, E. R., McLendon, K. E., Ross, S. G., & Begeny, J. C. (2018). Evaluating Tau-U with oral reading fluency data and the impact of measurement error. Behavior Modification. Advance online publication. doi: 10.1177/0145445518760174.

22.

Klubnik

Ardoin

S. P.

(2010) Examining immediate and maintenance effects of a reading intervention package on generalization materials: Individual versus group implementation. Journal of Behavioral Education 19: 7–29. doi: 10.1007/s10864-009-9096-3.

23.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

Rindskopf

D. M.

Shadish

W. R.

(2012) Single-case intervention research design standards. Remedial and Special Education 34: 26–38. doi: 10.1177/0741932512452794.

24.

Lee

Yoon

S. Y.

(2017) The effects of repeated reading on reading fluency for students with reading disabilities: A meta-analysis. Journal of Learning Disabilities 50: 213–224. doi: 10.1177/0022219415605194.

25.

Leon, C., Campagnaro, S., & Matos, M. (2007). School psychology in Venezuela. In S. R. Jimerson, T. D. Oakland, & P. T. Ferrell (Eds.), The handbook of international school psychology (pp. 427–435). Thousand Oaks, CA: SAGE.

26.

Lions, S. & Pena, M. (2016). Reading comprehension in Latin America: Difficulties and possible interventions. In D. D. Preiss (Ed.), Child and adolescent development in Latin America. New directions for child and adolescent development (p. 71–84). San Francisco, CA: Wiley.

27.

(2006) An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification 30: 598–617. doi: 10.1177/0145445504272974.

28.

Manolov

Onghena

(2017) Analyzing data from single-case alternating treatments designs. Psychological Methods 22: 1–25. doi: 10.1037/met0000133.

29.

Manolov, R., Solanas, A., & Leiva, D. (2010). Comparing ‘visual’ effect size indices for single-case designs. Methodology, 6, 49–58. doi: 10.1027/1614-2241/a000006.

30.

Morgan

P. L.

Sideridis

G. D.

(2006) Contrasting the effectiveness of fluency interventions for students with or at risk for learning disabilities: A meta analysis. Learning Disabilities Research and Practice 21: 191–210. doi: 10.1111/j.1540-5826.2006.00218.x.

31.

Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2017). PIRLS 2016: International results in reading. Retrieved from Boston College, TIMSS & PIRLS International Study Center.

32.

Nastasi

B. K.

(2017) Cultural competence for global research and development. Implications for school and educational psychology. International Journal of School and Educational Psychology 5: 207–210. doi: 10.1080/21683603.2016.1276817.

33.

National Center for Education Statistics. (2015). The nation’s report card: Reading, 2015, National results overview (NCES 2015–136). Washington, DC: US Department of Education: NAEP.

34.

O’Gorman

Shum

Halford

K. W.

Ogilvie

(2012) World trends in psychological research output and impact. International Perspectives in Psychology: Research, Practice, Consultation 1: 268–283. doi: 10.1037/a0030520.

35.

Pearson Education, Inc. (2011). AIMSweb. Bloomington, MN: Author.

36.

Piocuda

J. E.

Smyers

J. O.

Knyshev

Harris

R. J.

Rai

(2015) Trends of internationalization and collaboration in US psychology journals 1950–2010. Archives of Scientific Psychology 3: 82–92. doi: 10.1037/arc0000020.

37.

Richards-Tutor

Baker

Gersten

Baker

Smith

(2016) The effectiveness of reading interventions for English learners: A research synthesis. Exceptional Children 82: 144–169. doi: 10.1177/0014402915585483.

38.

Ross

S. G.

Begeny

J. C.

(2015) An examination of treatment intensity with an oral reading fluency intervention: Do intervention duration and student-teacher instructional ratios impact intervention effectiveness? Journal of Behavioral Education 24: 11–32. doi: 10.1007/s10864-014-9202-z.

39.

Ross

S. G.

Begeny

J. C.

(2014) Single-case effect size calculation: Comparing regression and non-parametric approaches across previously published reading intervention data sets. Journal of School Psychology 52: 419–431. doi: 10.1016/j.jsp.2014.06.003.

40.

Seethaler

Fuchs

(2005) A Drop in the bucket: Randomized controlled trials testing reading and math interventions. Learning Disabilities Research and Practice 20: 98–102. doi: 10.1111/j.1540-5826.2005.00125.x.

41.

Soto, C. M., Casapia, M. D., Ponce, L. Z., & Morales, L. B. (2007). School psychology in Peru. In S. R. Jimerson, T. D. Oakland, & P. T. Ferrell (Eds.), The handbook of international school psychology (pp. 299–307). Thousand Oaks, CA: SAGE.

42.

Strein

Cramer

Lawser

(2003) School psychology research and scholarship. School Psychology International 24: 421–436. doi: 10.1177/01430343030244005.

43.

Therrien

W. J.

(2004) Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education 25: 252–261. doi: 10.1177/07419325040250040801.

44.

van de Vijver

F. J. R.

(2013) Contributions of internationalization to psychology: Toward a global and inclusive discipline. American Psychologist 68: 761–770. doi: 10.1037/a0033762.

45.

Vaughn

Linan-Thompson

Kouzekanani

Pedrotty Bryant

Dickson

Blozis

(2003) Reading instruction grouping for students with reading difficulties. Remedial and Special Education 24: 301–315. doi: 10.1177/07419325030240050501.

46.

Villarreal

Castro

M. J.

Umana

Sullivan

J. R.

(2017) Characteristics of intervention research in school psychology journals: 2010–2014. Psychology in the Schools 54: 548–559. doi: 10.1002/pits.22012.