Effectiveness of Comprehension Strategies Mobile App (COSMA) on Reading Comprehension Performances of Students With Learning Disabilities

Abstract

The aim of this study is to investigate the effectiveness of the Comprehension Strategies Mobile App (COSMA) on the reading comprehension performance of students with learning disabilities. Four students with learning disabilities (three boys, one girl) participated in the study. The experimental process of the study was carried out according to the multiple baseline across participants model of single-subject experimental designs. Reading comprehension performance of students was assessed with multiple-choice tests and retells. Students’ use of strategies was evaluated with on-task metacognitive interviews. Findings showed that COSMA has a positive impact on students’ reading comprehension performances. It was determined that the students’ motivation to use the mobile app was high, the opinions of families and teachers of the software were positive and they observed an increase in students’ reading skills. Findings are discussed further.

Keywords

specific learning disability exceptionality mobile devices technology perspectives reading content/curriculum area apps

Reading is the key component of a student’s school life. It is essential for academic success (O’Connor & Vadasy, 2011). Many students are faced with difficulties in other academic areas because of poor reading skills. It is very likely for some of those students to be diagnosed with learning disabilities (American Psychiatric Association DSM-V, 2013).

The National Joint Committee on Learning Disabilities (NJCLD, 1994) defined learning disability as a disorder including significant difficulties in speaking, reading, writing, reasoning, acquiring and using numerical skills. It is known that individuals with learning disabilities have problems in one or more psychological processes that are essential for understanding and using written and spoken language (IDEA, 2004). These psychological processes can be listed as attention, visual and auditory processing, memory and processing speed. Individuals with learning disabilities may show different developmental characteristics due to their limitations in these areas (Bender, 2008). As a result of those deficiencies they often experience difficulties on tasks related with reading.

Reading comprehension is the final goal of reading instruction. To achieve this goal, students are expected to show adequate reading fluency. Because of the rapid naming, auditory and visual processing problems, students with learning disabilities show decoding errors and poor reading fluency performances (Speece & Ritchey, 2005), and these problems affect their comprehension. However, studies have shown that even if they have adequate fluency, some students still have difficulties in extracting meaning from texts (Hulme & Snowling, 2011). Reading comprehension is related with use of comprehension strategies as well as fluency and decoding (Adlof et al., 2010; Anastasiou & Griva, 2009). Reading comprehension strategies are defined as purposeful efforts to extract meaning from the text and create relations between information units (Afflerbach et al., 2008). Students with learning disabilities show limitations in the use of comprehension strategies. This affects their reading comprehension performances (Gersten et al., 2001). The reason for this is that they do not have the necessary strategies or their metacognitive skills are limited and they have problems in using appropriate strategies when needed (Malone & Mastropieri, 1992; Wong et al., 1989). Although it is important to teach students comprehension strategies, their metacognitive skills should be developed in order to generalize these strategies to different situations and use them when they need them (Berkeley & Larsen, 2018).

In the National Reading Panel (2000), evidence-based reading comprehension strategies are listed as monitoring comprehension, asking/generating questions, reviewing and summarizing the text. Duke and Pearson (2009) listed the basic strategies as predicting, determining the purpose of reading, activating the background knowledge, finding the main idea, thinking out loud, summarizing, asking questions and monitoring comprehension. Many studies have shown that strategy instruction has a positive impact on the comprehension performances of students from different class levels and disability groups (Okkinga et al., 2018; Scammacca et al., 2015).

There is a growing interest in use of technology in reading comprehension intervention studies. In considering improving reading comprehension, there are many studies that involve typically developing students (Sung et al., 2008; Wijekumar et al., 2013, 2017), students at risk (Horne, 2017; Hughes et al., 2013; Park et al., 2017; Walcott et al., 2014) and students with learning disabilities (Cullen et al., 2014; Hall et al., 2015; Jozwik & Douglas, 2017). Some studies have used assistive technology with features like “text to speech” (Wood et al., 2018). On the other hand, there are studies that have tried to integrate effective reading comprehension strategy interventions with technology. McNamara et al. (2004) combined comprehension monitoring, paraphrasing, elaboration, making predictions, making bridges, and inference strategies in a multiple-strategy instruction called Self Explanation Reading Training (SERT) and used this strategy intervention in Interactive Strategy Trainer for Active Reading and Thinking (iStart) software. Researchers conducted an intervention study with 39 middle school students and reported that iStart has a positive impact on reading comprehension performances (McNamara et al., 2006). In another study a computer program called CASTLE was developed, which includes self-questioning, comprehension monitoring strategies and concept mapping, highlighting, and graphic organizer support. It was found to be effective on participants’ reading comprehension performances (Sung et al., 2008). In a recent experimental study, researchers used a technology-mediated multicomponent reading comprehension intervention called Comprehension Circuit Training (CCT; Fogarty et al., 2017). CCT is a video-based intervention used on tablet computers and includes comprehension monitoring, text structure, finding main ideas, inferencing, summarizing and fix-up strategies. The researchers expressed that CCT had positive effects on the reading comprehension performances of students.

This Study

Comprehension Strategies Mobile App (COSMA) includes comprehension strategies found to be effective in other studies. COSMA includes activating background knowledge, prediction, paragraph shrinking, comprehension monitoring and self-instruction strategies. COSMA is an app designed for independent practice. Unlike previous studies, there are no model reading functions such as text-to-speech or voice recordings. The COSMA interface is very simple. All functions and strategies are determined so students work with the content already set for them. Thus we aimed to decrease cognitive load while they read the texts. COSMA is prepared with consideration for students who have not received any comprehension strategies instruction.

In this study, we aimed to determine the effectiveness of a multicomponent reading comprehension mobile app on reading comprehension performances of students with learning disabilities. For this purpose the following questions were raised:
Does the participants’ number of correct answers increase when they work with COSMA?

Do participants’ retell scores increase when they work with COSMA?

Does participants’ use of comprehension strategies increase when they work with COSMA?

Does the difference in performance between participants and their peers decrease when participants work with COSMA?

Method

Participants

Participants in the study are four Turkish-speaking fourth-grade students diagnosed with learning disabilities in Ankara, Turkey. For selection of participants, 13 students in seven schools were assessed. Students were selected from those who showed 90% reading accuracy (percentage of correct words read in whole text) with at least 50 correct words per minute, who answered at most four comprehension questions out of 10. Reading fluency criteria were set considering the duration of an intervention session and relationship between reading fluency and reading comprehension. Participants did not have any additional disabilities, their families were not from high-disadvantaged socioeconomic groups, and they had not been receiving intensive reading comprehension and strategy instruction. Four students were selected for the study based on these criteria. Researchers informed families about the study, and families gave written consent for their children to participate. Table 1 shows participants’ demographic information.

Table 1.
Participants’ Demographic Information.

Student Gender Age CWPM* (Rate) Ethnicity Tablet Experience

Harry Male 9 years 10 months 55 (%96) Turkish Games

Hermione Female 9 years 8 months 63 (%93) Turkish Games & Youtube

Neville Male 9 years 4 months 53 (%91) Turkish Games

Ron Male 9 years 5 months 51 (%93) Turkish Games

* Correct words per minute.

The first participant of the study, “Harry,” is a 9 years 10 months old male student, diagnosed with a learning disability in second grade. Harry showed 96% reading accuracy with 55 CWPM in the preliminary assessment. The peer selected for Harry is a 9 years 9 months old male student who showed 100% reading accuracy with 98 CWPM in preliminary assessment.

The second participant, “Hermione,” is a 9 years 8 months old female student, diagnosed with a learning disability in second grade. Hermione showed 93% reading accuracy with 63 CWPM in the preliminary assessment. The peer selected for Hermione is a 9 years 9 months old female student who showed 100% reading accuracy with 107 CWPM in preliminary assessments.

The third participant, “Neville,” is a 9 years 4 months old male student, diagnosed with a learning disability in second grade. Neville showed 91% reading accuracy with 53 CWPM in the preliminary assessment. The peer selected for Neville is a 9 years 4 months old male student who showed 100% reading accuracy with 96 CWPM in preliminary assessment.

The fourth and final participant, “Ron,” is a 9 years 5 months old male student, diagnosed with a learning disability in second grade. Ron showed 93% reading accuracy with 51 CWPM in preliminary assessment. The peer selected for Ron is a 9 years 6 months old male student who showed 99% reading accuracy with 101 CWPM in preliminary assessment.

Comparison Peers

After selection of the participants, teachers were asked to choose a peer from their classes to be evaluated as comparison data for each participant. For this they were asked to choose a peer of close chronological age, the same gender and a student who reflects the average success of the class. Four peers were assessed in the preliminary, baseline and follow-up sessions. The matched peers continued in the “business as usual” curriculum. The matched students were assessed using multiple-choice tests and retell procedure.

There was no observed ceiling effect on the scores for the experimental or control conditions.

Setting

All sessions took place in private, quiet small classrooms in participants’ schools. Tablet holders were provided to students to help them navigate tablets effortlessly in sessions.

Data Collection Tools

Multiple choice tests and retell procedures were used in this study in order to assess reading comprehension. Multiple-choice tests and retell procedures frequently used assessment methods in reading comprehension researches (Collins & Lindström, 2021). Students’ comprehension strategy use was assessed with on-task metacognitive interviews. Assessment procedures explained in detail under dependent variables section.

Independent Variable

Independent variable of the study is the comprehension strategies mobile app named COSMA. Researchers planned, designed and created the mobile app, including voice recordings, texts, tests, app structure, organization, graphics and strategies. COSMA includes descriptive texts and multiple-choice tests and retell sections for all texts. In every text COSMA presents strategies in the following order: activating background knowledge, prediction, self-instruction (attention focusing, self-monitoring, self-reinforcement) and paragraph shrinking. Students create a profile and start to use the app. On the main menu, they choose the model student’s gender from the settings section and track their own reading performances from “my summaries” and “my performance” sections. See Figure 1 for screenshots.

Figure 1.
COSMA screenshots. Top left user selection menu, top right main menu, bottom left texts, and bottom right settings.

In the text sections there are 10 different descriptive texts. Students are free to choose any of them. When they choose a text and start to study, they have to complete the whole process. The first page is for activating background knowledge. This page includes photos for every paragraph. According to text structure, photos are related to the paragraph topics. COSMA automatically tells students to look at the photos and think of what they know about the topic. On the second page is the first paragraph of the text. All texts consist of six paragraphs. The first paragraph is short and aims to catch the student’s attention. On the third page, the actual reading with strategies begins. COSMA tells students to look at the photo and try to predict the topic of the paragraph. When they answer correctly, the paragraph appears on the screen and they read it. As can be seen in Figure 2, under the paragraph there is a green button which says “Okudum”—“I have read”—in Turkish. When the student touches the button, an animated character gives models of the self-instruction strategies. We decided to use three functions of self-instruction. First, monitoring comprehension with: “Did I understand the paragraph? Do I need to reread it?” the second, focused attention with: “If I read carefully, I can comprehend better!” and the third as self-motivation: “I am doing great!” After hitting the “I have read” button, a paragraph-shrinking question appears on the screen. The paragraph—shrinking question is finding the best summarizing sentence of the paragraph out of three choices. Students must choose the correct answer to advance to the next paragraph. The next three paragraphs are constructed in the same way. The last paragraph is short, as it is a farewell to the reader. On this page, the animated character models a self-instruction strategy as: “Now it’s time to answer questions, I can do it!” Students answer 10 multiple-choice questions on the screen. Questions appear on the screen one at a time. On the last page, the animated character asks the students to retell the text. The students push the microphone button and if they try to close it before 3 min, COSMA prompts them to say more about the text. When they complete the retell section, COSMA presents the result of the test.

Figure 2.
COSMA text reading screens. Top left activating background knowledge with visuals and strategy hint, top right prediction question, bottom left reading paragraph, and bottom right choosing best summarizing sentence for paragraph.

Dependent Variables

Multiple choice test scores

The researchers wrote 30 texts for this study. All texts are in descriptive structure; they consist of six paragraphs and include at least one possible unknown target word for students. Texts were evaluated for readability to confirm that they are on the fourth-grade level. All texts are between 300 and 330 words in length and include 19 specific information units. In choosing the topics, fourth-grade science textbooks were taken into consideration. Expert consultations were gathered for every single text, including the multiple-choice test. Pilot studies took place before the study to see if multiple-choice tests are capable of showing the difference between good readers and poor readers. Participants’ reading comprehension levels were assessed with multiple-choice tests (four choices) and retells. Tests include ten questions presented in the same order on all tests. Question types are: general question (what is the text is about), target word, sequential information (what happens before, after, first, last…), cause and effect, summarizing, headline, inference and the aim of the writer. Every correct answer counts as 1 point, the maximum score being 10 and the minimum score 0.

Retell scores

Participants’ oral retellings were evaluated with the retell scoring rubric. The rubric consists of four items in total: Number of information units (how much specific information students retell), accuracy of retell (retelling information correctly), sentence structure and organization of the retell. The lowest score that can be obtained from the retell scoring rubric is 4, the highest score is 16. See Appendix for retell scoring rubric. In pilot studies students were asked to retell the text they read after answering the multiple-choice tests. Results were consistent with the multiple-choice test scores and it was found that both assessments are reliable.

Comprehension strategy use

Participants’ use of reading comprehension strategies were assessed with “on task metacognitive interviews.” Researchers combined think-aloud protocol and metacognitive interviews and created a new modified method to assess strategy use. The reason behind this was the observations and findings from the pilot studies. In pilot studies, metacognitive interviews and think-aloud assessments were conducted with several students. In metacognitive interviews students’ answers to questions were very limited. Students found the think-aloud procedure very hard and got confused. In one task, metacognitive interviews, researchers asked students to think aloud about how they think, what they do before-during-after reading to understand better. The procedure explained to participants as follows: “I want you to read a text and understand it very well and learn what it is about. You should understand it very thoroughly, ok? I wonder how you think, what comes to your mind when you read, what you do to understand better. This is our thinker puppy, he says ideas aloud when they come to his mind. Whenever you see him remember to say what you do to comprehend better. Let’s try….” Additionally researchers asked participants metacognitive interview questions about how they think, what they do before-while-after reading. For example: “What came to your mind when I said, ‘Understand it very well?’ What did you think when you read the title of the text? Is there any word that you don’t know so far? What do you do when you see one? What do you do to remember better after you finish reading?” Strategy use was assessed twice on baseline and follow-up phases. Texts used in these assessments are above participants’ grade level to make them use comprehension strategies.

Experimental Design

Procedure

Baseline

Before the intervention, participants’ reading comprehension performances and strategy use were assessed in baseline sessions. They were given print-outs of texts and tests. Participants were asked to read aloud first and then read silently. After silent reading, participants completed the tests and they were asked to retell the text. Participants’ strategy use was assessed twice with on-task metacognitive interviews on different days. All sessions were recorded.

Intervention

The intervention process began with the first participant when he showed consistent performance on baseline sessions. The first introductory sessions students were shown how to use COSMA. Participants studied with the app independently in intervention sessions and the researcher stayed in the room for observations. A tablet computer was placed on the table on a holder and iPad’s guided access feature activated. This feature allows students to use only the app on screen and prevents possible problems such as the student pressing the power button accidentally. All intervention sessions started with an explanation by the researcher: “I want you to study with our app. You should use the features to comprehend the text better. We have plenty of time, so you can listen to them twice or more if you wish. After you have finished reading, you will answer questions and retell what you have read. I will stay here so that if you need anything, you can tell me. I would love to listen to your reading, please read aloud. Let’s begin.” Students read the text on app with the strategy models from animated characters, prediction before reading the paragraph and paragraph-shrinking questions. As described in the independent variable, they received the models in the same order and they had to engage with all buttons to advance to the next page. When they finished reading they answered multiple-choice questions on the tablet. After completing the multiple-choice test, the app asked them to press the record button and retell what they have learned. COSMA collected the comprehension test scores and recorded the retells. The researcher recorded the whole session for procedural fidelity. Intervention data were gathered on screen with COSMA because we wanted to reduce the teachers’ role, create an independent practice opportunity and show its effects.

Intervention sessions took place for 3 weeks. Each participant attended a total of 10 sessions. The average intervention session was 25 min long.

Follow-Up

After the 1st, 2nd and 3rd weeks of the last intervention session, follow-up assessments were done for all participants. Texts and multiple-choice tests were given to students in printed form. Participants’ retell and strategy use were assessed as explained in baseline.

Procedural Fidelity

Researchers had prepared procedural fidelity forms in order to assess whether the experimental process had been done as planned. Procedural fidelity is about the behaviors of researchers in all phases of the study (Ledford & Gast, 2014). Four independent researchers were asked to choose records randomly from all phases (baseline, intervention, follow-up), listen to at least 33% of recordings, then complete procedural fidelity forms. The number of consensus was divided by the total of consensus and disagreement and multiplied by 100 (House, House & Campbell, 1981). Procedural fidelity of this study was calculated at 99%.

Interobserver Reliability

Data gathered in all sessions were recorded and filed. After the data collection process was completed for all participants, texts, tests and retell records were given to four independent researchers. For an objective evaluation, no information was given about the phase and participants. All four observers scored all of the participants. Kendall’s Tau-b coefficient of consistency calculated and the inter-observer reliability of the research was determined as 92%.

Social Validity

Participants’, teachers’ and parents’ views of COSMA and use of technology in education were asked for in semi-structured interviews after the experimental process. Three different sets of questions were prepared and participants’ ideas, views, and expectations from educational technology and advice on new educational mobile apps were requested. The participants’ answers were examined with descriptive analysis.

Data Analysis

Visual analyses were used to evaluate the results of this intervention and comprehension test and retell scores shown in graphs. Findings were examined according to level, trend, variability, immediacy of the effect, percentage of non-overlapping data (PND) and consistency of data patterns across similar phases. Descriptive analysis was used to examine results of on-task metacognitive interviews to show participants’ strategy use.

Results

All data gathered in this study are shown in Table 2. For visual analysis of the study, Figure 3 and 4 were drawn.

Table 2.
Comprehension Test and Retell Scores.

Reading Comprehension Test Retell

Student Number of Sessions X SS Range X SS Range

Harry Baseline 5 3.80 .44 3–4 5.40 .54 5–6

Intervention 10 7.4 1.26 6–10 9.7 1.25 8–12

Follow-up 3 6.66 .57 6–7 8.33 .57 8–9

Hermione Baseline 6 3.33 .51 3–4 5.66 1.03 5–7

Intervention 10 7.3 .82 6–8 9.8 1.39 7–12

Follow-up 3 7.66 .57 7–8 8.33 1.52 7–10

Neville Baseline 7 3 .57 2–4 4.85 .89 4–6

Intervention 10 6.5 1.58 4–9 7.2 1.22 5–9

Follow-up 3 7 1.0 6–8 7.33 .57 7–8

Ron Baseline 8 3.5 .53 3–4 5.12 .83 4–6

Intervention 10 8.0 .94 7–10 10.5 1.17 8–12

Follow-up 3 7.66 .57 6–8 8.66 .57 8–9

Harry

Harry is the first participant of the study. He showed average 3.8 (range: 3–4) multiple-choice test scores and average 5.4 (range: 5–6) retell scores in five baseline sessions. In the COSMA intervention sessions, his test score average is 7.4 (range: 6–10) and average retell score is 9.7 (range: 8–12). In the follow-up sessions Harry protected his gains with a slight decrease, showing an average of 6.66 (range: 6–7) and an average of 8.33 (range: 8–9) in retell scores. In Figure 3 and 4 it can be seen that in baseline sessions both his comprehension test and retell scores are lower than his peer selected from his class. After intervention, Harry’s test and retell performance improved and got closer to those of his peer. PND for Harry’s graph is 100% for both test and retell scores. On the other hand, Harry showed more strategy use in post-test on-task strategy use interviews. He used self-instruction strategies in the post-test.

Hermione

Hermione is the second participant of the study. She showed an average of 3.44 (range: 3–4) on multiple-choice test scores and an average of 5.66 (range: 5–7) retell scores in six baseline sessions. In the COSMA intervention sessions, her test scores average is 7.3 (range: 6–8) and her average retell score is 9.8 (range: 7–12). In follow-up sessions Hermione protected her gains, showing an average of 7.66 (range: 7–8) test score and an average of 8.33 (range: 7–10) on her retell score. In Figure 3 and 4 it can be seen that in the baseline sessions, both her test and retell scores are lower than her peer selected from her class. After intervention, Hermione’s test and retell performance improved and she showed the same performance as her peer on the multiple-choice test and she outscored her peer on the second follow-up evaluation of retell. PND for Hermione’s graph is 100% for both test and retell scores. As can be seen in Table 3, Hermione showed more strategy use in post-test on task strategy use interviews. She used repeated reading, self-instruction, thinking about the topic and questioning strategies in post-tests (see Table 3).

Table 3.
Strategies Used by Participants.

Sessions Harry Hermione Neville Ron

Pretest-1 Repeated Reading — Repeated Reading Repeated Reading

Pretest-2 Repeated Reading Repeated Reading Repeated Reading Repeated Reading

Posttest-1 Repeated Reading, Self-Instruction Repeated Reading, Thinking on Topic, Questioning, Self-Instruction Repeated Reading Repeated Reading, Retelling and Verbal Summarization

Posttest-2 Repeated Reading Repeated Reading, Thinking on Topic, Questioning, Self-Instruction Repeated Reading Repeated Reading, Thinking on Topic, Prediction and Checking Predictions, Retelling

Neville

Neville is the third participant of the study. He showed an average of 3 (range: 2–4) on the multiple-choice test score and an average of 4.85 (range: 4–6) on the retell score in seven baseline sessions. In the COSMA intervention sessions, his average test score is 6.5 (range: 4–9) and average retell score is 7.2 (range: 5–9). In follow-up sessions Neville protected his gains, showing an average of 7 (range: 6–8) test score and an average 7.33 (range: 7–8) on retell scores. In Figure 3 and 4 it can be seen that in baseline sessions both his test and retell scores are lower than his peer selected from his class. After intervention, Neville’s comprehension test and retell performance improved but remained below those of his peer. PND for Neville’s graph is 100% for both comprehension test and retell scores. Neville did not use strategies except on the repeated reading in the task strategy use interviews.

Figure 3.
Multiple-choice test scores graph of participants and peers.

Figure 4.
Retell scores graph of participants and peers.

Ron

Ron is the fourth participant of the study. He showed an average of 3.5 (range: 3–4) on the multiple-choice test score and an average of 5.12 (range: 4–6) retell score in eight baseline sessions. In the COSMA intervention sessions, his average test score is 8 (range: 7–10) and average retell score is 10.5 (range: 8–12). In the follow-up sessions Ron protected his gains, showing an average of 7.66 (range: 6–8) test and an average of 8.66 (range: 8–9) on retell scores. In Figure 3 and 4 it can be seen that in the baseline sessions, both his comprehension test and retell scores are lower than his peer selected from his class. After intervention, Ron’s performance of comprehension improved and got closer to that of his peer. PND for Ron’s graph is 100% for both comprehension test and retell scores. Ron had used strategies from COSMA in the post-test use of strategy assessments.

Social Validity

Semi-structured interviews with participants, their teachers and parents took place after data collection process. Students’ thoughts about using COSMA were positive. They said it was fun to work with COSMA, and all participants indicated that they would like to use it again. A participant responded “It was beautiful, my teacher, I liked it, [it] was fun, I would read again.” Participants emphasized different features of COSMA such as graphics and photos, animated characters, listening to their own retellings, feedback of test scores and “My Performance” section. Teachers’ and parents’ responses are generally positive and they indicated that COSMA is better than expected. Teachers and parents both expressed concerns about using tablets in education, but applications like COSMA may be useful for students. A teacher responded “I like this app, I would use it in my class, looks good and I see kids are eager to use it, also creating a profile is good because you know we don’t have tablets for everyone. With one tablet many students can use [it] and I can follow the process.”

Discussion

In this study, the effectiveness of the Comprehension Strategies Mobile App (COSMA) on reading comprehension performance of students with learning disabilities was examined. Findings show that participants’ reading comprehension performances increased meaningfully. This result is consistent with the studies using multiple component strategy interventions via technology (Hall et al., 2015; Jackson et al., 2012; McNamara et al., 2006; Sung et al., 2008). Furthermore, similar to other studies it can be seen that participants used the mobile app easily and showed high motivation in intervention sessions (Curtis, 2015; Wijekumar et al., 2013).

It is not possible for us to say which feature of COSMA has the most effect on comprehension performances, but we can say that multicomponent strategy instruction with COSMA has positive effects. Especially important is that participants were able to sustain their gains on follow-up assessments and showed similar performances to their peers. Although it is unusual to have peers for comparison in single-subject studies, we wanted to support our research in an innovative way. Even though participants show significant reading comprehension development in single-subject studies, it may be seen as a result of employing biased evaluation. In the current study, peers’ results from the pre- and post-assessments prove that our multiple-choice tests and retell procedure give reliable results. Typical developing peers did not have top scores from multiple-choice tests and retell assessments. This indicates that there is no ceiling effect for peers in this study. The second participant, Hermione, received exactly the same test scores as her peer. This is an important result and should be considered together with the intervention duration. All participants used COSMA independently for 10 sessions in 3 weeks. These findings support the idea of using technology to improve students’ reading comprehension skills outside of as well as within schools. COSMA can be used as a tool for supporting class activities at home. All participants said that they would like to use COSMA again for studying. Although this research was completed in 2019, recent events such as Coronavirus (COVID-19) worldwide has shown the need for educational technologies such as COSMA.

We examined the test scores of the children to understand whether there was a consistent pattern of wrong answers according to question types. As explained in Method section, multiple-choice tests have different types of questions and some of them can be considered harder to answer. However, there is no consistent situation. All participants gave different wrong answers to different questions and when it is calculated, there is no obvious distinction. This can be seen as a positive outcome. Fırat and Ergül (2020) found that Turkish-speaking students with learning disabilities show greater difficulties on inferential questions than on literal questions.

Retell assessments were done to evaluate reading comprehension in this study. It was seen that the baseline levels of the participants were quite low. All intervention and follow-up sessions retell scores are higher than the baselines and participants performed better. This result is also seen in different studies in which retell is taken as an independent variable. (Cullen et al., 2014; Gonzalez, 2014; Manset-Williamson et al., 2008). Because of their difficulties in working memory and expressive language skills (Snowling & Melby-Lervåg, 2016), retell tasks are very demanding for students with learning disabilities. For a good retell, students need to recall, organize and sequence the information they have learned. Although intervention did not involve any systematic instruction for retell, participants’ scores increased. They performed better and the gap between them and their peers decreased. COSMA includes a modified version of paragraph-shrinking strategy. This strategy asks students to summarize the paragraph they read in not more than 10 words. After reading a paragraph on COSMA, participants were asked to choose the best summarizing sentence. While they were answering this question, the researcher observed and noted that all participants read the sentences and reviewed the paragraphs quickly. It can be seen that participants sustain their gains in follow-up sessions. We believe that these findings are noteworthy.

Table 2, Figure 3 and 4 indicate there was a functional relationship between use of COSMA and dependent variables. Variability of data is low and baseline levels of participants were consistent. It is clear that all participants’ scores increased from the first intervention session and this reflects the immediacy effect of IV on DV. Percentage of non-overlapping data (PND) is calculated as 100% for all participants. Consistency of data patterns across similar phases can be seen in Figure 3 and 4. In this context it can be said that COSMA has an effect on dependent variables and the study met the single-subject studies criteria for demonstrating evidence of effect of an independent variable (Kratochwill et al., 2013). Additionally, procedural fidelity and inter-observer reliability show that the IV was implemented as planned and the results are reliable.

Strategy use was assessed with a modified procedure which combines think-aloud and metacognitive interviews. We called this on-task metacognitive interviews. Findings from those interviews are given in Table 3. Participants found think-aloud very hard and only Hermione and Ron expressed how they thought while reading in follow-up assessments. In baseline, participants’ answer to questions on what they do or would do to understand better is only repeated reading. In follow-up assessments Hermione and Ron used different strategies while reading and expressed them aloud. They also answered questions and said what they would do. Harry and Neville did not use any strategies except on posttest 1, when Harry used self-instruction. Hermione and Ron used the strategies (Thinking about the topic, questioning, self-instruction, prediction and checking predictions, retelling) which COSMA repeatedly modeled to them. A typical strategy intervention involves systematic instruction of strategy and the teacher explains the steps of the strategy and uses mnemonics and scaffoldings. COSMA does not have any of these elements. It has animated characters embedded in the pages and reminds the students of reading comprehension strategies. It seems that only two participants used different strategies in the posttests, but this is also a good result.

While thinking about these findings, one thing we should keep in mind is the difficulty of assessing children’s strategy use. First we planned to use think-aloud procedures to assess strategy use, but in the pilot study we saw that for students expressing how they think while reading is a very hard task. Also, metacognitive interviews have their deficits. For example, students try to give correct answers. Because of that, we used on-task metacognitive interviews as explained in the methods section and tried to use the advantages of both procedures. A participant’s comment about think-aloud strategy use assessment will help to clarify this matter. Neville told the researcher: “Sir, I am doing, aaa…I am reading yes I try to think, aaa…I get confused when I say something, something to this side, aaa that is, two things at the same time. Teacher, I read it first, then my teacher, let me answer after it.” Considering the working memory and verbal language problems experienced by students with learning disabilities, this answer becomes more understandable (Bock & Levelt, 1994).

In every intervention session the researcher took observation notes according to applied behavior analysis anecdotal record principles. According to the researcher’s notes, participants were highly engaged with COSMA and they used every feature of the app. For example, all participants carefully answered the prediction and summarizing questions about paragraphs. When they make the wrong choice, the program gives feedback and the choice turns red, which helps them to find the right answer. When they choose the right answer, a green tick comes on screen. It was seen that participants made comments like “green on the first! Yay! All green at once! This time I won’t get any red.” Furthermore, they listened to the strategy models two or three times, and counted their stars on the menu. In the activating background knowledge phase, they examined the photos carefully and read the paragraphs twice to find the right choice. These behaviors can be considered as indicators of students’ motivation toward COSMA.

All participants, their parents and their teachers were asked their views of COSMA and educational technologies in the social validity interviews. Consistent with the researcher’s observations, all participants expressed that they loved to work with the tablet and they loved the animated characters, pictures, voice recording and test score feedback features of COSMA. Participants were asked for their suggestions for making COSMA better. Some of the answers are remarkable. Hermione said, “Mmm I think there should be more character options, and there should be more children, I would like to name them, change their clothes with my test stars.” This is should be considered important information about user experience, because she is asking for more individualization. Another answer came from Ron, who said, “Put some games, aa…if I read well, I can open the games and I can play games with my test stars.” This suggestion points to a conditional prize for students and it can be useful for making COSMA more enjoyable as well as allowing it to be used for a longer period of time. Parents and teachers expressed that they have seen positive changes in students’ reading comprehension performances. Although they had serious concerns about the use of educational technology, after they had examined COSMA they expressed that programs like COSMA would be good for students. In particular, teachers said they hope to see many more apps and software like COSMA for smartboards too.

Directions for Future Research

Application developing process for special education requires a detailed planning and a variety of different skills. Because of this, multidisciplinary work is obligatory to achieve a good result. In this research we had support from a computer engineer and a graphic designer. In this study, researchers created the content and decided on every detail carefully. Then with the help of professionals, COSMA was completed. It is seen that students are extremely interested in visual aspects and the more professional the program looks, the more likely it is that they will engage with it. In future research, this important view should be considered.

The COVID-19 pandemic has shown the need for such applications as COSMA. Independent practice and instruction via apps should be investigated further. It is very clear that in every country, students need this kind of support. Students with learning disabilities, especially, need much more intensive instruction. Because of that, structured systematic programs like COSMA would be helpful. Reading comprehension and reading fluency are both important problems for students with learning disabilities. A mobile app which includes both aspects with a holistic approach would be beneficial for all students. The effectiveness of this balanced reading intervention should be examined. COSMA is an app which students can practice independently without teachers, but in our study for experimental control we used guided access and the researcher stayed with students. Distance education in quarantine periods has shown the need for independent study materials with guidance. A new program which includes strategy instruction, reading fluency and reading comprehension practices, math problems, etc. and collects data online with a cloud would be a good research topic. Such a program would show the effectiveness better with system data support (time users spend on app, features actively used, etc.).

Considering the variety of needs of students with special needs, instructional technologies should be much more adaptable to different users. In this study we worked with fourth-graders, but reading comprehension is an issue for older and younger students too. There are other comprehension strategies that we can teach to students. Individualization is very important in special education. If the educational technologies included different levels of practice (texts, sentence, word drills) with different strategy instruction density options (graphic organizers, summary writing, monitoring comprehension) for users, more students would benefit from them.

Implications for Practice

Reading comprehension strategy instruction is a widely used evidence-based method. Teachers should use it with or without technology. The results from this study present evidence that tablet devices can be used for a good cause. Teachers should examine the contents of apps from AppStore and Google Play. There are many good examples of supporting students’ performance outside of school, and in this they can collaborate with parents.

Limitations of the Study

This study conducted with fourth-grade students diagnosed with learning disabilities and COSMA includes only descriptive texts. Additionally, before intervention researchers gave a demonstration to participants of how to use COSMA and asked them to try the app. All participants were able to use COSMA without any problem. However, an assessment with a checklist of app-using steps would be good to prove that participants used COSMA as planned during intervention sessions.

Student	Gender	Age	CWPM* (Rate)	Ethnicity	Tablet Experience
Harry	Male	9 years 10 months	55 (%96)	Turkish	Games
Hermione	Female	9 years 8 months	63 (%93)	Turkish	Games & Youtube
Neville	Male	9 years 4 months	53 (%91)	Turkish	Games
Ron	Male	9 years 5 months	51 (%93)	Turkish	Games

			Reading Comprehension Test	Retell
Harry	Baseline	5	3.80	.44	3–4	5.40	.54	5–6
Intervention	10	7.4	1.26	6–10	9.7	1.25	8–12
Follow-up	3	6.66	.57	6–7	8.33	.57	8–9
Hermione	Baseline	6	3.33	.51	3–4	5.66	1.03	5–7
Intervention	10	7.3	.82	6–8	9.8	1.39	7–12
Follow-up	3	7.66	.57	7–8	8.33	1.52	7–10
Neville	Baseline	7	3	.57	2–4	4.85	.89	4–6
Intervention	10	6.5	1.58	4–9	7.2	1.22	5–9
Follow-up	3	7	1.0	6–8	7.33	.57	7–8
Ron	Baseline	8	3.5	.53	3–4	5.12	.83	4–6
Intervention	10	8.0	.94	7–10	10.5	1.17	8–12
Follow-up	3	7.66	.57	6–8	8.66	.57	8–9

Sessions	Harry	Hermione	Neville	Ron
Pretest-1	Repeated Reading	—	Repeated Reading	Repeated Reading
Pretest-2	Repeated Reading	Repeated Reading	Repeated Reading	Repeated Reading
Posttest-1	Repeated Reading, Self-Instruction	Repeated Reading, Thinking on Topic, Questioning, Self-Instruction	Repeated Reading	Repeated Reading, Retelling and Verbal Summarization
Posttest-2	Repeated Reading	Repeated Reading, Thinking on Topic, Questioning, Self-Instruction	Repeated Reading	Repeated Reading, Thinking on Topic, Prediction and Checking Predictions, Retelling

Footnotes

Appendix

Acknowledgments

We would like to thank to all students and their teachers and parents for their collaboration in the study. The paper presented here is informed by a dissertation written by the first author.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ahmet Bilal Özbek

References

Adlof

S. M.

Catts

H. W.

Lee

(2010). Kindergarten predictors of second versus eighth grade reading comprehension. Journal of Learning Disabilities, 43, 332–345. https://doi.org/10.1177/002219410369067

Afflerbach

Pearson

P. D.

Paris

S. G.

(2008). Clarifying differences between reading skills and reading strategies. The Reading Teacher, 61(5), 364–373. https://doi.org/10.1598/RT.61.5.1

American Psychiatric Association. (2013). DSM 5: Diagnostic and statistical manual of mental disorders (5th ed.). https://www.psychiatry.org/psychiatrists/practice/dsm

Anastasiou

Griva

(2009). Awareness of reading strategy use and reading comprehension among poor and good readers. Elementary Education Online, 8(2) 283–297.

Bender

W. N.

(2008). Differentiating instruction for students with learning disabilities (2nd ed.). Corwin Press.

Berkeley

Larsen

(2018). Fostering self-regulation of students with learning disabilities: Insights from 30 years of reading comprehension intervention research. Learning Disabilities Research & Practice, 33(2), 75–86. https://doi.org/10.1111/ldrp.12165

Bock

J. K.

Levelt

W. J.

(1994). Handbook of psycholinguistics. In Gernsbacher

M. A.

(Ed.), Language production: Grammatical encoding (pp. 741–779). Academic Press.

Collins

A. A.

Lindström

E. R.

(2021). Making sense of reading comprehension assessments: Guidance for evaluating student performance. Intervention in School and Clinic. https://doi.org/10.1177/1053451221994806

Cullen

J. M.

Alber-Morgan

S. R.

Schnell

S. T.

Wheaton

J. E.

(2014). Improving reading skills of students with disabilities using headsprout comprehension. Remedial and Special Education, 35(6), 356–365. https://doi.org/10.1177/0741932514534075

10.

Curtis

S. D.

(2015). Perspectives of special educators on the use of tablets and handhelds [Unpulished doctoral dissertation. University of Minnesota].

11.

Duke

N. K.

Pearson

P. D.

(2009). Effective practices for developing reading comprehension. The Journal of Education, 189(1/2), 107–122. https://doi.org/10.1177/0022057409189001-208

12.

Fırat

Ergül

(2020). The effect of TWA strategy instruction on students with learning disabilities developing cognitive and metacognitive strategies. Kastamonu Education Journal, 28(3), 1390–1406. https://doi:10.24106/kefdergi.4025

13.

Fogarty

Clemens

Simmons

Anderson

Davis

Smith

Wang

Kwok

Simmons

L. E.

Oslund

(2017). Impact of a technology-mediated reading intervention on adolescents’ reading comprehension. Journal of Research on Educational Effectiveness, 10(2), 326–353. https://doi.org/10.1080/19345747.2016.1227412

14.

Gersten

Fuchs

L. S.

Williams

J. P.

Baker

(2001). Teaching reading comprehension strategies to students with learning disabilities: A review of research. Review of Educational Research, 71(2), 279–320. https://doi.org/10.3102/00346543071002279

15.

Gonzalez

(2014). The effect of embedded text-to-speech and vocabulary eBook scaffolds on the comprehension of students with reading disabilities. International Journal of Special Education, 29(3), 111–125.

16.

Hall

T. E.

Cohen

Vue

Ganley

(2015). Addressing learning disabilities with UDL and technology: Strategic reader. Learning Disability Quarterly, 38(2), 72–83. https://doi.org/10.1177/0731948714544375

17.

Horne

J. K.

(2017). Reading comprehension: A computerized intervention with primary-age poor readers. Dyslexia, 23(2), 119–140. https://doi.org/10.1002/dys.1552

18.

House

A. E.

House

B. J.

Campbell

M. B.

(1981). Measures of interobserver agreement: Calculation formulas and distribution effects. Journal of Behavioral Assessment, 3(1), 37–57. https://doi.org/10.1007/BF01321350

19.

Hughes

J. A.

Phillips

Reed

(2013). Brief exposure to a self-paced computer-based reading programme and how it impacts reading ability and behaviour problems. PLoS One, 8(11), e77867. https://doi.org/10.1371/journal.pone.0077867

20.

Hulme

Snowling

M. J.

(2011). Children’s reading comprehension difficulties: Nature, causes, and treatments. Current Directions in Psychological Science, 20(3), 139–142. https://doi.org/10.1177/0963721411408673

21.

Individuals with Disabilities Education Improvement Act (IDEA). (2004). P.L. 108-446, 20 U.S.C.

22.

Jackson

G. T.

Dempsey

K. B.

McNamara

D. S.

(2012). Game-based practice in a reading strategy tutoring system: Showdown in iSTART-ME. In Reinders

(Ed.), Digital games in language learning and teaching (pp. 115–138). Palgrave Macmillan.

23.

Jozwik

S. L.

Douglas

K. H.

(2017). Effects of a technology-assisted reading comprehension intervention for English learners with learning disabilities. Reading Horizons, 56(2), 4. https://scholarworks.wmich.edu/reading_horizons/vol56/iss2/4

24.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26–38. https://doi.org/10.1177/0741932512452794

25.

Ledford

J. R.

Gast

D. L.

(2014). Measuring procedural fidelity in behavioural research. Neuropsychological Rehabilitation: An International Journal, 24(3–4), 332–348, https://doi.org/10.1080/09602011.2013.861352

26.

Malone

Mastropieri

M. A.

(1992). Reading comprehension instruction: Summarization and self-monitoring training for students with learning disabilities. Exceptional Children, 21, 270–279. https://doi.org/10.1177/001440299105800309

27.

Manset-Williamson

Dunn

Hinshaw

Nelson

J. M.

(2008). The impact of self-questioning strategy use on the text-reader assisted comprehension of students with reading disabilities. International Journal of Special Education, 23, 123–135.

28.

McNamara

D. S.

Levinstein

I. B.

Boonthum

(2004). iSTART: Interactive strategy training for active reading and thinking. Behavior Research Methods, Instruments, & Computers, 36(2), 222–233. https://doi.org/10.3758/BF03195567

29.

McNamara

D. S.

O’Reilly

T. P.

Best

R. M.

Ozuru

(2006). Improving adolescent students’ reading comprehension with iSTART. Journal of Educational Computing Research, 34(2), 147–171. https://doi.org/10.2190/1RU5-HDTJ-A5C8-JVWE

30.

National Joint Committee on Learning Disabilities. (1994). Secondary to postsecondary education transition planning for students with learning disabilities. In Collective perspectives on issues affecting learning disabilities: Position papers and statements (pp. 97–104). PRO-ED.

31.

O’Connor

R. E.

Vadasy

P. F.

(2011). Handbook of reading interventions. Guilford.

32.

Okkinga

van Steensel

van Gelderen

A. J. S.

van Schooten

Sleegers

P. J. C.

Arends

L. R.

(2018). Effectiveness of reading-strategy interventions in whole classrooms: A meta-analysis. Educational Psychology Review, 30(4), 1215–1239. https://doi.org/10.1007/s10648-018-9445-7

33.

Park

H. J.

Takahashi

Roberts

K. D.

Delise

(2017). Effects of text-to-speech software use on the reading proficiency of high school struggling readers. Assistive Technology, 29(3), 146–152. https://doi.org/10.1080/10400435.2016.1171808

34.

Scammacca

N. K.

Roberts

Vaughn

Stuebing

K. K.

(2015). A meta-analysis of interventions for struggling readers in grades 4–12: 1980–2011. Journal of Learning Disabilities, 48(4), 369–390. https://doi.org/10.1177/0022219413504995

35.

Snowling

M. J.

Melby-Lervåg

(2016). Oral language deficits in familial dyslexia: A meta-analysis and review. Psychological Bulletin, 142(5), 498–545. https://dx-doi-org.web.bisu.edu.cn/10.1037/bul0000037

36.

Speece

D. L.

Ritchey

K. D.

(2005). A longitudinal study of the development of oral reading fluency in young children at risk for reading failure. Journal of Learning Disabilities, 38(5), 387–399. https://doi.org/10.1177/00222194050380050201

37.

Sung

Y. T.

Chang

K. E.

Huang

J. S.

(2008). Improving children’s reading comprehension and use of strategies through computer-based strategy training. Computers in Human Behavior, 24(4), 1552–1571. https://doi.org/10.1016/j.chb.2007.05.009

38.

Walcott

C. M.

Marett

Hessel

A. B.

(2014). Effectiveness of a computer-assisted intervention for young children with attention and reading problems. Journal of Applied School Psychology, 30(2), 83–106. https://doi.org/10.1080/15377903.2013.874389

39.

Wijekumar

K. K.

Meyer

B. J.

Lei

(2013). High-fidelity implementation of web-based intelligent tutoring system improves fourth and fifth graders content area reading comprehension. Computers & Education, 68, 366–379. https://doi.org/10.1016/j.Compedu.2013.05.021

40.

Wijekumar

K. K.

Meyer

B. J.

Lei

(2017). Web-based text structure strategy instruction improves seventh graders’ content area reading comprehension. Journal of Educational Psychology, 109(6), 741–760. https://doi.org/10.1037/edu0000168

41.

Wong

B. Y.

Wong

Blenkinsop

(1989). Cognitive and metacognitive aspects of learning disabled adolescents’ composing problems. Learning Disability Quarterly, 12(4), 300–322. https://doi.org/10.2307/1510212

42.

Wood

S. G.

Moxley

J. H.

Tighe

E. L.

Wagner

R. K.

(2018). Does use of text-to-speech and related read-aloud tools improve reading comprehension for students with reading disabilities? A meta-analysis. Journal of Learning Disabilities, 51(1), 73–84. https://doi.org/10.1177/0022219416688170

			Reading Comprehension Test			Retell
Student		Number of Sessions	X	SS	Range	X	SS	Range
Harry	Baseline	5	3.80	.44	3–4	5.40	.54	5–6
	Intervention	10	7.4	1.26	6–10	9.7	1.25	8–12
	Follow-up	3	6.66	.57	6–7	8.33	.57	8–9
Hermione	Baseline	6	3.33	.51	3–4	5.66	1.03	5–7
	Intervention	10	7.3	.82	6–8	9.8	1.39	7–12
	Follow-up	3	7.66	.57	7–8	8.33	1.52	7–10
Neville	Baseline	7	3	.57	2–4	4.85	.89	4–6
	Intervention	10	6.5	1.58	4–9	7.2	1.22	5–9
	Follow-up	3	7	1.0	6–8	7.33	.57	7–8
Ron	Baseline	8	3.5	.53	3–4	5.12	.83	4–6
	Intervention	10	8.0	.94	7–10	10.5	1.17	8–12
	Follow-up	3	7.66	.57	6–8	8.66	.57	8–9