Abstract
Computers can be an effective teaching method for students with learning disabilities (LD). The use of mobile devices as education tools for students with disabilities has received considerable attention in special education recently. Parents, teachers, and professionals look for effective applications (i.e., apps) that meet the needs of their children and students. This is often challenging for them due to the large number of education apps that are available in the market. This article introduces and describes an evaluation rubric developed to help parents, teachers, and other professionals evaluate apps for students with LD.
Over the past 30 years, research has found the use of computers to be an effective supplementary teaching method for teaching students with learning disabilities (LD). Computers can be used effectively to help these students compensate for their challenges and enhance their skills in such areas as reading, writing, and mathematics (B. Bryant et al., in press; Hecker, Burns, Elkind, Elkind, & Katz, 2002; Higgins & Raskind, 2005). Research has also demonstrated that computers can be helpful for increasing student motivation, attention in learning, time on task, and independence (Manset-Williamson, Dunn, Hinshaw, & Nelson, 2008).
As technology has evolved, a new type of computer, the mobile device (e.g., smartphone, tablet), has gained popularity. The mobile device is a small-sized, portable computer that typically has touch-screen features; the iPad is one of the more popular mobile devices used in schools today. Upon the initial release of the iPad in 2010, the device and instructional applications (i.e., apps) quickly gained popularity in educational settings despite many teachers’ lack of knowledge concerning the devices’ implications for learning (Peluso, 2012). Nevertheless, teachers and parents have reported positive results using mobile devices and apps, and professionals have also demonstrated that mobile devices and apps have the potential to be useful tools for students with disabilities (Korner & Leske, 2012). Mobile devices provide the availability of downloadable, inexpensive software that can serve as cost-effective assistive technology (Douglas, Wojcik, & Thompson, 2011). In addition, their touch-screen feature allows students with disabilities to use the device without having to operate a mouse or a touch pad. Most mobile devices also have Internet access, built-in video, a camera, and audio-capture capabilities (Korner & Leske, 2012). These devices can be easily individualized to meet the needs of individual students with disabilities.
Many parents of struggling students and the teachers who work with them seek apps that are valid for academic improvement. Not surprisingly, however, they often express challenges in finding effective apps. By the fall of 2013, there were approximately 1 million apps available (Costello, n.d.), and more apps are being released in the market daily (Buckler & Peterson, 2012). Thus, it is challenging to find appropriate apps among the many that are available. Unfortunately, parents and teachers often make the mistake of using inappropriate apps; they select apps randomly and use them without examining their educational quality (Cooper, 2012). As Clark (1983) noted, not technology itself—but rather instructional variables inherent in the technology—are critical for effective instruction. However, too few technology tools include effective instructional variables based on valid learning principles.
Several years ago, educators faced the issue of ineffective instructional programs with computer-assisted instruction (CAI). At that time, researchers (Boone & Higgins, 2007) suggested instructional design variables that should be carefully considered for effective CAI for teaching students with disabilities. The instructional variables included (a) feedback and error correction opportunities, (b) multiple practice/examples and appropriate review opportunities (e.g., prerequisite skills, cumulative review, technology training), (c) empirically validated instructional strategies or principles (e.g., direct instruction), (d) systematic curriculum organized with logically sequenced skills, (e) adjustable individual preferences (e.g., pace, level, time, goal), (f) student data recording for progress monitoring, (g) motivation enhancements, and (h) content provision in multiple formats (e.g., text, graphics, spoken words).
How well do current mobile device instructional apps employ effective instructional variables? This is a question that is also important for researchers comparing instructional apps to traditional teacher-directed instruction. Several researchers have also provided criteria for evaluating instructional apps (see http://learninginhand.com/blog/ways-to-evaluate-educational-apps.html). This column addresses the quality of apps by presenting an evaluation rubric for examining apps for students with LD, and likewise presents information about the validity of the evaluation rubric, introduces the rubric, and examines how the rubric can be of value now and in the future.
Validity of the Rubric
Much of what is known about effective instruction for students with LD has long been available in the literature (Archer & Hughes, 2010; Swanson, Hoskyn, & Lee, 1999). Many of the features of effective instruction contained in these works and noted previously are included in the rubric. After the rubric was developed, five experienced instructional technology and disability professionals examined each survey item and responded to wording and relevance. Three were classroom teachers, one was the head of a charter school, and one was a researcher and instructional technology author. All rated each item on a Likert scale as being important to very important, provided suggestions as to wordings (e.g., the rubric originally included the term school-based font; the recommendation was to change the wording to easy-to-read font), and one reviewer suggested that an item be added that examined error analysis, which was done.
The Mobile Device App Evaluation Rubric
The app evaluation rubric for students with LD consists of three sections: identifying information, evaluation, and grading. See Figures 1, 2, and 3, which provide the three sections of the rubric used to evaluate a mathematics application; a blank evaluation form is available for download at https://utexas.box.com/appevals. Each section of the form is described here.
Section 1 of the evaluation rubric.
Section 2 of the evaluation rubric.
Section 3 of the evaluation rubric.
Section 1: Identifying Information
Basic information
The evaluation form begins by providing basic information about the app. This information consists of the name of app, the publisher, and the price. In Figure 1, neither the name of the app nor its publisher is listed; however, the price is noted to be $1.99.
Content area
This section of the evaluation form notes the primary and secondary content areas that are the focus of instruction. For example, the primary content area of reading contains seven secondary areas: (a) general learning, (b) basic knowledge, (c) comprehension, (d) fluency, (e) word/phonics analysis, (f) word/letter recognition, and (g) other, so consumers can access, find, and identify apps of specific interest. For the mathematics app evaluated, the content of the app is mathematics and calculation.
Objective(s)
Each app provides an area of focus or at least one objective. Often, objectives are not specifically provided by the publisher, but they are fairly easily discerned as one works through the app. For the evaluated app, the objective is “to practice addition, subtraction, addition, division basic facts/to improve fluency.”
Content level
Apps are categorized according to one or more of four levels: (a) primary, (b) upper elementary, (c) middle school, and (d) high school. Again, such information may not be readily available from the publisher, but it is easily identified by looking at the instructional content. The mathematics app is appropriate for students at the primary and upper-elementary levels.
Graphics/theme level
The graphics and theme level should also be considered when matching the application to the student. Of concern here is information about the types of graphics and themes presented, such as the characters used for the app, background images, guide voices, and language level. Such information may be particularly important for students with LD, who present a myriad of learning challenges. The raters decided that the graphic content of the mathematics app was at the upper-elementary and middle school levels.
Types of application
Finally, the types of apps and the purpose for their use are examined. If the app provides students with opportunities to practice certain concepts repeatedly, they are considered “drill and practice.” An app that uses a game format to build concepts and skills is appropriately called a “game.” If the students with LD can learn the new concept and skill through following the procedures of the application, it will be marked as “lecturing or tutoring.” For applications that simulate true-life situations or activities that allow students with LD to extend their experiences through the application, the “simulation” box is checked. Finally, if the application is not directly related to teaching or to practicing some concepts or skills but instead was developed for helping students’ learning (e.g., to help develop study skills), this application is classified as a “supplementary tool for learning.” The evaluated app is an example of a drill-and-practice application.
Section 2: Evaluation
Swanson et al. (1999) found that students with LD learn most effectively via a combination of explicit (e.g., sequencing skills logically, breaking down complex skills and strategies into smaller instructional units) and strategic (e.g., using a specific routine to accomplish a goal, such as problem solving, comprehending text) instruction. This finding, when combined with the work of researchers, theorists, and practitioners who studied CAI previously, formed the basis of this apps evaluation. Yet, even though these findings are readily available, many parents and teachers continue to have difficulty in determining the appropriate instructional apps that provide both effective instruction and an enjoyable learning environment for their child/student (Sim, MacFarlane, & Horton, 2005). The rubric’s framework in decision making concerns the most appropriate mobile device application using a consumer-centered approach. The evaluation is highly affected by (a) instructional elements, (b) personal factors, and (c) environmental factors. The first seven components involve several elements of effective instruction and strategies (Hunter & Russell, 1994; Swanson, 2001). The remaining six components, presented in no particular order of importance, involve personal factors and environmental factors. Each evaluation component and its rating criteria are described. It should be noted that there is an element of subjectivity when evaluating design features of apps. Thus, during the evaluation, one should consider the variance that exists across students with LD and rate the app based on its flexibility with regard to design features listed. It should be noted that the rubric does not contain an item that relates to a student’s instructional objective. Such a feature, although very important when selecting an app for use, is not related to the quality of the app itself; thus, that feature is not considered in the evaluation.
Here, along with the description of the evaluation form’s sections, information and ratings are provided by the three lead authors. Initial agreement across the items among the three raters was over 90%; the raters met, discussed and reconciled their differences, and achieved 100% agreement.
Objective
The objective is a guide for lessons and assessment and should be clear and concise. The highest rating is reserved for objectives that are clearly stated and easily identified. Consumers should not have to work through the application to identify its objective; the objective should be forthright and accessible. The next highest rating is reserved for objectives that are easily identified although not clearly stated. The lowest rating is reserved for apps that have objectives that are difficult to identify (i.e., there is no consistent instructional theme). For the mathematics app evaluated, the reviewers rated the objective as 3, meaning that the objective was overtly stated.
Strategy
As noted earlier, research has demonstrated that students with LD learn best when provided a combination of explicit and strategic instruction. Apps should include a strategy to help students learn in a logical, sequential manner. Strategies for encouragement, reinforcement, and assessment of the changing needs of students are highly related to positive outcomes of the decision-making process for technology devices (Scherer, Jutai, Fuhrer, Demers, & DeRuyter, 2007). Some strategies, but certainly not all, involve mnemonics. The highest rating is reserved for apps that provide strategies for doing the work and where skills are broken down into manageable and connected steps. The next rating is given when no strategy is explicitly stated but skills are broken down. The lowest rating is assigned when no strategy is provided and when skills are not broken down into small steps. For the mathematics app evaluated, no specific strategy was presented, but the skills were broken down sufficiently to warrant a 2 rating.
Examples
One of the most important elements of effective instruction involves providing examples that students can use as models in effective learning. Examples in the app help to make connections between instructional steps while allowing the learner to combine the steps to meet the instructional objective. To provide more effective instruction, the application should offer multiple examples. For many students with LD, learning becomes disorganized, disconnected, and isolated; they require multiple examples for each concept or skill being taught. The highest rating is awarded when the app provides three or more examples for each concept/skill. Apps that give one or two examples for each concept/skill are awarded the middle rating. When no example is given for each concept/skill, the lowest rating is assigned. Three or more examples were provided in the mathematics application, so the raters assigned a rating of 3.
Practice
Students with LD must be given multiple practice opportunities to enhance automaticity and deeper learning. Thus, to allow for sufficient practice opportunities, the application should provide consistent and regular practice. When students practice, they begin to break intellectual boundaries by using and elaborating their skills and form more connections to learning. The ratings reflect this premise. The highest rating goes to apps that provide at least five practice opportunities before moving on to a new skill/concept. The next highest rating goes to apps that provide from one to four practice opportunities before moving on to the new skill/concept. The lowest rating is assigned when no practice opportunities exist. Sufficient practice opportunities, that is, more than four, were provided on the mathematics app, so the evaluators assigned a rating of 3.
Error correction and feedback
Error correction and feedback give students explicit information about their performance. Students who are allowed to continue making mistakes are likely to automatize errors rather than correct responses. Students who receive positive reinforcement know that they are being successful and are encouraged to continue what they are doing. The highest app rating is awarded when students are notified of a correct/incorrect response and are given the correct answer. The next rating is assigned when students are notified of a correct/incorrect response but are not given the correct answer. The lowest rating is given when no notification and no correct answer are given. A 2 was awarded for the mathematics app because the user was notified of the correct/incorrect response but was not provided the correct answer when the student gave an incorrect answer.
Item error analysis
Students who are given multiple opportunities to practice provide rich data to teachers about how well they are learning and where problems occur during learning. Some students make haphazard errors that may occur randomly and inconsistently, yet others make strategic and consistent errors that allow for a degree of information about students’ thinking as they learn. Effective instruction is one that analyzes the data that are accumulated, determines whether errors are random or strategic, and uses those data to guide student learning. To examine this important instructional element, evaluators give the highest rating when a record is kept of the types of errors that the student makes and the analysis is reported. They award the next highest rating when a record is kept of the types of errors that the student makes but nothing is reported. They give the lowest rating for apps when no error analysis is provided. For the mathematics application evaluated, the app maintained a record of the types of errors that were made, but no report was available. Thus, a rating of 2 was assigned.
Progress monitoring
Progress monitoring has become a staple of effective instructional programs by providing ongoing, accurate recording of student performance. Progress monitoring is used to alert the teacher and learner when the student is or is not meeting the instructional objective and/or making sufficient progress toward some goal. Apps receive the highest rating when total points are provided and progress is detected by application of a tracking system. The next rating is awarded if total points are provided but no tracking is available. The lowest rating is given when no total points are provided and no tracking is available. The mathematics app was awarded a 3 for progress monitoring because not only was the total number of points provided but progress was tracked as well.
Motivation
Mobile device applications should support user engagement (i.e., an important aspect of motivation) during learning, as it guides direction and maintains persistence. The application should sustain the students’ motivation; otherwise, students may get easily discouraged and feel like they have little reason for continuing their learning. Apps that consistently maintain student engagement are given the highest rating. Those that engage students at first, but soon lose their attention, are rated lower, and the lowest rating is given for apps that result in little or no engagement. It was determined that the mathematics app kept students engaged in learning, so a rating of 3 was awarded.
Navigation
For students with LD, apps should be easy and simple and allow students to receive assistance, either technical or instructional, and be fully accessible. An accessible application allows all students to use it in a simple, consistent, and straightforward manner. For example, in terms of accessibility, students benefit the most when they are able to log in, customize the interface, and save their preferences without difficulty (Marino, Tsuruski, & Basham, 2011). Thus, the highest rating is given when navigation is easy/simple and students can easily get assistance. The next rating is given when the app has easy/simple navigation but it is difficult to receive help. The lowest rating is awarded when the navigation is not easy/simple and users find it difficult to get help. The app was awarded a 3 because it provided simple navigation and users could get help easily if problems arose.
Visual and auditory stimuli
Visual and auditory stimuli in an application provide a virtual space where students interact with learning materials. For maintaining student motivation and engagement, the application should provide this in a positive way. Audio in educational software gives students a sense of mood as part of the multimedia instruction. For instance, adding background music to a computer game or simulation can help set the mood (Sharples & Beale, 2002).
At the same time, if the app has distracting and disorganized stimuli, it may become difficult for students, parents, and teachers to use and thus prevent students from learning effectively and efficiently. An appropriate application design maximizes the likelihood that students achieve their goals and the intended learning outcomes. A negative and distracting design may impede learning and achievement. It is often suggested to choose multimedia that foster an active learning environment through images and sounds in multimedia instruction (Mayer, 2009). In addition, sounds should be optional and choices should be provided for visual backgrounds, depending on student needs and preferences. Thus, to receive the highest rating, the background image and sound should not be distracting and the sound can be turned off. The next highest rating is provided when either the background image and sound are not distracting or the sound can be turned off but not both. Finally, the lowest rating is awarded when the background image and/or sound are distracting and sound cannot be turned off. For the evaluated app, the raters awarded a 3, meaning that the background image and sound were not distracting and the sound could be turned off if desired.
Font
Text, as a main medium for teaching, should be carefully structured and well designed to inform, instruct, and aid in readability. Screen design provides a first impression by users and is a critical component in the overall desirability, usability, and effectiveness of a system. Proper fonts can assist students’ interaction with the system of educational software, and in terms of accessibility, relative font sizes rather than those that are fixed should be used (Sharples & Beale, 2002). When the font size is sufficient and modifiable and font type is easy to read, the app receives the highest rating. When either the font size is sufficient or modifiable or the font type is easy to read (but not both), the app receives a lower rating. Finally, when the font size is not sufficient or modifiable and the font type is not easy to read, the app receives the lowest rating. In the raters’ opinions, the font size of the evaluated app was sufficient and modifiable and the font type was easy to read; thus, a 3 was awarded.
Customized settings
A defining characteristic of students with LD is that they have learning strengths and weaknesses. Customized settings can provide individualization and reflect each student’s unique characteristics and prior knowledge. Background knowledge influences how students interpret and acquire what they are learning. Many students with LD come to the learning situation with inconsistent or nonexistent background knowledge that can interfere with or impede their learning new skills and concepts. When it comes to selection, the process should consider the student’s strengths and limitations and the device as the most appropriate compensatory tool (D. Bryant & Bryant, 1998). Therefore, the app with the highest rating is one that includes available customization for an individual student. The next highest rating is given when customization is available but is limited for an individual student. Finally, the lowest rating is assigned when customization is not available for an individual student. Because the mathematics app could be customized easily for an individual student, the evaluators assigned a rating of 3.
Content error and bias
Educational resources carry many subtle messages that may convey that the view described is acceptable or at least preferred by society (Spiegel, 1990). For example, according to the Environmental Education and Training Partnership Resource Library (1999), the content on a website may contain bias in terms of viewpoints that may not pertain to those who have different beliefs and values. The biased content can send a negative message about self-worth that is reflected in harmful lessons. Students should learn to be proud of themselves and respect differences (Derman-Sparks & Edwards, 2010). In addition, learning is inhibited when errors are presented during instruction. Incorrect examples or misstated strategies can contribute to confusion. The evaluation system takes these important instructional features into account by providing the highest rating to apps that have content that is free of errors, is up-to-date, and is free from bias (e.g., race, gender) to the extent possible. The second highest rating is awarded when the content is free of errors, is up-to-date, or is free from bias but not all three. The lowest rating is given when the content is not free of errors, is dated, and is not free from bias. In the judgment of the raters, the app’s content was error-free, up-to-date, and free from race, gender, and other biases. Therefore, they provided a rating of 3.
Section 3: Grading
To calculate a score for each app, there is a section for entering each score of 1, 2, or 3 and the number of categories assessed to calculate the total score percentage. Because there are no agreed-upon criteria for app excellence, the evaluation uses the ranking used in many schools, that is, 90% to 100% would be an A (excellent), 80% to 90% would be a B (good), and so forth for C, D, and F grades. Some schools may use different grading criteria, but this procedure has some precedence for grading. For the sample evaluation, the mathematics app received 10 item ratings of 3 and three item ratings of 2, for a total of 36 points. The 36 points were divided by the 39 (total points possible), yielding a decimal of .923, which is multiplied by 100 to compose 92.3%, which translates to an A grade.
Conclusion
Mobile devices, including smartphones and tablets, and available instructional applications are becoming increasingly popular as a means to provide engaging instructional opportunities to students. This article introduced and described an evaluation rubric developed to help parents, teachers, and other professionals evaluate apps for students with LD.
Researchers have shown that students with LD learn most effectively when provided with explicit and strategic instruction. The extent to which apps incorporate these instructional features, along with engaging and effective design features, can determine the relative effectiveness that the apps will be to promote learning. The 13 features of the rubric were described, and a sample app was evaluated. Scores across the multiple features can be summed to provide an overall index of quality ranging from A to F. The evaluation can be used by teachers and parents to examine the overall quality of apps they are interested in selecting for use with their students/children.
Footnotes
Authors’ Note
Although the evaluation form can be used by anyone to evaluate apps, the authors’ goal is to create a repository of app reviews that can be accessed by educators and parents before they go about the purchasing process. The apps being evaluated are in the following areas: reading, writing, math, science, and social studies. However, the rubric can be used in other instructional areas. The overall goal of the evaluation effort is to provide a repository of app ratings that can be accessed online free of charge. Such a repository can be of benefit to those looking for apps that can be used with students who have LD or students who struggle and share similar learner characteristics. A sampling of evaluations collected thus far is available at https://utexas.box.com/appevals. The example of a mathematics application provided in Figures 1 through 3 can serve as a model. Additional and more extensive instructions for ratings, as well as a blank evaluation rubric, can be obtained by going to 
. We invite readers to visit the website to examine the existing evaluations. For those who use the rubric to evaluate apps (including teachers who use apps with their students or those in higher education who may wish to build apps evaluations into their student assignments) and wish to include their evaluations on the website, contact the fourth author at
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.