Abstract
In recent years, mobile telephone usage has skyrocketed in the general population (Perrin, 2021) and for persons with visual impairments (Locke et al., 2020). Research indicates that persons with visual impairments (i.e., those who are blind or have low vision) used various types of applications (apps) for iOS and Android devices, with the usage of specialized apps topping 90% (Griffin-Shirley et al., 2017). A plethora of educational and informational mobile apps have been developed on numerous topics, including parenting (Virani et al., 2019), literacy (DeForte et al., 2020), and health (Kim & Xie, 2017; Madrigal-Cadavid et al., 2020).
An important part of the app development process is usability, defined as “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (International Organisation for Standardisation, 2018). Teachers of students with visual impairments reported that their students encountered usability problems when using mobile devices and felt that usability problems contributed to students’ difficulties learning to use technology (Baker et al., 2019). Issues that impact mobile app usability include small screen sizes, connectivity issues, and limited input modalities (Harrison et al., 2013), and usability issues were evident even among established, widely used apps (Ahmad et al., 2018).
Some researchers have examined the usability of mainstream and specialized apps with users with visual impairments (Griffin-Shirley et al., 2017; Lee & Lee, 2019; Locke et al., 2020). In a survey of persons with visual impairments, most individuals surveyed rated mainstream and specialized apps as user friendly and accessible (Griffin-Shirley et al., 2017). Findings from other studies were less positive. Locke et al. (2020) documented issues with specialized apps, including technical problems, difficulties with use or understanding, and lack of accessibility. In an evaluation of two mainstream educational apps, young adults who are blind identified major usability issues related to efficiency, effectiveness, and satisfaction (Lee & Lee, 2019). Considering usability only when evaluating the final versions of apps can be problematic because major changes may not be feasible (Ahmad et al., 2018). Conducting iterative usability testing during the app development process may alleviate this problem by allowing developers to identify and address usability issues in early prototypes (Beatty et al., 2018; Schnall et al., 2016).
Despite the prevalence of apps for education and information, and the importance of usability in app development, few studies have focused on the usability of educational or informational apps for students with visual impairments as part of the development process. Correa et al. (2018) examined usability during the development of an educational game for students with visual impairments. Other educational app development publications (Beal & Rosenblum, 2015; Kamei-Hannan et al., 2015) had more emphasis on accessibility than usability.
Under a 5-year intervention development project, we produced an informational app called 4to24, which supports independence and employment preparation for students with visual impairments from childhood to young adulthood (aged 4 to 24 years). To encourage parental involvement and student engagement, the app delivers timely, relevant modules to parents and older students with information, activity suggestions, and resources related to independent living skills, technology, social skills, orientation and mobility, postsecondary education, career preparedness, and other topics. In this article, we will use the term “parents” to include parents, guardians, grandparents, or any other parental figure who has primary responsibility for a child. See Figure 1 for excerpts from a module with examples of parent and student versions. The app provides a customized experience for each user based on the child's age and grade level, a built-in benchmarking system, and user activity within the app. App development encompassed a multi-phase user-centered, iterative process that included strategically timed usability testing with parents and students that informed subsequent design decisions.
Excerpts from parent and student versions of a 4to24 module: Work! Job Shadow Experience.
This article focuses on usability testing of the app and provides information about the app's technical development. The project aims for these phases of the development process included:
Use an iterative process to develop the app, determine user interface and functional accessibility issues, and correct them prior to field testing. Evaluate subjective usability of the app at key developmental stages.
Method
Procedure
App design phases included extensive planning sessions, stakeholder input from an advisory board and user focus groups, content development and validation, usability testing in two phases, and a final field test (Figure 2). See Antonelli et al. (2021) for information about the overall development process, stakeholder input, content development, and content validation.
App development timeline.
For the app's design and technical development, a design team that included researchers with varying areas of expertise, including blindness rehabilitation, human factors design and testing, and education, collaborated with technical developers from an organization in the blindness field. We used an iterative development process based on the agile methodology (Beck et al., 2001), a commonly used approach to software development (Al-Zewairi et al., 2017; Flora & Chande, 2014). App design team members met with the technical developers regularly throughout the development process to determine functionality, features, interface design, and all other app specifications.
Technical Specifications
Through extensive planning, testing, and iterations of potential design options, the design and development teams ultimately devised a system in which the app could automatically select and deploy modules to users based on multiple factors: the student's age, grade in school, benchmarks indicated during account setup, and the user's completion of modules within specific content categories. Information about each module is stored in a metadata file that the app uses to determine content deployment to each user's account. To determine the best beginning modules for a new user, the app uses a benchmarking system, a series of in-app checklists completed by the user to indicate the student's knowledge and previous experience in multiple topic areas (see Figure 3 for an example). From that point forward, the app automatically deploys new modules to the user's account based on the user's activity (e.g., marking modules as complete) and the student's age or grade level. In this way, the user account moves through the entire library of modules from the user's starting point to the end of the app content. Depending on the student's age, this process could take anywhere from several months for an older student to multiple years for a younger student.
Example screen from the 4to24 app benchmarking system.
Members of the technical development team were well-versed in standards for creating accessible software from the ground up; accessibility was considered in the build of every feature, and the app meets Web Content Accessibility Guidelines 2.1 A and AA standards for accessibility. The app was built to be compatible with standard assistive technology such as braille displays and to be responsive to built-in smartphone accessibility features such as magnification and screen reading technology for both iOS and Android platforms. The app was tested for accessibility internally by the development team and through user testing.
The app's architecture is composed of a thin software client built using NativeScript, which communicates with backend servers to handle the business logic for the app. These backend servers are running Django to serve requests to the app and send push notifications to users’ devices. The Django servers are hosted on Google Cloud Compute Engine virtual machines. The app's database is stored on a dedicated Google Cloud structured query language (SQL) instance.
Technical developers followed industry standards for establishing data security within the app. The design team consulted with their institution's legal department to create a privacy policy and terms of service agreement to inform users about data security and data use. The app collects the user's email address for login purposes and date of birth to verify their age. It also collects and stores the student's month and year of birth and grade level for the purpose of deploying content by age and grade. The student's and parent's first names are collected for personalization, but users can input any name they desire. Google Analytics is used to collect anonymous, aggregated usage data such as what app pages are being used, and the app database stores user activity for each account (i.e., modules the users have marked complete).
Usability Testing
App usability testing was conducted in person at two time points in the second and third years of the development process to determine the app's ease of use and intuitiveness. The Institutional Review Board for the Protection of Human Subjects at Mississippi State University reviewed the usability study protocol and concluded that it did not require institutional review board oversight. The board determined that the study did not meet the second part of the federal definition of research (Protection of Human Subjects, 2009) because its intent was to inform technical development of the app instead of developing or contributing to generalizable knowledge. Participation in the usability sessions was voluntary; parents and students received information about the app and the purpose of the sessions before deciding whether to participate, and we obtained parental permission for all student participants. All participants had to be smartphone users, and parents had to have a child with a visual impairment between the ages of 4 and 24 years. Student participants had to have a visual impairment and be between the ages of 14 and 24 years. Participants in Round 1 of testing received a $35 incentive for participating, and those in Round 2 received $50.
Two researchers conducted each usability session: one researcher provided instructions and observed the participant, and the other managed the audio and video recording equipment and documented field notes. Usability sessions were held individually with each participant and lasted approximately 1 hour. The usability protocol included a description of the app and instructions for the tasks to be completed. The design team and the technical developer predetermined usability tasks before each round of testing to assess the implemented elements at those time points. We followed industry-established norms for usability testing of five participants for each type of user (Turner et al., 2006). In Round 1, the users were asked to create an account and navigate to a module. In Round 2, the users were asked to create an account, navigate to a module, review it, mark it complete, and interpret the user's progress display on the dashboard. Users completed tasks on either a laptop (using the web app) or a mobile phone using either an Android or iOS operating system, depending on user preference and experience. To assess the intuitiveness of the app interface, the researchers specified the task to complete but did not instruct the users on how to complete the tasks. Researchers only provided guidance if a user was unable to proceed to the next step of a task without instruction. Following each round of usability testing, the researchers shared summaries, including key details about user interface issues, with the app technical developers.
Participants
Round 1
In Round 1, students were recruited from a summer camp at a residential rehabilitation facility in the southern United States. Parents were recruited via the residential rehabilitation facility, advisory board members, national consumer conferences, local contacts in the surrounding state, and an online registry of potential app users. Five students, aged 16 to 23 years (M = 19.00, SD = 3.24), participated in the student usability sessions. The parent usability sessions included four parents. Table 1 provides additional information about participants’ characteristics and the devices and accessibility features they used.
Participant Characteristics and Devices and Accessibility Features Used.
Note. All values are ns.
Round 2
Round 2 included parent and student participants who were recruited through industry contacts and a national consumer organization. Five students, aged 18 to 24 years (M = 20.60, SD = 3.13), participated in the student usability sessions. The parent usability sessions included five parents or guardians (one participant was a grandparent of a child with a visual impairment). See Table 1 for more information.
Measures
We used a think-aloud protocol to capture participants’ thoughts and gain insight into their behavior as they completed the testing tasks (Cooke, 2010). Participants verbally described their thought process at each step as they observed each interface screen and decided the next necessary action to take to complete the task. Having participants verbalize their thoughts allowed the researchers to identify places within the app that caused confusion. Usability sessions were audio- and video-recorded to capture users’ verbal descriptions, hands, and device screens as they completed the tasks.
At the end of each session, users completed the System Usability Scale (SUS), a 10-item Likert-scale measure that assesses the global, subjective usability of a system (Brooke, 1996). A sample item from the SUS is “I thought the system was easy to use.” (1 = strongly disagree, 5 = strongly agree). The SUS has evidence of construct validity (Bangor et al., 2008), concurrent validity (Bangor et al., 2009), and internal consistency reliability (Bangor et al., 2008; Sauro, 2011) across samples and products. We followed the scoring instructions provided by Brooke (1996) to calculate total scores. Potential values could range from 0 to 100, with higher values being more positive. Based on responses from 446 studies, the average SUS score was 68; therefore, SUS scores above 68 are considered above average (Sauro & Lewis, 2012).
Data Analysis
Descriptive statistics (i.e., frequencies and means) of participant demographics and SUS scores from both rounds of usability testing were generated using SAS 9.4. Two researchers independently reviewed the audio recordings, video recordings, and field notes from each session to identify design and accessibility issues. The researchers compiled and jointly verified the identified issues.
Results
Round 1
Table 2 indicates that the Round 1 SUS scores were above average. Feedback from this round primarily focused on ease of navigation and identification of points in the interface where users had difficulty when creating an account. During Round 1, some technical functionality aspects, such as how the interface was hosted, produced problems for navigation (e.g., having to repeat a double-click gesture to enter an edit field using a screen reader and not providing feedback about which edit field the user was in). Once the researchers became aware of those issues, they informed subsequent participants about them to minimize frustration and focus on gathering user feedback related to improving the interface. Examples of feedback included two function buttons that were located too close together on the screen so that users selected the wrong one when attempting to tap it, breaking up lengthy account setup screens into multiple screens with fewer input fields to keep one's place or locate errors to correct easily, and relabeling some function buttons for clarity. Before beginning Round 2, the development team addressed the major issues identified during Round 1 by eliminating the need to repeat a double-click gesture, creating multiple account setup pages with less information on each page, and removing an unnecessary function button.
App System Usability Scale (SUS) Scores.
Round 2
As shown in Table 2, the SUS scores for Round 2 were also above average. Round 2 feedback focused predominantly on the labeling of information and the need for cues on how to accomplish tasks in the interface. One user recommended adding explanatory text before the benchmarks section to explain the task's purpose. Other users thought some benchmark statements were too complex and would be easier to answer if they were simplified. Several users identified function buttons or entry fields that could be better labeled or provide better feedback when accessed. For example, one user advised that the “Complete Profile” button at the end of the benchmarking process was confusing; she stated she would prefer it to be labeled “Done,” “Next,” or “Finish.” Another notable response from multiple users was that two pages in the app—the dashboard and another page that only listed modules—caused confusion because users did not understand how their purposes were different. This finding informed a subsequent decision to consolidate those pages and eliminate the separate page of modules. One low vision user indicated that the color contrast of the interface was not high enough; he began his session using the device's built-in magnification feature but had to switch to VoiceOver to complete the testing. Following Round 2, the development team corrected all major issues in preparation for field testing the app as the next stage of development.
Discussion
This article describes the technical design and usability testing of the 4to24 app, including how the development and design teams considered accessibility and usability throughout the development process. 4to24 provides relevant, appropriately timed resources and information for students with visual impairments and their parents to help students prepare for employment as an adult. We ultimately developed a simple system to perform a fairly complicated task: deploying informational modules spanning 21 years’ worth of content to app users on a customized schedule via an automated process. Considering that usability issues occur even in widely used apps and can adversely impact learning for students with visual impairments (Baker et al., 2019), we recognized the importance of addressing usability when developing our app. The usability testing sessions we conducted allowed us to identify usability issues and correct them while development was ongoing, as well as assess overall usability of the app at different stages of the process.
Participants identified several specific functional and interface issues during the usability sessions. Functional issues included text entry fields that were not yet fully functional (e.g., having to double-click twice to enter a text field). Interface issues included actions that were not easy to select (e.g., buttons located too close together so that tap gestures could activate the wrong button), poorly labeled buttons and fields that made it difficult for users to predict what action would occur when selected, and unclear feedback after performing certain actions. We corrected these issues early in the process before the app's technical development was locked into a specific design.
Despite the identified functional and usability issues, participants’ SUS ratings were above average (Sauro, 2011; Sauro & Lewis, 2012) at each testing point. The 4to24 app's acceptable SUS ratings may be attributable to several factors. First, we made intentional design decisions to consider intuitiveness, simplicity, and ease of use. Second, we obtained user input at key steps in the development and design process, starting from the earliest stages (Antonelli et al., 2021). Finally, we collaborated with technical developers who are experts in designing usable and accessible products, resulting in the 4to24 app being “born accessible” (Capiel, 2014).
Our usability sessions followed the industry-established norm of using only five testers for each user type (Turner et al., 2006). Most major issues surfaced early in the testing sessions, and further testing generally did not reveal new or different issues. Approximately five testers per group per round were more than adequate for obtaining feedback and information about the users’ experiences. We were confident that participants discovered all major issues. Our participants had a range of technical skill levels and functional vision. We were responsive to this variability by allowing participants to use whatever devices and accessibility features they needed. We also allowed flexibility in the allotted time to complete the usability tasks and the amount of instruction or assistance provided if a less skilled participant got stuck on a particular step.
Using an iterative development process allowed us to make design decisions and seek stakeholder input as the project unfolded while being responsive to end-user feedback on early prototypes. In our case, the app's content development and technical development phases occurred concurrently, and the content informed the design of the content deployment system. This process also enabled us to test accessibility features and usability simultaneously and incorporate feedback related to both accessibility and usability into the app's design. Because we considered accessibility when designing and developing all features, accessibility did not have to be retrofitted into a product that was already finalized.
Limitations
We must acknowledge several limitations to this phase of the development project. First, most of the participants were experienced smartphone users. Therefore, we may have missed usability issues that would impact less experienced users. Second, most of our participants were iPhone users, and the few people who used an Android smartphone or laptop for the usability testing did not require assistive technology. It will be important for us to seek additional feedback from users of these platforms and be responsive to any reported usability or accessibility issues to ensure that all users can benefit from the app. Finally, the SUS was unmoderated, but researchers were present when participants completed the survey, creating potential social desirability bias with the SUS scores.
Implications and Future Directions
These early findings from our usability testing support that the 4to24 app is usable and accessible for students with visual impairments and their parents. We used information gained from the two rounds of usability testing to improve the app's design and functionality based on user feedback and experience.
Following the second round of usability testing, changes and upgrades were implemented before our final phase of testing the app, which was a 6-month, real-time field test with parents and students. During that field test, usability was measured with the SUS at two additional time points to continue an ongoing assessment of 4to24's accessibility, effectiveness, efficiency, and user satisfaction. Field test findings are reported in a separate manuscript. The app is currently available at no cost through iOS and Android app stores and on the web at 4to24.org.
Footnotes
Authors' note
The contents of this report were developed under a grant from the U.S. Department of Health and Human Services, NIDILRR grant 90RT5040. However, these contents do not necessarily represent the policy of the Department of Health and Human Services and should not indicate endorsement by the Federal Government.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute on Disability, Independent Living, and Rehabilitation Research (grant number 90RT5040).
