Components of Valid Learning Media Assessments

Abstract

Introduction

The learning media assessment (LMA) process is the mechanism professionals use to determine a student's optimal learning and reading media. Currently, no tool has been validated for the purpose of conducting the LMA process.

Methods

A Delphi approach was used whereby a panel of identified experts in the LMA process went through iterative phases recommending items to be included in the LMA process and voting on keeping, changing, or deleting items until a final list of agreed upon items was established.

Results

At the completion of five rounds of voting and amending items, the final list of agreed upon items included 2 general goals, 66 items, and 18 acceptable modifications to the LMA process. The items to be included in an LMA process were categorized into logistical items to be noted (9 items), LMA procedural characteristics (11 items), materials to be used (2 items), information to be gathered (20 items), student medical/visual characteristics (8 items), student reading or academic characteristics (10 items), and student technology and literacy tool use (6 items). Eighteen acceptable modifications or adjustments that could be made to assessment procedures were also agreed upon.

Discussion

The framework of what is critical in the LMA process, as shown in this Delphi study, can be used to assess the validity of LMA tools.

Implications for Practitioners

Practitioners can use the framework built by this Delphi process to evaluate the validity of LMA tools they use. Practitioners can assess the tools they use by asking if all the validated content areas are addressed, if validated procedures are being followed, and if validated accommodations are in place.

Keywords

learning media assessment LMA visual impairment braille individualized education program IEP

According to the Individuals with Disabilities Education Act (IDEA, 2004) the default reading medium for all students who are visually impaired (ie, those who are blind or have low vision) is braille unless the individualized education program (IEP) team decides otherwise. Therefore, IEP teams need an objective way to determine whether print, braille, or a combination of both media should be a student's primary reading medium. The learning media assessment (LMA) process is used for determining the preferred learning and literacy medium or mediums for students with visual impairments, including those with additional disabilities (Koenig & Holbrook, 1989). The process was first formalized by Koenig and Holbrook (1995) with the text, Learning Media Assessment of Students with Visual Impairments: A Resource Guide for Teachers, but other tools have been developed since the first LMA tool was introduced in the mid-1990s. Two tools include National Reading Media Assessment (National Federation of the Blind, 2012) and Functional Vision and Learning Media Assessment (Sanford & Burnett, 2008). One of the tools is designed to be brief (National Federation of the Blind, 2012) while the other two tools call for extended administration (Koenig & Holbrook, 1995; Sanford & Burnett, 2008). These three LMA tools vary in format, categories of assessment, and the accommodations that assessors should make available to the individuals being assessed. The variation in LMA tools, procedures, and accommodations leads to a lack of specificity as to content areas that should be assessed, procedures that should be implemented during testing, and whether accommodations (e.g., prescription optical devices, preferential viewing) should be permitted during the LMA process.

According to Rosenblum et al. (2021), 17.73% of a sample of 79 teachers of students with visual impairments are not currently using a published instrument at all for structuring LMAs. Among participants who were using a published tool, the three most common tools were Koenig and Holbrook (1995), Sanford and Burnett (2008), and Oregon Project for Preschool Children who are Blind or Visually Impaired (Anderson et al., 2007). One of these tools, Anderson et al. (2007), was not actually designed for the purpose of conducting an LMA. These findings highlight the confusion surrounding preferred practices for conducting an LMA and threatens the content validity of the LMA.

Content validity is defined as the judgement of how well an instrument or tool evaluates the variables or characteristics that it purports to measure. A lack of content validity means it is unclear whether the content being assessed is appropriate or not. In other words, is the result achieved from the tool meaningful, and does the result allow accurate conclusions to be drawn (Mohamad et al., 2015)? When the content validity of a tool has not been examined, vital items might be missing from the assessment. Also, irrelevant items may be present on the assessment decreasing the internal consistency, affecting structural validity, and clouding interpretability (Terwee et al., 2018).

One method of establishing content validity is using the Delphi research technique (Alamanasreh et al., 2019; Colton & Hatcher, 2004; Roach et al., 2005). The Delphi research technique is a form of consensus building among experts using a survey format where panelists vote on statements or questions. Panelists are usually selected on the basis of expertise on a topic; published Delphi studies typically have between 10 and 50 panelists, depending on the complexity of the topic and availability of panelists with expertise (Nasa et al., 2021). Delphi surveys typically result in three or more rounds of voting on statements or questions, updating the document based on panelist votes and comments, and resending the survey for another vote. Each round of voting and modifying the questionnaire based on panelist votes and comments aids in the consensus building process (Avella, 2016; Colton & Hatcher, 2004). Delphi studies have previously been used to establish content or face validity for motor skill perception questionnaires (Stribing et al., 2022) and to set standards that guide teachers of students with visual impairments in the functional vision assessment (FVA) process (Kaiser et al., 2020). A Delphi study was used to establish the content to be included on the Birth to 6 Orientation and Mobility Skills Inventory (B6OMSI; Baguhn, 2021). Delphi studies have also been used to reach consensus on the concepts and skills that O&M specialists should teach (Wall Emerson & Corn, 2006) and assistive technology competencies for teachers of students with visual impairments (Smith et al., 2009). Further, the Delphi process was used to determine quality literacy instruction for students with visual impairments (Corn & Koenig, 2002; Koenig & Holbrook, 2000).

In light of the variation in LMA tools, the lack of consistency surrounding the process, and a paucity of content validity results, a Delphi study was conducted to determine what the critical components of the LMA process were and, by extension, the construction of a framework for measuring the content validity of LMA tools. The following questions guided the study:
What content areas should be included during the LMA process?

What primary procedures should be followed during the LMA process?

What materials, technology and accommodations are considered necessary or acceptable for use during the LMA process?

Method

Design

A Delphi study is an appropriate method in health and human services research to determine current best practices in a professional field (Falzarano & Zipp, 2013). Establishing consensus among a panel of experts on a list of answers to a research question demonstrates face validity and expert content validity. A Delphi approach is an appropriate research design when insufficient literature on a topic exists to otherwise establish validity (Falzarano & Zipp, 2013). The literature on learning media or assessment validation in the field of vision is not extensive enough to pursue other methods of validation currently. Approval was obtained from the Human Subjects Institutional Review Board at the University of Pittsburgh, Pittsburgh, Pennsylvania.

Panelists

Panelists were recruited by sending an email invitation with a recruitment script to identified experts in LMA. Experts were identified as individuals who were teachers of visually impaired students (TVIs) and who had developed and published LMA tools or published books or peer-reviewed articles on LMA with children who are visually impaired within the past 10 years. Of an initial list of 15 potential LMA experts, 12 were active in training TVIs at universities on the LMA process. Of the 15 potential experts, 8 agreed to take part and completed the first round of the study. Six panelists completed the entire Delphi process however, one of the strengths of a Delphi study is that the results are based on the level of consensus rather than the size of the expert pool (Avella, 2016; Falzarano & Zipp, 2013; Powell, 2003). Potential panelists who did not take part in the study were generally younger professionals and those who took part were generally professionals who had been involved in braille literacy and LMA issues for many years. Consent was obtained from all panelists before beginning the first round of the Delphi process.

Materials and Procedures

Each round of the survey was conducted via email. Previously, researchers who have used a Delphi study design found that giving panelists a list of topics in the first round of a Delphi study was a limitation, as it may influence people's responses toward only the suggested topics (Wall Emerson & Corn, 2006). To avoid this limitation, the first round of the current survey was made up of only two open-ended questions: what items need to be included in the LMA process and what modifications are acceptable in the LMA process. Panelists in the first round were asked to answer these questions based on their professional expertise, without looking at existing LMA tools. Panelists were asked to complete the first round within 2 weeks. The comprehensive list of items for round two was generated from the responses of the group.

In order to make the large list of generated items more readable by the panel of experts, the survey was organized into five sections or categories made up of similarly oriented items (overall assessment goals, items related to assessment procedures, items pertaining to student characteristics, items regarding student reading characteristics, and allowable modifications to assessment procedures). Each item was stated, then followed by check boxes labeled “keep,” “drop,” or “change,” with space provided for an indication of what needed to be changed. Each section concluded with an open-ended question asking for any new items to add to that section. Panelists were also told that they could move items between categories.

Subsequent rounds included a revised list, based on panelist consensus from the previous round, edits made to items, the voting record for each item from the previous round, and any rationales panelists gave for wanting to drop or change items. After each round, the level of agreement was tallied, and items were accepted, rejected, or sent back to the group for another round. Items with 80% consensus of “keep” were accepted into the final list and not included in the next round of voting. This level of consensus allowed for one member of the panel of experts to disagree on inclusion of any given item, but maintained a high level of agreement however, of the final list, only 13 items had less than 100% consensus on inclusion. Items with less than 30% consensus were considered rejected by consensus in these rounds, with the exception of newly submitted items, as this demonstrated that over half of the panelists did not support inclusion of the item. Note that this 30% level included any panelists who elected to keep or change the item and so reflected 70% of panelists wanting to drop an item. Intermediate values or items that underwent changes beyond grammatical alteration were returned to the group for further consideration. Each round had progressively fewer items due to the number of items accepted each round. After five rounds of the Delphi study were completed, two items remained with minor changes that were accepted by the researchers without the items beings sent back to the panelists.

Results

During the expert recruitment phase, from an initial potential expert list of 15 professionals, 12 individuals expressed interest in participating in the Delphi study, and 8 eventually agreed to take part. Eight panelists completed rounds 1 and 2, 6 completed round 3, and 7 completed rounds 4 and 5. The initial request for expert input was sent out in January of 2020 and responses were received between March and May of 2020. Due to the COVID-19 pandemic placing extra burdens on panelists’ time and energy, round 2 responses were not all gathered until November of 2020. Round 3 responses were received between February and March of 2021, round 4 responses in May of 2021, and round 5 responses were received in September and October of 2021.

The 8 panelists in the first round generated 88 items that were grouped into 5 sections and sent out in round 2. After removing items that reached consensus or that were dropped, round 3 contained 49 items with moderate consensus in round 2 or that had been significantly changed and eight new items. Round 4 contained 30 items with moderate consensus (more than 30% but less than 80%) in round 3 or that had been significantly changed. Round 5 contained seven items with moderate consensus in round 4 or that had been significantly changed. The 85 items that reached consensus among the panel members are organized into two goals for the LMA process, 6 start-up items to be considered in the LMA process, 15 LMA procedural characteristics, 20 items about information to be gathered in the LMA process, 8 items about student medical or visual characteristics to be noted, 10 items about student reading or academic characteristics, 6 items about student technology or literacy tool use, and 18 acceptable modifications to the LMA process. Tables 1 –8 shows these items by category with items organized by the round in which they were accepted. Note that the language used in each goal and item reflect the wording agreed on by the expert panel.

Table 1.
Agreed Upon Goals of the LMA process.

Accepted goal wording Agreement round and level

To support decision making related to providing a child access to literacy and learning medium to be used in instruction and independent application of skills 4 (100%)

To gather data that will assist in determining the literacy and learning medium/media most useful for a student, both at the present time and for future needs 4 (83%)

Table 2.
Agreed Upon Start-Up Items to be Noted in the LMA Process.

Accepted item wording Agreement round and level

Is this an initial or continuing assessment? 2 (100%)

Assessment dates and time of day 2 (100%)

Environment of assessment, especially lighting 2 (100%)

Instruments and procedures used during LMA 2 (100%)

Ensure and note that the LMA was conducted by a qualified teacher of students with visual impairment who is trained in the LMA process and understands the educational, social, cultural, linguistic, emotional and personal implications of how a visual impairment blindness impacts the child in these areas. 4 (100%)

Describe any visual clutter in the environment that impacts the student's literacy experience during the assessment 4 (100%)

Table 3.
Agreed Upon Procedural Characteristics of the LMA Process.

Accepted item wording Agreement round and level

Repeat LMA when any changes occur that put literacy program or medium in question or when additional literacy tools are needed. 2 (100%)

Differentiate LMA process for children at the beginning of their literacy program (making a decision about literacy media) compared to children who have a literacy media/medium (evaluating effectiveness of the current media). 2 (100%)

Update LMA at least annually, or as needed, as part of ongoing assessment. 4 (100%)

Ensure all appropriate forms are completed with fidelity as per procedures. 4 (100%)

Conduct the LMA process in familiar and unfamiliar environments. 4 (100%)

Consider the student's individual characteristics and plan assessment procedures with those characteristics in mind. 4 (100%)

Use a team approach with input from the student, as appropriate, and ensure that other educational team members, including parents or guardians, are given the opportunity to provide input to the child's strengths and needs as they pertain to literacy. 4 (100%)

When gathering data regarding the student's reading ability, examine how the student identifies words and how the student reads connected text. Use word lists to evaluate a student's ability to immediately recognize and/or decode words, and use reading passages (connected text) to begin collecting data on reading speed, accuracy, fluency, and miscue analysis. 4 (100%)

Note time requirements for completing literacy tasks. 4 (100%)

Use reading material in different sizes, fonts, and attributes, including photos, illustrations, etc. based on the student's individual abilities (for the ongoing LMA assessment of students who are already using print), noting the sizes and fonts used. 4 (86%)

Collect LMA data over a period of time and in varied environments. 5 (100%)

The LMA process should either be performed as part of the eligibility determination process or immediately after a child is identified as “visually impaired” in order to assist the educational team in supporting the student's literacy development. 5 (100%)

In the recommendations, use statements that are based on assessment findings and that are written to be accessible to family members and other members of the educational team. 5 (100%)

For a print reader, compare the child's digital reading ability to text printed on paper. Indicate how the digital text was displayed (e.g., color of font, background color, type of device). 5 (100%)

Measure the child's reading rate periodically in order to measure growth over time. Include a section for previous reading speed, accuracy, and comprehension from other LMAs if this is not the first time you are getting a reading rate on the child. 5 (86%)

Table 4.
Agreed Upon Information To Be Gathered in the LMA Process.

Accepted item wording Agreement round and level

Interviews with, and suggestions from, families. 2 (100%)

Interviews with, and suggestions from, teachers. 2 (100%)

Interviews with, and suggestions from, other specialists. 2 (100%)

Oral reading speed. 2 (100%)

Oral reading accuracy. 2 (100%)

Oral reading comprehension 2 (100%)

Oral reading grade level of text that is mastery/independent, instructional, and frustration level. 2 (100%)

Silent reading speed. 2 (100%)

How the child responds to sensory information in their natural environment. 2 (100%)

How the child accesses information visually, tactually, or auditorily at close range and at a distance. 2 (100%)

How the child uses sensory channels as shown by concrete, observable behaviors (noting that current use of sensory information is not the only indication of learning medium or media). 4 (100%)

Child's reading fluency (e.g., speed, accuracy, prosody, comprehension). 4 (100%)

An analysis of the student's miscues when reading orally (include braille miscues). 4 (100%)

Listening comprehension from an informal reading inventory (such as the Basic Reading Inventory or others). 4 (100%)

Handwriting samples were collected from the student and reviewed for facility in handwriting (e.g., are letters made quickly and accurately, what is the size of lettering). Indicate whether the child can read their own handwriting. 4 (100%)

Collect information about reading attitudes/motivation (e.g., Does student read for leisure? Does the student enjoy reading? Does the child enjoy being read to? What is the child's preferred media?). 4 (86%)

Note how much time it takes the child to complete class assignments in the medium compared to peers. 4 (86%)

Describe any fatigue demonstrated by the child when performing visual tasks. Indicate how fatigue was measured and length of time and tasks involved. 4 (86%)

Assess the student's comprehension after silent reading using a tool such as an informal reading inventory. 4 (86%)

Document evidence of reading fatigue using quantitative and qualitative data collected through interviews with family and teachers, observations, and reading speed comparisons. 5 (100%)

Table 5.
Agreed Upon Student Medical or Visual Characteristics to Be Gathered in the LMA Process.

Accepted item wording Agreement round and level

Child's chronological age 2 (100%)

Medical information (additional cognitive, medical, or health related disabilities) 2 (100%)

Medications and levels 2 (100%)

Child's cause of visual impairment and implications of the visual impairment (e.g., possibility that the visual impairment will progress) 2 (100%)

Child's visual impairment in terms of type of vision loss (e.g., central or peripheral vision loss) 2 (100%)

Results of the child's clinical vision evaluation 4 (100%)

Child's functional vision 4 (100%)

Child's natural visual working distance and size preferences (font size and size of objects) 4 (100%)

Table 6.
Agreed Upon Student Reading or Academic Characteristics to Be Gathered in the LMA Process.

Accepted item wording Agreement round and level

Child's academic achievement, grade reading level 2 (100%)

How the child demonstrates interest in reading and writing 2 (100%)

Student work flow expectations and how student participates (e.g., use of Google Classroom, student ability to hand in work independently on time) 2 (100%)

Indicate whether the child is an emergent reader or reader (with indication of how the assessment is made) 4 (100%)

Indicate the child's ability to read and communicate through writing including the means of writing (e.g., handwriting, using technology, etc.) 4 (100%)

Indicate how the child accesses subject-matter (grade level)texts and how available texts are 4 (100%)

Indicate the child's access to written mathematics materials (e.g., reading math textbook and materials, electronic/digital materials, graphs and images, etc.) 4 (100%)

Indicate whether the child knows the alphabet and how this is demonstrated (e.g., sees and names letters, tactually reads letters and names them, recites alphabet by rote) 4 (86%)

Assess the child's ability to read from a distance with indication of the distance and media (e.g., white board, media projection) 4 (86%)

Assess the child's ability to copy information presented at a distance (note distance, media used to present information, accuracy of copying, use of any prescribed device, and how quickly) 4 (86%)

Table 7.
Agreed Upon Student Technology and Literacy Tool Use in the LMA Process.

Accepted item wording Agreement round and level

Note the child's use of prescribed optical devices 2 (100%)

Note the child's use of “low tech” devices such as reading stands, copy holders 2 (100%)

Note what literacy tools the child uses and what the availability of the tools are 4 (100%)

Consider the child's current and future media needs when making recommendations 4 (100%)

Note current accommodations to access the curriculum that has been put in place (and how they are working) 4 (100%)

Note accommodations necessary for access to standardized tests and make recommendations for needed accommodations (to add to IEP) 4 (86%)

Table 8.
Agreed Upon Acceptable Modifications or Adjustments in the LMA Process.

Accepted item wording Agreement round and level

Devices to hold reading materials if child has motor difficulties 2 (100%)

Shorter time periods 2 (100%)

Frequent breaks, include as necessary 2 (100%)

Use of familiar objects instead of novel ones 2 (100%)

Positioning of materials (e.g., not at midline, placed in best place for physical response) 2 (100%)

High contrast between items and background 2 (100%)

Having a co-assessor familiar with the student 2 (100%)

For children who are non-verbal, looking for subtle physical signs such as changes in breathing, position, and gaze 2 (100%)

Being aware of latency, more wait time required so child can respond 2 (100%)

Use of braille, large print, or magnification 2 (100%)

Use of schedules (pictures, objects, etc.) 2 (100%)

Conduct the LMA process often enough to allow for robust assessment of the child's progress 2 (100%)

Braces, supports, or modified chair if child has inability to maintain posture 4 (100%)

What tools are used by the student (e.g., object calendar, tactile symbols, picture books, picture symbols, and other ACC tools) 4 (100%)

Which background (e.g., black background) the child performs best with. 4 (100%)

Documentation made of all accommodations the child typically uses to access information (and allowed during the LMA) 4 (100%)

Use of a sensory assessment tool for students with significant intellectual disability, including the names of specific products used 4 (86%)

Document how the student uses technology for various literacy-based tasks 4 (86%)

Discussion

This Delphi study identified key components to be considered and included when conducting a valid LMA. This framework of how to conduct the LMA and what elements to include can be used to assess the content validity of any LMA tool for students who are visually impaired. This study is the first documentation of consensus around LMA administration among experts in the area.

Content

The first research question that guided this study asked what content areas should be included in the LMA process. As many assessments do, the LMA process must begin with the child's history, social, medical, educational, and functional data about vision. A sensory channels analysis should be conducted with attention to tactile, auditory, and visual channels. Detailed and comprehensive information about reading level, skills, and comprehension, as well as writing and other output and creation options must be gathered. Functional data from classroom work in a range of media should be incorporated as available. Information about print or digital media should include size and other enhancements as discussed below. Proficiency with optical and technological devices the student currently uses to access materials should be evaluated as well. The experts identified information that should be gathered during the LMA process and reached consensus on items including information gathered, student reading and academic characteristics and coalescing around the two goals for the LMA. It is clear that working toward data-driven decision making for completing the LMA process can direct the rest of the assessment process.

Procedures

The second research question that guided this study sought information on how the LMA process should be conducted. Panelists identified several key topics, including a thorough documentation of instruments used, testing environment, examiner and student information, and time spent. The panel agreed on key considerations for the reporting of results to families and other stakeholders. The required planning and ongoing nature of assessment and reassessment were outlined. They identified the team who should be included in providing information related to a student's reading, writing, listening, and other literary needs.

Accommodations

When looking at the materials, technology, and accommodations considered necessary and acceptable for use during the LMA process, items fell under two lists: the technology and literacy tools and the acceptable modifications and or adjustments. Items related to this topic include specialized materials, technology and equipment that may be necessary to produce and access brailled or written material for students who are visually impaired. Students’ testing should be timed according to the child's needs, including allowing for visual and response latency. Positioning, seating, contrast, and lighting should all be customized to the student. A co-assessor familiar to the student, preferred materials from their environment, and all accommodations typically used by the child should be part of the testing procedures, and clearly documented and reported. These may include picture or tactile schedules, AAC devices, and other accommodations related or unrelated to the visual needs. Although this study provides a list of expert-agreed-upon accommodations, other, personalized accommodations should not necessarily be excluded. If additional accommodations are used, they should be noted in the LMA report to assist with interpretation since additional accommodations may impact the validity of the LMA process. The issue of accommodations gets to the heart of where many of the discrepancies have occurred among LMA tool developers. The researchers are confident the consensus achieved by the panelists in this area may lead to LMA tools that are inclusive and designed to meet the needs of students who are visually impaired across a wide range of ages and academic abilities and challenges.

Implications for Practice

According to Rosenblum et al. (2021), a survey of teachers of students with visual impairments indicated that one of the most commonly used LMA tools was not designed for that purpose and that 17.73% of teachers of students with visual impairments were not using any formal assessment tool to conduct the LMA. The confusion over which type of tool to use may stem from the lack of a validated LMA tool. Practitioners can use this framework built by a Delphi process to evaluate the validity of LMA tools they use by asking if all the validated content areas are addressed, if validated procedures are being followed, and if validated accommodations are in place.

Limitations

The study panelists included many of the foremost professionals who have developed or refined the LMA process, developed LMA tools, and published in the area of LMA. Practicing teachers of students with visual impairments, however, were not included in the field of experts. The group began with eight panelists with six completing all rounds of the project. The study occurred during the pandemic and often many months passed between rounds. Demands on panelists’ time and attention created the need to send multiple reminders to obtain as many panelist responses as possible. The lengthy time between rounds may have allowed panelists to think more deeply about items to add but most likely led to panelists forgetting about the study and then having to become reacquainted with the intent as they read a list of returned items.

Future Directions

Future research should apply the framework outlined in this paper to assess the validity of currently existing LMA tools. The evaluation of these efforts should include practicing teachers of students with visual impairments. Similarly, the new framework for a valid LMA process can be used by LMA tool developers to assess and improve the current tools to include all identified items in the framework. This LMA framework can be used by teachers of students with visual impairments to provide assurance that the LMA tool they have selected is measuring what they expect it to measure and that they can be confident that their recommendations regarding use of braille and any other media indicated as being useful for a student with visual impairment is based on an accepted assessment process that assures validity of findings.

Accepted goal wording	Agreement round and level
To support decision making related to providing a child access to literacy and learning medium to be used in instruction and independent application of skills	4 (100%)
To gather data that will assist in determining the literacy and learning medium/media most useful for a student, both at the present time and for future needs	4 (83%)

Accepted item wording	Agreement round and level
Is this an initial or continuing assessment?	2 (100%)
Assessment dates and time of day	2 (100%)
Environment of assessment, especially lighting	2 (100%)
Instruments and procedures used during LMA	2 (100%)
Ensure and note that the LMA was conducted by a qualified teacher of students with visual impairment who is trained in the LMA process and understands the educational, social, cultural, linguistic, emotional and personal implications of how a visual impairment blindness impacts the child in these areas.	4 (100%)
Describe any visual clutter in the environment that impacts the student's literacy experience during the assessment	4 (100%)

Accepted item wording	Agreement round and level
Repeat LMA when any changes occur that put literacy program or medium in question or when additional literacy tools are needed.	2 (100%)
Differentiate LMA process for children at the beginning of their literacy program (making a decision about literacy media) compared to children who have a literacy media/medium (evaluating effectiveness of the current media).	2 (100%)
Update LMA at least annually, or as needed, as part of ongoing assessment.	4 (100%)
Ensure all appropriate forms are completed with fidelity as per procedures.	4 (100%)
Conduct the LMA process in familiar and unfamiliar environments.	4 (100%)
Consider the student's individual characteristics and plan assessment procedures with those characteristics in mind.	4 (100%)
Use a team approach with input from the student, as appropriate, and ensure that other educational team members, including parents or guardians, are given the opportunity to provide input to the child's strengths and needs as they pertain to literacy.	4 (100%)
When gathering data regarding the student's reading ability, examine how the student identifies words and how the student reads connected text. Use word lists to evaluate a student's ability to immediately recognize and/or decode words, and use reading passages (connected text) to begin collecting data on reading speed, accuracy, fluency, and miscue analysis.	4 (100%)
Note time requirements for completing literacy tasks.	4 (100%)
Use reading material in different sizes, fonts, and attributes, including photos, illustrations, etc. based on the student's individual abilities (for the ongoing LMA assessment of students who are already using print), noting the sizes and fonts used.	4 (86%)
Collect LMA data over a period of time and in varied environments.	5 (100%)
The LMA process should either be performed as part of the eligibility determination process or immediately after a child is identified as “visually impaired” in order to assist the educational team in supporting the student's literacy development.	5 (100%)
In the recommendations, use statements that are based on assessment findings and that are written to be accessible to family members and other members of the educational team.	5 (100%)
For a print reader, compare the child's digital reading ability to text printed on paper. Indicate how the digital text was displayed (e.g., color of font, background color, type of device).	5 (100%)
Measure the child's reading rate periodically in order to measure growth over time. Include a section for previous reading speed, accuracy, and comprehension from other LMAs if this is not the first time you are getting a reading rate on the child.	5 (86%)

Accepted item wording	Agreement round and level
Interviews with, and suggestions from, families.	2 (100%)
Interviews with, and suggestions from, teachers.	2 (100%)
Interviews with, and suggestions from, other specialists.	2 (100%)
Oral reading speed.	2 (100%)
Oral reading accuracy.	2 (100%)
Oral reading comprehension	2 (100%)
Oral reading grade level of text that is mastery/independent, instructional, and frustration level.	2 (100%)
Silent reading speed.	2 (100%)
How the child responds to sensory information in their natural environment.	2 (100%)
How the child accesses information visually, tactually, or auditorily at close range and at a distance.	2 (100%)
How the child uses sensory channels as shown by concrete, observable behaviors (noting that current use of sensory information is not the only indication of learning medium or media).	4 (100%)
Child's reading fluency (e.g., speed, accuracy, prosody, comprehension).	4 (100%)
An analysis of the student's miscues when reading orally (include braille miscues).	4 (100%)
Listening comprehension from an informal reading inventory (such as the Basic Reading Inventory or others).	4 (100%)
Handwriting samples were collected from the student and reviewed for facility in handwriting (e.g., are letters made quickly and accurately, what is the size of lettering). Indicate whether the child can read their own handwriting.	4 (100%)
Collect information about reading attitudes/motivation (e.g., Does student read for leisure? Does the student enjoy reading? Does the child enjoy being read to? What is the child's preferred media?).	4 (86%)
Note how much time it takes the child to complete class assignments in the medium compared to peers.	4 (86%)
Describe any fatigue demonstrated by the child when performing visual tasks. Indicate how fatigue was measured and length of time and tasks involved.	4 (86%)
Assess the student's comprehension after silent reading using a tool such as an informal reading inventory.	4 (86%)
Document evidence of reading fatigue using quantitative and qualitative data collected through interviews with family and teachers, observations, and reading speed comparisons.	5 (100%)

Accepted item wording	Agreement round and level
Child's chronological age	2 (100%)
Medical information (additional cognitive, medical, or health related disabilities)	2 (100%)
Medications and levels	2 (100%)
Child's cause of visual impairment and implications of the visual impairment (e.g., possibility that the visual impairment will progress)	2 (100%)
Child's visual impairment in terms of type of vision loss (e.g., central or peripheral vision loss)	2 (100%)
Results of the child's clinical vision evaluation	4 (100%)
Child's functional vision	4 (100%)
Child's natural visual working distance and size preferences (font size and size of objects)	4 (100%)

Accepted item wording	Agreement round and level
Child's academic achievement, grade reading level	2 (100%)
How the child demonstrates interest in reading and writing	2 (100%)
Student work flow expectations and how student participates (e.g., use of Google Classroom, student ability to hand in work independently on time)	2 (100%)
Indicate whether the child is an emergent reader or reader (with indication of how the assessment is made)	4 (100%)
Indicate the child's ability to read and communicate through writing including the means of writing (e.g., handwriting, using technology, etc.)	4 (100%)
Indicate how the child accesses subject-matter (grade level)texts and how available texts are	4 (100%)
Indicate the child's access to written mathematics materials (e.g., reading math textbook and materials, electronic/digital materials, graphs and images, etc.)	4 (100%)
Indicate whether the child knows the alphabet and how this is demonstrated (e.g., sees and names letters, tactually reads letters and names them, recites alphabet by rote)	4 (86%)
Assess the child's ability to read from a distance with indication of the distance and media (e.g., white board, media projection)	4 (86%)
Assess the child's ability to copy information presented at a distance (note distance, media used to present information, accuracy of copying, use of any prescribed device, and how quickly)	4 (86%)

Accepted item wording	Agreement round and level
Note the child's use of prescribed optical devices	2 (100%)
Note the child's use of “low tech” devices such as reading stands, copy holders	2 (100%)
Note what literacy tools the child uses and what the availability of the tools are	4 (100%)
Consider the child's current and future media needs when making recommendations	4 (100%)
Note current accommodations to access the curriculum that has been put in place (and how they are working)	4 (100%)
Note accommodations necessary for access to standardized tests and make recommendations for needed accommodations (to add to IEP)	4 (86%)

Accepted item wording	Agreement round and level
Devices to hold reading materials if child has motor difficulties	2 (100%)
Shorter time periods	2 (100%)
Frequent breaks, include as necessary	2 (100%)
Use of familiar objects instead of novel ones	2 (100%)
Positioning of materials (e.g., not at midline, placed in best place for physical response)	2 (100%)
High contrast between items and background	2 (100%)
Having a co-assessor familiar with the student	2 (100%)
For children who are non-verbal, looking for subtle physical signs such as changes in breathing, position, and gaze	2 (100%)
Being aware of latency, more wait time required so child can respond	2 (100%)
Use of braille, large print, or magnification	2 (100%)
Use of schedules (pictures, objects, etc.)	2 (100%)
Conduct the LMA process often enough to allow for robust assessment of the child's progress	2 (100%)
Braces, supports, or modified chair if child has inability to maintain posture	4 (100%)
What tools are used by the student (e.g., object calendar, tactile symbols, picture books, picture symbols, and other ACC tools)	4 (100%)
Which background (e.g., black background) the child performs best with.	4 (100%)
Documentation made of all accommodations the child typically uses to access information (and allowed during the LMA)	4 (100%)
Use of a sensory assessment tool for students with significant intellectual disability, including the names of specific products used	4 (86%)
Document how the student uses technology for various literacy-based tasks	4 (86%)

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Dawn Anderson

References

Alamanasreh

Moles

Chen

(2019). Evaluation methods used for establishing content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066

Anderson

Boigon

Davis

deWaard

(2007). Oregon Project for Preschool Children who are Blind or Visually Impaired (6^th ed.). Southern Oregon Education Service District.

Avella

J. R.

(2016). Delphi panels: Research design, procedures, advantages, and challenges. International Journal of Doctoral Studies, 11, 305–321. Retrieved from http://www.informingscience.org/Publications/3561. https://doi.org/10.28945/3561

Baguhn

(2021). Early intervention orientation and mobility: A delphi study of the content of the birth to 6 orientation and mobility skills inventory (B6OMSI). Journal of Visual Impairment & Blindness, 115(5), 361–367. https://doi.org/10.1177/0145482X211047626

Colton

Hatcher

(2004). The Web-based Delphi research technique as a method for content validation in HRD and adult education research. Online Submission. Retrieved from https://eric.ed.gov/?id=ED492146

Corn

A. L.

Koenig

A. J.

(2002). Literacy for students with low vision: A framework for delivering instruction. Journal of Visual Impairment & Blindness, 96(5), 305–321. https://doi.org/10.1177/0145482X0209600503

Falzarano

Zipp

G. P.

(2013). Seeking consensus through the use of the delphi technique in health sciences research. Journal of Allied Health, 42(2), 99–105.

Individuals with Disabilities Education Act. 20 U.S.C. § 1400 (2004).

Kaiser

J. T.

Rosenblum

L. P.

Herzberg

T. S.

(2020). Building consensus about the functional visual assessment process: A delphi study. Journal of Visual Impairment & Blindness, 114(6), 461–474. https://doi.org/10.1177/0145482X20971937

10.

Koenig

A. J.

Holbrook

M. C.

(1989). Determining the Reading medium for students with visual impairments: A diagnostic teaching approach. Journal of Visual Impairment & Blindness, 83(6), 281–309. https://doi.org/10.1177/0145482X8908300609

11.

Koenig

A. J.

Holbrook

M. C.

(1995). Learning media assessment of students with visual impairments: A resource guide for teachers (2^nd ed.). Texas School for the Blind and Visually Impaired.

12.

Koenig

A. J.

Holbrook

M. C.

(2000). Ensuring high-quality Instruction for students in braille literacy programs. Journal of Visual Impairment & Blindness, 94(11), 677–694. https://doi.org/10.1177/0145482X0009401102

13.

Mohamad

M. M.

Sulaiman

N. L.

Sern

L. C.

Salleh

K. M.

(2015). Measuring the validity and reliability of research instruments. Social and Behavioral Sciences, 204, 164–171. https://doi.org/10.1016/j.sbspro.2015.08.129

14.

Nasa

Jain

Juneja

(2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://doi.org/10.5662/wjm.v11.i4.116

15.

National Federation of the Blind. (2012). National reading media assessment. Retrieved from: http://www.nfbnrma.org

16.

Powell

(2003). The delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382. https://doi.org/10.1046/j.1365-2648.2003.02537.x

17.

Roach

A. T.

Elliott

S. N.

Webb

N. L.

(2005). Alignment of alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin alternate assessment. The Journal of Special Education, 38(4), 218–231. https://doi.org/10.1177/00224669050380040301

18.

Rosenblum

L. P.

Herzberg

T. S.

Mason

Anderson

Reisman

Edstrand

Abner

Carter

(2021). Learning media assessment experiences of teachers of students with visual impairments. Journal of Visual Impairment & Blindness, 115(1), 55–62. https://doi.org/10.1177/0145482X20987018

19.

Sanford

Burnett

(2008). Functional vision and learning media assessment. American Printing House for the Blind.

20.

Smith

D. W.

Kelley

Mausak

N. J.

Griffin-Shirley

Lan

W. Y.

(2009). Assistive technology competencies for teachers of students with visual impairment. Journal of Visual Impairment & Blindness, 103(8), 457–469. https://doi.org/10.1177/0145482X0910300804

21.

Stribing

Stodden

D. F.

Monsma

Lieberman

L. J.

Brian

A. S.

(2022). Content/face validity of motor skill perception questionnaires for youth with visual impairments: A delphi method. British Journal of Visual Impairment, 40(2), 369–377. https://doi.org/10.1177/0264619621990687

22.

Terwee

C. B.

Prinsen

C. A. C.

Chiarotto

Westerman

M. J.

Patrick

D. L.

Alonso

Bouter

L. M.

de Vet

H. C. W.

Mokkink

L. B.

(2018). COSMIN methodology for evaluating the content validity of patient-reported outcome measures: A delphi study. Quality of Life Research, 27, 1159–1170. https://doi.org/10.1007/s11136-018-1829-0

23.

Wall Emerson

Corn

(2006). Orientation and mobility content for children and youths: A delphi approach pilot study. Journal of Visual Impairment & Blindness, 100(6), 331–342. https://doi.org/10.1177/0145482X0610000604