Selecting Universal Screening Measures to Identify Students at Risk Academically

Identify the purpose

First and foremost, educators must identify the purpose of implementing a screening measure in their school. By doing so, measures that can collect the required data may be identified and selected. Too often, school personnel fail to consider the purpose of their screening measure prior to implementation and realize too late that they did not obtain the information needed (VanDerHeyden, 2013). Examples of these different purposes for screening might be (a) to identify students at risk for learning or other disabilities, (b) to individualize Tier 2 or 3 interventions, (c) to determine which students are high achievers, (d) to look at achievement in specific academic domains, or (e) to examine students’ performance in nonacademic areas. For instance, a school looking to identify students with deficits in basic skills areas could employ the System to Enhance Educational Performance screening measure for this target purpose, while professionals looking for a measure to determine which students have learning difficulties might use the preschool screening measure adapted by Scott and Delgado (Gansle & Noell, 2008; iSTEEP, 2009; Scott & Delgado, 2003, 2006; VanDerHeyden, 2013). In determining the purpose of the screening process, school professionals may want to consult particular staff members, such as the school psychologist, reading or mathematics specialists, or school counselors, to aid in identifying the specific academic needs of the students and, subsequently, the targeted purpose of the screening process.

Following the description of each substep outlined in this article, critical questions that should be considered by the selection committee and any consulted specialists will be listed to further direct professionals in how to successfully select an appropriate screening measure for their school. The critical questions for identifying the purpose of the screening measure are the following:

The answers to these questions will immediately narrow the field of potential screening measures. The response to Critical Question 1 will allow screening measures outside the target purpose to be eliminated. For example, if the purpose of the screening process is to identify students who are at risk for learning difficulties or disabilities, any screening measure that does not advertise this function may be discarded. Similarly, the answer to the second critical question can lead professionals to focus on screening measures developed for the target grade levels, further concentrating the pool of potential measures. The answers to these two questions should be considered immediately and used as a means of identifying potential screening measures to research. For each of the remaining substeps within Steps 1 and 2 of this process, the responses to the critical questions should be meticulously recorded, as they will be reviewed in depth in Step 3.

Determine available resources

School personnel must also determine the resources available for the screening process. Screening measures can vary in financial cost, time requirements, and personal labor (Kettler, 2007; Lilles et al., 2009; VanDerHeyden, 2013). Measures should be selected that use the minimum quantity of resources while ensuring the maximum level of effectiveness. Therefore, schools should decide what resources are available by consulting budget or resource specialists within the school and then select the most efficient and accurate screening measure accessible with these resources.

Critical questions to determine what resources are available are the following:

These critical questions, like those in the previous section, allow professionals to exclude various measures from the group under consideration. Specifically, the first question on available resources focuses the search on screening measures that will not exceed the defined financial limit. The selection process can be further concentrated with the second and third questions by eliminating any measures that require too much time or personnel for their administration. The fourth question in this segment will be helpful in identifying any other constraints that should be considered when identifying the best fit in Step 3.

Research required resources

After deciding what resources are available for the screening process, schools should research the resources needed for the implementation of existing screening measures. Ideally, schools should select the most efficient screening measure accessible with the available resources. One method of conducting universal screening that requires relatively low levels of financial resources and professional training for implementation is the teacher rating scale, which is employed by several established screening measures. Studies have also consistently found support for the accuracy of teacher rating scales to identify at-risk students (Gansle & Noell, 2008; Gray, McCallum, & Bain, 2009; Kettler, 2007; Kettler & Elliott, 2010). Teachers have the opportunity to interact with students on a daily basis and have the knowledge to compare students’ achievement levels to age- and grade-level norms (Kettler, 2007; Kettler & Elliott, 2010). They can also consider the whole child when rating him or her, outside of the narrow range of academic skills (Gray et al., 2009). As compared to spontaneous teacher referrals, these rating scales offer a much more structured platform through which teachers can evaluate their students’ performance and achievement. Often, screening measures that employ teacher rating scales as either the primary assessment or a subcomponent of an evaluation, as with the Academic Competency Evaluation Scales, are especially effective for use with younger students (DiPerna & Elliott, 2000; Kettler & Elliott, 2010). On the other hand, teacher rating scales require relatively high levels of labor resources, as teachers must complete an individual rating scale for each of their students, rendering these less useful for certain schools. However, there is a broad selection of measures that may be more applicable for schools with different types and quantities of resources available for the implementation of the screening measures.

Critical questions to investigate necessary resources are the following:

Professionals should record the answers to each critical question for future consideration in selecting a screening measure. The school staff should document the financial cost, the time requirement, the type and quantity of staff that will be needed to administer each of the measures, and any other resources necessary to implement the measures. At this point in the process, the only action required is the documentation of these responses, as the consideration and decision portion will be accomplished later in the procedure.

Look at multiple sources of information to explore students’ context

Another consideration when selecting a universal screening measure is the established effectiveness of gathering information from multiple sources. Though this, too, may require more resources in exchange for greater accuracy, it is something that should be specifically considered as it has been consistently supported in the literature. Studies have shown that accuracy can be increased by the use of both multiple procedures—such as standardized assessment, rating scales, and observations—and multiple informants—such as parents, teachers, other professionals, and the students themselves (Compton et al., 2010; Gray et al., 2009; Lilles et al., 2009).

Each screening method and each informant can comment only on portions of the student. By combining several of each, the school can gain a more complete picture of the student as a whole. Standardized, norm-referenced assessments gather information only on specific skills sets for comparison with age- or grade-level peers. This procedure fails to consider the child’s varied and nuanced strengths, weaknesses, and life experiences (Gansle & Noell, 2008; VanDerHeyden, 2013). For instance, though two students may receive the same score on a screening measure, one student may be much more resilient, responsible, or persistent than the other, meaning that these two students will require different types of support to achieve academically (Oakland & Rossen, 2005). By gathering information about the child as a whole, the real problems and possible solutions can be identified. Similarly, observations of students can provide information only about a student’s performance in a specific environment on a particular day. The perspectives of individual informants are also often lacking in observation- or rating scale–based screening methods. For instance, although parents have seen their child develop, they lack the ability to compare their child’s achievement to grade-level norms, while teachers have contrasting capabilities (Gray et al., 2009; Oakland & Rossen, 2005).

Furthermore, students’ backgrounds can affect their performance in school, either academically or behaviorally. Students from particular localized geographic areas often share similar backgrounds. By gathering information about the local geographic area or standardizing assessment measures for this region, some of the students’ contexts can be taken into consideration in the screening process (Oakland & Rossen, 2005; Scott & Delgado, 2003).

Critical questions to consider in looking at the sources of information that screening measures assess and how these measure address students’ context are the following:

Again, the answers to these questions should be recorded by school staff members. Ideally, measures should employ multiple procedures and informants, consider students’ backgrounds, and be standardized for local geographic norms. However, various screening measures will possess different combinations of strengths and weaknesses that can be weighed in more depth in Step 3 of the screening measures selection process. The developers of some measures may not present the responses to some of the questions in this section and those in the following sections. In this case, professionals should attempt to locate these answers in external research or evaluations when possible. New measures may have undergone less research, resulting in insufficient information to answer these critical questions. The lack of information should be noted but should not necessarily be cause for elimination of the measure.

Examine relative biases of screening measures

School personnel should also consider the relative bias of possible screening measures. Such bias might be affected by variance in language proficiency, socioeconomic background, or previous education. Though some research has been conducted on the biases of various measures, this is still a difficulty for some established screening measures (Gray et al., 2009). In selecting screening measures for use in schools, professionals should consult the existing research on the biases of different measures. The simple presence of research on various biases for a screening measure may be evidence of its merit, as many measures lack any research on this topic. However, professionals should seek out all research on the matter and personally evaluate the according value of various screening measures. Some measures have been designed with the specific aim of reducing particular types of bias. For example, the Universal Nonverbal Intelligence Test–Gifted Screening Scale was developed to minimize test bias for children with communication difficulties or limited English language proficiency (Gray et al., 2009). Depending on school characteristics, the issue of bias may be more or less important to consider when selecting screening measures. Specifically, it is imperative to consider the quality of core instruction within a school so variance in prior education does not create bias in the evaluation of screening measures.

Critical questions to examine the relative biases of screening measures are the following:

Professionals should obtain more than a simple confirmation or negation for critical questions in this section. For instance, if the response to the first question is found to be affirmative, school staff should then record the languages in which the measure can be administered as well as any research on the quality of the measure in the alternate languages. Similarly, professionals should explore how a nonverbal alternate form would be administered and any research on the alternate form. The determined level of cultural bias and how this was established should also be recorded. It may be helpful to retain citations of any references used in answering these questions to allow for consultation in the decision-making process to follow. Similar to the responses in the prior section, the answers to these questions should not be used as a vehicle for immediate elimination; rather, they should be carefully weighed in the decision-making process. Preferably, each of these questions will be answered positively and supported by sufficient and quality research, although measures that are found to be less than ideal in this category may still be the best fit for a particular school due to strengths or capabilities in other areas.

Think about further assessments

When looking at screening measures, schools should also think about further assessments or evaluations to employ with students who score poorly on initial measures in order to determine the necessary supports or interventions for the individual students. Research supports the use of a three-tiered RTI approach to screening and to intervention, in which assessment and intervention become increasingly intensive as a child progresses through the tiers (Kettler, 2007). Typically developing students do not require evaluation or support beyond the first tier, while students with greater academic difficulties or disabilities will progress on to Tiers 2 and 3 as necessary (Kettler, 2007). Specifically, students who are identified as at risk by the initial, universal screening measure can be further evaluated through teacher, parent, or student interviews; more specialized or intensive assessment measures; careful review of the student’s previous development; or additional formal progress monitoring measures (Gansle & Noell, 2008; VanDerHeyden, 2013). Some screening measures, such as the Brief Academic Competence Evaluation Screening System, incorporate assessments for Tiers 2 and 3 to be used when necessary (Kettler, 2007; Kettler & Elliott, 2010). Through this process, individual supports or plans can be designed to help each student achieve to his or her potential. When selecting a screening measure, it is important for school personnel to identify the next steps for low-scoring students prior to the implementation of the measure.

Critical questions to study how measures address further assessments for low-scoring students are the following:

Often, the answers to these two questions can be drawn directly from the manual or from other publications written by the developers of the screening measures. If affirmative responses are found for either of these questions, professionals should record detailed descriptions of the further assessments or how the measures reflect three-tiered models, respectively. Measures with included further assessments for low-scoring students that are based on three-tiered models of assessment are preferred, although those offering specific external recommendations for further assessments are also valuable. Additional resources required for further assessments should be taken into account when weighing the measure’s costs and benefits.

Study test metrics

Finally, test metrics for potential screening measures should be carefully reviewed and considered when selecting a measure to implement in a school. Such metrics are often already present in the literature for professionals to research and utilize in comparing possible screening measures for use in their school. Several resources are also available to professionals in which these metrics for various measures have already been researched and compiled, such as the Center on Response to Intervention (2015) and Understood.org (2015). Important metrics to examine include tests of reliability, validity, sensitivity, specificity, and predictive power, which all affect the accuracy of screening measures. Reliability, for example, determines whether a screening measure consistently evaluates the same skill or construct. There are several types of reliability that schools can look for when selecting screening measures, such as item-total reliability, which is the correlation between individual items on a scale and the scale as a whole, and interrater reliability, which is the correlation between various raters’ or observers’ scores (Kettler, 2007). It is critical to look at reliability because if a screening measure does not dependably assess the same skill in the same way, then the results cannot be trusted.

Validity, on the other hand, is evidence of whether a screening measure actually evaluates what it sets out to evaluate. Types of validity to consider when selecting screening measures are construct validity, which is the extent to which a measure assesses what it intends to assess; concurrent validity, which is whether the results of the screening measure match up with results of other established sources; and predictive validity, which is the degree to which the screening measure matches up with a student’s later achievement or performance (Kettler, 2007; Scott & Delgado, 2003). Screening measures must also be valid for the target population. For instance, some measures may be valid only for specific grade levels or for certain language competencies. If screening measures are being selected for an elementary school composed of primarily English language learners, for example, the school personnel must examine the validity of measures for these specific populations.

Sensitivity and specificity are two metrics that measure how accurate a screening measure is in identifying the correct students. Sensitivity is a measure of the proportion of the students with learning disabilities or difficulties who are correctly identified by the screening measure, while specificity is a measure of how many children who are not at risk of academic difficulties are correctly identified by the screening measure (Scott & Delgado, 2003, 2006; VanDerHeyden, 2013). Once it is determined that reliability and validity levels are sufficiently high, sensitivity and specificity are the next metrics to check when selecting a screening measure. It is also important to know that as sensitivity increases, specificity decreases and vice versa, so the goal is to find the optimum balance between the two.

The final test metrics that will be discussed are the positive and negative predictive powers of screening measures. Although these measures are related to specificity and sensitivity, they are much more variable, depending on the participants, and measure probability of accuracy rather than the actual proportion of accurate identifications. Specifically, positive predictive power is a measure of the probability that a student identified by a screening measure as at risk academically actually does fall into this category, while negative predictive power is the probability that a student identified by the measure as typically developing is truly not at risk (VanDerHeyden, 2013). This metric would be one of the final measures to look at when selecting a screening measure because of its relative variance based on population and the dependence of its accuracy on the levels of reliability and validity of the measure. When selecting a screening measure, however, it is important to examine all metrics available to gain the greatest understanding possible before making a decision.

Critical questions in studying findings about test metrics are the following:

Responses to these questions are especially significant, as high values for these test metrics are necessary to ensure the function of a screening measure. Before answering the critical questions in this segment, professionals should review the meaning and importance of the various metrics. Further, it is important that the questions in this section are answered and considered in the order in which they are listed, as these values are influenced by those preceding. For instance, if the reliability is found to be poor in response to Question 1, it follows that the validity queried in Question 2 will also be lacking.