An Independent Evaluation of the Diagnostic Accuracy of a Computer Adaptive Test to Predict Proficiency on an End of Year High-Stakes Assessment

Abstract

Star Math (SM) is a popular computer adaptive test (CAT) schools use to screen students for academic risk. Despite its popularity, few independent investigations of its diagnostic accuracy have been conducted. We evaluated the diagnostic accuracy of SM based upon vendor provided cut-scores (25^th and 40^th percentiles nationally) in predicting proficiency on an end of year state test in a sample of highly achieving grade three (n = 210), four (n = 217), and five (n = 242) students. Specificity exceeded sensitivity across all grades and cut-scores. Acceptable levels of sensitivity and specificity were achieved in grade three and four but not grade five when using the 40^th percentile.

Keywords

mathematics screening computer adaptive testing response to intervention

Universal screening is a core component of Response to Intervention frameworks, providing a means of early identification which leads to the delivery of preventive supports (Kettler et al., 2014). Four outcomes exist when students complete a universal screening assessment: true positive (TP), false positive (FP), false negative (FN), and false positive (FP; Kettler et al., 2014). A student who is at-risk of failure who is incorrectly identified by a screener as not being at risk would be considered an FN. Conversely, students identified as at-risk whom are truly not at risk can be considered FPs. Ideal outcomes are TPs and TNs, where students who truly are and are not at risk, are identified accurately as such.

Sensitivity (Sn) and specificity (Sp) are two metrics to assess the diagnostic accuracy of a cut-score to identify students at-risk on the universal screener. Sensitivity is the proportion of students whom are truly at-risk identified as such. Specificity is the proportion of students truly not at-risk identified as such (Kettler et al., 2014). An ideal cut-score achieves an appropriate balance between Sn and Sp, and aligns with school resources and student need.

Math is an important area to accurately identify students in need of support, due to its ties to future academic success and high stakes exam performance (Nicholas & Berliner, 2007). Previous studies support the value of math screening tools in predicting students’ end of year performance on state exams (Fuchs et al., 2011). A common assessment used to screen for academic difficulties in math is Star Math (SM) published by Renaissance Learning (2017). Studies that consider diagnostic accuracy of SM are limited in that they are often provided by the publisher despite the tool being used by over 34,000 schools/districts in the country (Renaissance Learning, n.d.). The purpose of this study is to provide an independent evaluation of the diagnostic accuracy of default cut-scores from SM in a sample of elementary-aged students. Default cut-scores were selected to mimic how educators would use the tool in their school. Research questions include (a) To what degree is SM performance predictive of student performance on state exams (i.e., Pennsylvania System of School Assessment; PSSA)? and (b) Do default cut-scores effectively identify students who need additional support to succeed on state exams?

Method

Participants and Setting

Data were obtained from students in Grades 3–5 who attended an elementary school in southcentral Pennsylvania. The sample consisted of 669 students: grade three = 210; grade four = 217; and grade five = 242. Demographic characteristics are summarized in Table 1.

Table 1.

Demographic Characteristics Across Grades 3–5.

	Grade 3 (N= 210)	Grade 4 (N= 217)	Grade 5 (N=242)
Race/Ethnicity
% White	79.10	77.43	79.60
% Native Hawaiian/Other Pacific Islander	0.90	0.44	0.00
% African American	4.50	8.41	5.60
% Asian American	15.00	13.71	13.60
% American Indian/Alaskan Native	0.45	0.00	0.00
% Qualify free/Reduced lunch	31.82	26.99	24.80
% Enrolled in special education	17.73	11.50	17.20

Measures

Star Math: SM is a computer adaptive test (CAT) that measures math achievement in grades 1–12. It assesses competency in numbers and operations, algebra, geometry and measurement, and data analysis, statistics, and probability (Renaissance Learning, 2015). Assessments consists of approximately 34 questions from an item bank of over 6000 items and takes about 20-min to complete. Performance is represented as a scaled score from 0 to 1400. Reports provide grade specific benchmarks and expected growth rates. Split-half reliability was .91 for Grades 3 and 4, and .92 for Grade 5 (Renaissance Learning, 2021). Within-grade average predictive validity was .72, .73, and .75 for Grades 3, 4, and 5 (Renaissance Learning, 2021) There are various approaches to deciding cut-score placement. One involves using vendor or publisher-provided scores. Renaissance Learning provides default cut-scores that correspond to performance at approximately the 25^th (high risk) and 40^th (some risk) percentile at beginning of the school year for each grade level.

Pennsylvania System of School Assessment: The PSSA is an annually-administered standards-based, criterion-referenced assessment to measure academic achievement and proficiency of students in the state of Pennsylvania and served as the criterion measure in the study. All students Grades 3 through 8 take assessments in the areas of English Language Arts and Math, along with Science in Grades 4–8. Student performance falls within a category of “Below Basic,” “Basic,” “Proficient,” or “Advanced.” Categorization is determined by cut-scores provided by the Pennsylvania Department of Education (2020). In general, the target for end of year performance corresponds to “Proficient”.

Data Analysis

Descriptive statistics were calculated for each grade using the SM Fall scaled scores and PSSA Spring scaled scores. Pearson’s correlation between Fall SM scores, and Spring PSSA scores were also calculated across grades. Sensitivity and specificity were derived for each grade across vendor provided cut-scores for risk (e.g., 25^th and 40^th percentiles) predicting performance levels of “Basic” and “Proficient.” Sensitivity was derived by calculating the number of TPs divided by the total TPs plus FNs within the sample. Specificity was determined by calculating the total TNs divided by the sum of TNs and FPs. Positive and negative predictive values, or the ratio of students that were truly at-risk to all those identified as at-risk (TP/[TP+FP]), and the ratio of students that were truly not at-risk to those identified as not-at risk (TN/[TN+FN]) were also computed.

Results

Descriptive and Correlational

Descriptive statistics are displayed in Table 2. The present sample was high performing. Average performance on SM corresponded roughly to the 75th percentile nationally across grades, and mean PSSA performance was in the “Proficient” range across grades. The base rate of students below basic levels of performance was .05 in all three grades. The base rate of students performing below proficient was .27 in grade three, .24 in grade four, and .35 in grade five. The correlation between Fall SM and Spring PSSA scaled scores was calculated for each grade. SM and Spring PSSA scores were found to be strongly, positively correlated; in grade three, r =.76 (p < .001), grade four, r = .79 (p < .001), and grade five, r =.78 (p < .001).

Table 2.

Descriptive Statistics.

Grade	Mean	SD
Grade 3
SM scaled score	555.21	81.36
PSSA scaled score	1081.67	126.03
Grade 4
SM scaled score	651.01	96.77
PSSA scaled score	1083.01	121.74
Grade 5
SM scaled score	732.03	82.65
PSSA scaled score	1049.19	107.54

Note. SM – Star Math; PSSA – Pennsylvania System of School Assessment.

Diagnostic Accuracy

Diagnostic accuracy results are presented in Table 3. When predicting students at Basic levels of performance, Sp was greater than Sn across all grades for each cut-score. At the 40th percentile, Sn was relatively balanced with Sp, especially in grades 3 and 4. When cut-scores were placed at the 25th percentile, Sn decreased. Although Sn at the 25th percentile remained relatively acceptable for grades three and four, in grade five Sn dropped to 0.

Table 3.

Diagnostic Accuracy Results of Star Math predicting Levels of Performance on Pennsylvania System of School Assessments.

	Basic
Percentile	Grade	BR	TP	TN	FP	FN	Sn	Sp	PPV	NPV
25^th	3	.05	8	192	7	3	.73	.96	.53	.98
	4	.05	7	193	13	4	.64	.94	.35	.98
	5	.05	0	220	9	13	.00	.96	.00	.94
40th	3	.05	10	185	14	1	.91	.93	.42	.99
	4	.05	9	176	30	2	.82	.85	.23	.99
	5	.05	6	207	22	7	.46	.90	.21	.97
	Proficient
Percentile	Grade	BR	TP	TN	FP	FN	Sn	Sp	PPV	NPV
25^th	3	.27	14	152	1	43	.25	.99	.93	.78
	4	.24	16	160	4	37	.30	.98	.80	.81
	5	.35	7	156	2	77	.08	.99	.78	.67
40th	3	.27	22	151	2	35	.39	.99	.92	.81
	4	.24	27	152	12	26	.51	.93	.69	.85
	5	.35	23	153	5	61	.27	.97	.82	.71

Note. Percentile – Fall Percentile Cut-Score used with Star Math to predict performance on Pennsylvania System of School Assessments. BR – Base Rate; TP – True Positive; TN – True Negative; FP – False Positive; FN – False Negative; Sn – Sensitivity; Sp – Specificity, PPV – Positive Predictive Value, NPV – Negative Predictive Value.

Diagnostic accuracy results when predicting Proficient levels of performance are also presented in Table 3. When using the 25^th percentile as a cut-score, Sn did not exceed .30 across all grade levels. Specificity values were similar to those observed when predicting Basic performance. Considering the 40^th percentile, Sn values tended to increase but did not exceed those observed when predicting Basic performance across each grade. Specificity values remained largely unchanged.

Discussion

The present study addressed the diagnostic accuracy of SM in predicting elementary students’ proficiency on the PSSA. Research questions considered the correlation between Fall SM performance and Spring PSSA outcomes, along with variations in Sn and Sp with cut-scores at the 25th and 40th percentiles. The strong, positive correlation between Fall SM and Spring PSSA outcomes provides support for the predictive validity of SM. Sensitivity and specificity are also vital to consider when assessing the effectiveness of SM at identifying students in need of additional support. SM was highly accurate at identifying students who were not in need of additional support in the present sample. SM was less accurate at identifying students in need of additional support, particularly when using the 25th percentile threshold in higher grades. Increased Sn was observed when adjusting the cut-score from the 25th to 40th percentile in all grades but remained at suboptimal levels. This highlights the value of increasing cut-scores to more accurately catch those students in need of assistance, but underscores that additional data may be needed prior to making “rule-in” decisions if using SM.

These results have implications for identification and cut-score selection. In most circumstances, Sn and Sp should be balanced (Kettler et al., 2014). This study highlights the need to take into account local context, particularly in very high or low achieving settings, when selecting cut-scores since publisher-provided cut-scores may not be accurate at identifying students at risk of academic failure. In this study, the population was an overall high performing group of students. The publisher provided cut-scores did not accurately identify students who would not achieve proficiency on the PSSAs consistently. In any case, adjusting cut-scores to be more or less stringent will have direct implications on the number of students identified as at-risk and by extension the number of students incorrectly identified. For instance, if one were to prioritize identifying all students that may need supplemental intervention, one could increase the cut-score students need to exceed on SM. This would lead to an increase in Sn by virtue of identifying more students at risk. At the same time, there is an increased chance of FPs, decreasing Sp (or the accuracy of rule-out decisions). Conversely, if a school had limited resources and wanted to only identify those students at most risk for later difficulties, they could decrease the cut-scores students need to score above to be considered not at risk. This in turn will increase Sp (increasing the accuracy of rule-out decisions) at the cost of Sn (by virture of identifying fewer students).

A method to promote more accurate identification, specific to a population markedly different than a norm group, is the use of locally derived cut-scores. Research supports the value of using locally derived cut-scores (Leblanc et al., 2012), especially for the purpose of maximizing the balance between Sn and Sp (Silberglitt & Hintze, 2005). Future research may consider the effectiveness of SM for identifying academically at-risk students when locally derived cut-scores are applied.

One specific finding was relatively perplexing, the poor performance of grade five cut-scores relative to grade three and four. Namely, FN rates in grade five were larger than outcomes from other grades. This suggests that there was an increased likelihood SM would identify students not at-risk and a proportion of students were likely to not go on to achieve Basic or Proficient levels of performance on the end of year test. This may reflect that the content and or questions between SM and the PSSAs are more aligned with one another at lower grade levels. Similarly, a review of the content described on the grade five PSSA suggests that the depth and breadth of skills assessed are greater than those assessed in lower grades. Understanding the limitations of the cross-sectional nature of the present dataset, the base rate of students in grade five for Proficient levels of performance (.35) relative to grade three (.27) and grade four (.24) lends some support to this hypothesis.

The present study had some potential limitations. First, the sample lacked diversity, in that the majority of students across grades were White. This was also an overall high performing sample of students. Future research should investigate the effectiveness of SM when applied with diverse populations, and a student body with a greater level of academic need. In addition, demographic information was not available for student gender. Only two publisher provided cut scores were evaluated for Sn and Sp. It would be worthwhile to investigate the application of locally derived cut-scores for this and other samples markedly different from the SM norm group. However, the original purpose of this study was to evaluate the technical adequacy of cut-scores provided by SM to schools. It is also worth noting that high stakes exams like the PSSAs may not be equally valued as a criterion measure across different regions and communities (January & Ardoin, 2015).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Megan I. Turner

Ethan R. Van Norman

References

Fuchs

L. S.

Compton

D. L.

Fuchs

Hollenbeck

K. N.

Hamlett

C. L.

Seethaler

P. M.

(2011). Two-stage screening for math problem-solving difficulty using dynamic assessment of algebraic learning. Journal of Learning Disabilities, 44(4), 372–380, https://doi.org/10.1177/0022219411407867

January

S. A. A.

Ardoin

S. P.

(2015). Technical adequacy and acceptability of curriculum-based measurement and the measures of academic progress. Assessment for Effective Intervention, 41(1), 3–15. https://doi.org/10.1177/1534508415579095

Kettler

R. J.

Glover

T. A.

Albers

C. A.

Feeney-Kettler

K. A.

(2014). An introduction to universal screening in educational settings. In Kettler

R. J.

Glover

T. A.

Albers

C. A.

Feeney-Kettler

K. A.

(eds), School psychology book series. Universal screening in educational settings: Evidence-based decision making for schools (p. 3–16). American Psychological Association. https://doi.org/10.1037/14316-001

Leblanc

Dufore

McDougal

(2012). Using general outcome measures to predict student performance on state-mandated assessments: An applied approach for establishing predictive cutscores. Journal of Applied School Psychology, 28, 1–13. https://doi.org/10.1080/15377903.2012.643753

Nicholas

S. L.

Berliner

D. C.

(2007). A short history of high stakes testing. In Nichilas

S. L.

Berliner

D. C.

(Ed.), Collateral damage: How high-stakes testing corrupts America’s schools (pp. 1–8). Cambridge, MA: Harvard Education Press.

Pennsylvania Department of Education . (2020). Pennsylvania System of school assessment (PSSA). https://www.education.pa.gov/K-12/Assessment%20and%20Accountability/PSSA/Pages/default.aspx

Renaissance Learning . (2015). STAR assessment information and benchmarks. https://www.alexschools.org/cms/lib/MN01000334/Centricity/Domain/26/STAR%20Assessment%20Information%20and%20Benchmarks.pdf

Renaissance Learning . (2017). Related STAR reading and SM to the state of Texas assessments of academic readiness (STAAR). https://doc.renlearn.com/kmnet/r004573515gk64de.pdf

Renaissance Learning . (2020). The research foundation for star assessments: The science of star [White paper]. Renaissance Learning. http://doc.renlearn.com/KMNet/R001480701GCFBB9.pdf

10.

Renaissance Learning (2021). Star Assessments™ for Math: Technical Manual . Renaissance Learning. https://help2.renaissance.com/US/PDF/SM/SMRPTechnicalManual.pdf

11.

Renaissance Learning . (n.d.). State approvals. https://www.renaissance.com/state-approvals/

12.

Silberglitt

Hintze

(2005). Formative assessment using CBM-R cut scores to track progress toward success on state-mandated achievement tests: A comparison of methods. Journal of Psychoeducational Assessment, 23(4), 304–325, https://doi.org/10.1177/073428290502300402