Generalizability and Dependability of the Sentence Order Fluency Measure of Reading Comprehension

The purpose of this study was to examine the generalizability and dependability of scores produced by Sentence Order Fluency, a novel approach to progress monitoring of reading comprehension. Analyses were conducted to evaluate the performance of three alternative scoring methods—Absolute Correct, Pairs Correct, and Levenshtein Similarity—as well as test length and levels of aggregation (the numbers of passages or paragraphs used to calculate scores). Absolute and Pairs Correct scores performed similarly and appeared to show greater generalizability than Levenshtein Similarity scores. Students contributed more variance than probes in most models. Minimally sufficient levels of reliability for progress monitoring decisions could be possible using scores based on administration of 2 passages or 6 paragraphs using Pairs Correct scores. Levenshtein Similarity appeared to require a greater number of probes to obtain comparable reliability, suggesting limited practical value relative to other scoring procedures.