Technical Features of Sentence-Level Curriculum-Based Measures and Language Sample Analysis for Students With Writing Difficulties

Abstract

Given oral language’s role in writing proficiency and to address measurement issues in oral and written language, we trialed complementary scoring metrics in language sample analysis (LSA) with the sentence-level Picture Word Writing Curriculum-Based Measure (CBM-W). Using the Picture Word CBM-W samples of 123 students with writing difficulties, we investigated (1a) alternate form reliability, (1b) criterion-related validity with existing Picture Word CBM-W metrics, (2) criterion-related validity with a standardized written expression measure, and (3) sensitivity to growth from fall to spring for LSA and Picture Word CBM-W scoring mechanisms. Pearson product-moment correlations, Spearman’s correlations, and Bonferroni-corrected paired-samples t-tests revealed two promising LSA metrics with evidence of technical quality and sensitivity to growth as a complementary scoring mechanism for Picture Word CBM-W: mean length of T-Unit in morphemes (MLTU-M) using the mean of two forms in the fall, and number of different words (NDW) using the mean of two forms in fall and spring. Results support the role of oral language in sentence-level writing proficiency, as well as MLTU-M and NDW as complementary scoring mechanisms to provide more specific estimates of oral language skills in grammar/morphosyntax and semantics/lexical diversity not possible with current Picture Word CBM-W scoring mechanisms.

Keywords

writing curriculum-based measurement language

Get full access to this article

View all access options for this article.

References

Adlof

Hogan

(2019). If we don’t look, we won’t see: Measuring language development to inform literacy instruction. Policy Insights from the Behavioral and Brain Sciences, 6(2), 210–217. https://doi.org/10.1177/2372732219839075

Allen

A. A.

Jung

P.-G.

Poch

A. L.

Brandes

Shin

Lembke

E. S.

McMaster

K. M.

(2020). Technical adequacy of curriculum-based measures in writing in grades 1-3. Reading & Writing Quarterly: Overcoming Learning Difficulties, 36(6), 563–587. https://doi.org/10.1080/10573569.2019.1689211

Allen

A. A.

Poch

A. L.

Lembke

E. S.

(2018). An exploration of alternative scoring methods using curriculum-based measurement in early writing. Learning Disability Quarterly, 41(2), 85–99. https://www-jstor-org-443.web.bisu.edu.cn/stable/26742893

Bao

Komesidou

Hogan

T. P.

(2024). A review of screeners to identify risk of developmental language disorder. American Journal of Speech-Language Pathology, 33(3), 1548–1571. https://doi.org/10.1044/2023_AJSLP-23-00286

Beers

Nagy

(2009). Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? Reading & Writing, 22(2), 185–200. https://doi.org/10.1007/s11145-007-910

Berninger

V. W.

Amtmann

(2003). Preventing written expression disabilities through early and continuing assessment and intervention for handwriting and/or spelling problems Research into practice. In Swanson

H. L.

Harris

K. R.

Graham

(Eds.), Handbook of learning disabilities (pp. 345–363). Guilford Press.

Denman

Speyer

Munro

Pearce

W. M.

Chen

Y.-W.

Cordier

(2017). Psychometric properties of language assessments for children aged 4–12 years: A systematic review. Frontiers in Psychology, 8, 1–28. https://doi.org/10.3389/fpsyg.2017.01515

Deno

S. L.

Marston

Mirkin

(1982). Valid measurement procedures for continuous evaluation of written expression. Exceptional Children, 48(4), 368–371. https://doi.org/10.1177/001440298204800417

Deno

S. L.

Mirkin

Marston

(1980). Relationships among simple measures of written expression and performance on standardized achievement tests (Vol. IRLD-RR-22). Institute for Research on Learning Disabilities, University of Minnesota.

10.

Dockrell

Connelly

(2009). The impact of oral language skills on the production of written text. Teaching and Learning Writing, 11(6), 45–62. https://doi.org/10.1348/000709909X421919

11.

Dockrell

Connelly

Walter

Critten

(2015). Assessing children’s writing products: The role of curriculum-based measures. British Educational Research Journal, 41(4), 575–595. https://doi.org/10.1002/berj.3162

12.

Dockrell

Lindsay

Connelly

(2009). The impact of specific language impairment on adolescents’ written text. Exceptional Children, 75(4), 427–446. https://doi.org/10.1177/001440290907500403

13.

Dockrell

Lindsay

Connelly

Mackie

(2007). Constraints in the production of written text in children with specific language impairments. Exceptional Children, 73(2), 147–164. https://doi.org/10.1177/001440290707300202

14.

Dockrell

Ricketts

Charman

Lindsay

(2014). Exploring writing products in students with language impairments and autism spectrum disorders. Learning and Instruction, 32, 81–90. https://dx-doi-org-s.web.bisu.edu.cn/10.1016/j.learninstruc.2014.01.008

15.

Ebert

K. D.

(2020). Language sample analysis with bilingual children: Translating research to practice. Topics in Language Disorders, 40(2), 182–201. https://doi.org/10.1097/TLD.0000000000000209

16.

Fumero

Wood

(2021). Verb errors in 5th-grade English learners’ written responses: Relation to writing quality. Languages, 6(2), 71–86. https://doi.org/10.3390/languages6020071

17.

Gansle

K. A.

Noell

G. H.

VanDerHeyden

A. M.

Naquin

G. M.

Slider

N. J.

(2002). Moving beyond total words written: The reliability, criterion validity, and time cost of alternate measures for curriculum-based measurement in writing. School Psychology Review, 31(4), 477–497. https://doi.org/10.1080/02796015.2002.12086169

18.

Graham

Collins

Rigby-Wills

(2016). Writing characteristics of students with learning disabilities and typically achieving peers. Exceptional Children, 83(2), 199–218. https://doi.org/10.1177/0014402916664070

19.

Graham

Hebert

Fishman

Ray

Rouse

(2020). Do children classified with specific language impairment have a learning disability in writing? A meta-analysis. Journal of Learning Disabilities, 53(4), 292–310. https://doi.org/10.1177/0022219420917338

20.

Hall-Mills

Apel

(2015). Linguistic feature development across grades and genre in elementary writing. Language, Speech & Hearing Services in Schools, 46(3), 242–255. https://doi.org/10.1044/2015_LSHSS-14-0043

21.

Hammill

D. D.

Larsen

S. C.

(2009). Test of Written Language (TOWL-4) (4th ed.). Pro-Ed.

22.

Hughes

McGillivary

Schmidek

(1997). Guide to narrative language: Procedures for assessment. Pro-Ed.

23.

Hunt

K. W.

(1970). Syntactic maturity in schoolchildren and adults. Monographs of the Society for Research in Child Development, 35(1), 1–67. https://doi.org/10.2307/1165818

24.

Jagaiah

Olinghouse

Kearns

(2020). Syntactic complexity measures: Variation by genre, grade-level, students’ writing abilities, and writing quality. Reading & Writing, 33(10), 2577–2638. https://doi.org/10.1007/s11145-020-10057-x

25.

Justice

Bowles

Kaderavek

Ukrainetz

Eisenberg

Gillam

(2006). The index of narrative microstructure: A clinical tool for analyzing school-age children’s narrative performances. American Journal of Speech-language Pathology, 15(2), 177–191. https://doi.org/10.1044/1058-0360(2006/017)

26.

Kane

M., T.

(1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. https://doi.org/10.1037/0033-2909.112.3.527

27.

Kaufman

A. S.

Kaufman

N. L.

(2014). Kaufman Test of Educational Achievement, Third Edition (KTEA-3) [Technical manual]. Pearson.

28.

Kim

Al Otaiba

Folsom

Greulich

Puranik

(2014). Evaluating the dimensionality of first-grade written composition. Journal of Speech, Language, and Hearing Research, 57(1), 199. https://doi.org/10.1044/1092-4388(2013/12-0152)

29.

Kim

Al Otaiba

Wanzek

(2015). Kindergarten predictors of third grade writing. Learning and Individual Differences, 37, 27–37. https://dx-doi-org-s.web.bisu.edu.cn/10.1016/j.lindif.2014.11.009

30.

Kim

Graham

(2022). Expanding the direct and indirect effects model of writing (DIEW): Reading–writing relations, and dynamic relations as a function of measurement/dimensions of written composition. Journal of Educational Psychology, 114(2), 215–238. https://doi.org/10.1037/edu0000564

31.

Kim

Park

(2019). Unpacking pathways using the direct and indirect effects model of writing (DIEW) and the contributions of higher order cognitive skills to writing. Reading & Writing, 32(5), 1319. https://doi.org/10.1007/s11145-018-9913-y

32.

Kim

Puranik

Otaiba

(2015). Developmental trajectories of writing skills in first grade. The Elementary School Journal, 115(4), 593–613. https://doi.org/10.1086/681971

33.

Kim

Schatschneider

(2017). Expanding the developmental models of writing: A direct and indirect effects model of developmental writing (DIEW). Journal of Educational Psychology, 109(1), 35–50. https://dx-doi-org-s.web.bisu.edu.cn/10.1037/edu0000129

34.

Koutsoftas

Gray

(2012). Comparison of narrative and expository writing in students with and without language- learning disabilities. Language, Speech & Hearing Services in Schools, 43(4), 395–409. https://doi.org/10.1044/0161-1461(2012/11-0018)

35.

Lee

De La Paz

(2021). Writing scientific explanations: Effects of a cognitive apprenticeship for students with LD and English learners. Exceptional Children, 87(4), 458–475. https://doi.org/10.1177/0014402921999310

36.

Lenhard

(2016). Calculation of effect sizes. Bibergau (Germany): Psychometrica. https://www.psychometrica.de/effect_size.html(open in a new window)

37.

McMaster

K. L.

Lembke

E. S.

(2018a). Diagnostic Checklist. Early Writing Project. https://earlywritingproject.org/tools/writing-intervention-toolkit/

38.

McMaster

K. L.

Lembke

E. S.

(2018b). Picture-word prompts. Early Writing Project. https://earlywritingproject.org/tools/curriculum-based-measurement-toolkit/

39.

McMaster

K. L.

Lembke

E. S.

Shanahan

Choi

Schatschneider

Deusenberg-Marshall

Birinci

McCollom

Garman

Moore

(2024). Supporting teachers’ data-based individualization of early writing instruction: An efficacy trial. Journal of Learning Disabilities, 58(4), 287–303. https://doi.org/10.1177/00222194241300324

40.

McMaster

K. L.

Lembke

E. S.

Shin

Poch

Smith

R. A.

Jung

Allen

Wagner

(2020). Supporting teachers’ use of data-based instruction to improve students’ early writing skills. Journal of Educational Psychology, 112, 1–21. https://doi.org/10.1037/edu0000358

41.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 13–103). New York American Councilon Education Macmillan.

42.

Miller

Iglesias

(2023). Systematic Analysis of Language Transcripts (SALT) (Version 21) [Computer Software]. SALT Software, LLC.

43.

National Assessment of Educational Progress [NAEP]. (2022). NAEP report card: 2022 NAEP reading assessment. https://www.nationsreportcard.gov/highlights/reading/2022/

44.

National Center for Education Statistics [NCES]. (2022). Students with disabilities. U.S. Department of Education, Institute of Education Sciences. https://nces.ed.gov/programs/coe/indicator/cgg

45.

Pavelko

Owens

Ireland

Hahs-Vaughn

(2016). Use of language sample analysis by school-based SLPs: Results of a nationwide survey. Language, Speech & Hearing Services in Schools, 47(3), 246–258. https://doi.org/10.1044/2016_LSHSS-15-0044

46.

Petersen

D. B.

Stoddard

(2018). Psychometric requirements of oral and written language progress monitoring assessments. Perspectives of the ASHA Special Interest Groups, 3(Part 4), 180–197. https://doi.org/10.1044/persp3.SIG1.180

47.

Puranik

Duncan

Ying

(2020). Exploring the dimensionality of kindergarten written composition. Reading & Writing, 33(10), 2481–2510. https://doi.org/10.1007/s11145-020-10053-1

48.

Ratner

N. B.

Han

Yang

J. S.

(2024). Should we stop using lexical diversity measures in children’s language sample analysis? American Journal of Speech-Language Pathology, 33(4), 1986–2001. https://doi.org/10.1044/2024_AJSLP-23-00457

49.

Reno

E. A.

McMaster

K. L.

(2024). Measuring linguistic growth in sentence-level writing curriculum-based measures: Exploring complementary scoring methods. Language, Speech, and Hearing Services in Schools, 55(2), 529–544. https://doi.org/10.1044/2023_LSHSS-23-00056

50.

Sarmiento

C. M.

Truckenmiller

A. J.

Cho

Hwang

(2024). Academic language use in middle school informational writing. British Journal of Educational Psychology, 95(2), 384–404. https://doi.org/10.1111/bjep.12724

51.

Schrank

F. A.

Mather

McGrew

K. S.

(2014a). Woodcock-Johnson IV Tests of Achievement. Riverside.

52.

Scott

(2020). Language sample analysis of writing in children and adolescents: Assessment and intervention contributions. Topics in Language Disorders, 40(2), 202–220. https://doi.org/10.1097/TLD.0000000000000213

53.

Silliman

E. R.

Berninger

V. W.

(2011). Cross-disciplinary dialogue about the nature of oral and written language problems in the context of developmental, academic, and phenotypic profiles. Topics in Language Disorders, 31(1), 6–23. https://doi.org/10.1097/TLD.0b013e31820a0b5b

54.

Troia

Shen

Brandon

(2019). Multidimensional levels of language writing measures in grades four to six. Written Communication, 36(2), 231–266. https://doi.org/10.1177/0741088318819473

55.

Wechsler

(2001). Wechsler Individual Achievement Test–Second Edition (WIAT) [Database record]. APA PsycTests.

56.

Wood

Schatschneider

(2021). Examining writing measures and achievement for students of varied language abilities and linguistic backgrounds using structural equation modeling. Reading Writing Quarterly, 37(1), 65–81. https://doi.org/10.1080/10573569.2020.1716284

57.

Wood

C. L.

Schatschneider

Hart

(2020). Average one year change in lexical measures of written narratives for school age students. Reading & Writing Quarterly, 36(3), 260–277. https://doi.org/10.1080/10573569.2019.1635544

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.16 MB