Testing the Efficacy of a Tier 2 Mathematics Intervention

Abstract

The purpose of this closely aligned conceptual replication study was to investigate the efficacy of a Tier 2 kindergarten mathematics intervention. The replication study differed from the initial randomized controlled trial on three important elements: geographical region, timing of the intervention, and instructional context of the counterfactual. Similar to the original investigation, however, the current study tested the same intervention, used the same outcome measures and statistical analyses, and involved the same population of learners. A total of 319 kindergarten students with mathematics difficulties from 36 kindergarten classrooms participated in the study. Students who were randomly assigned to the treatment condition received the intervention in small-group formats, with 2 or 5 students per group. Control students participated in a no-treatment control condition. Significant effects on proximal and distal measures of mathematics achievement were found. Effect sizes obtained for all measures fell within or exceeded the upper bound of the effects reported in the initial study. Implications for systematically situating replication studies in larger frameworks of intervention research and reporting rates of treatment response across replication studies are discussed.

Get full access to this article

View all access options for this article.

References

Allison

P. D.

(1990). Change scores as dependent variables in regression analysis. Sociological Methodology, 20, 93–114. doi:10.2307/271083

Archer

A. L.

Hughes

C. A.

(2010). Explicit instruction: Effective and efficient teaching. New York, NY: Guilford.

Baldwin

S. A.

Bauer

D. J.

Stice

Rohde

(2011). Evaluating models for partially clustered designs. Psychological Methods, 16, 149–165. doi:10.1037/a0023464

Bauer

D. J.

Sterba

S. K.

Hallfors

D. D.

(2008). Evaluating group-based interventions when control participants are ungrouped. Multivariate Behavioral Research, 43, 210–236. doi:10.1080/00273170802034810

Bryant

D. P.

Bryant

B. R.

Roberts

Vaughn

Pfannenstiel

K. H.

Porterfield

Gersten

(2011). Early numeracy intervention program for first-grade students with mathematics difficulties. Exceptional Children, 78, 7–23.

Clarke

Doabler

C. T.

Fien

Baker

S. K.

Smolkowski

(2012). A randomized control trial of a Tier 2 kindergarten mathematics intervention (Project ROOTS). U.S. Department of Education, Institute of Education Sciences, Special Education Research, CFDA No. 84.324A, 2012-2016, Funding No. R324A120304.

Clarke

Doabler

C.T.

Smolkowski

Kurtz Nelson

Fien

Baker

S.K.

Kosty

(2016). Testing the immediate and long-term efficacy of a Tier 2 kindergarten mathematics intervention. Journal of Research on Educational Effectiveness. Advance online publication. doi:10.1080/19345747.2015.1116034

Clarke

Rolfhus

Dimino

Gersten

R. M.

(2012). Assessing Student Proficiency of Number Sense (ASPENS). Longmont, CO: Cambium Learning Group, Sopris Learning.

Clarke

Shinn

M. R.

(2004). A preliminary investigation into the identification and development of early mathematics curriculum-based measurement. School Psychology Review, 33, 234–248.

10.

Clarke

Smolkowski

Baker

S. K.

Fien

Doabler

C. T.

Chard

D. J.

(2011). The impact of a comprehensive tier 1 core kindergarten program on the achievement of students at-risk in mathematics. Elementary School Journal, 111, 1–24. doi:10.1086/659033

11.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

12.

Cook

B. G.

(2014). A call for examining replication and bias in special education research. Remedial and Special Education, 35, 233–246. doi:10.1177/0741932514528995

13.

Cook

B. G.

Collins

L. W.

Cook

S. C.

Cook

(2016). A replication by any other name . . . : A systematic review of replicative studies. Remedial and Special Education, 37, 223–234. doi:10.1177/0741932516637198

14.

Coyne

M. D.

Cook

B.G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37, 244–253. doi:10.1177/0741932516648463

15.

Coyne

M. D.

Little

Rawlinson

D. A.

Simmons

Kwok

O. M.

Kim

. . . Civetelli

(2013). Replicating the impact of a supplemental beginning reading intervention: The role of instructional context. Journal of Research on Educational Effectiveness, 6, 1–23. doi:10.1080/19345747.2012.706694

16.

Dasgupta

Lawson

K. A.

Wilson

J. P.

(2010). Evaluating equivalence and noninferiority trials. American Journal of Health-System Pharmacy, 67, 1337–1343. doi:10.2146/ajhp090507

17.

Doabler

C. T.

Clarke

Fien

(2012). ROOTS Assessment of Early Numeracy Skills. Unpublished measure, University of Oregon, Eugene.

18.

Doabler

C. T.

Fien

(2013). Explicit mathematics instruction: What teachers can do for teaching students with mathematics difficulties. Intervention in School and Clinic, 48, 276–285. doi:10.1177/1053451212473151

19.

Flay

B. R.

Biglan

Boruch

R. F.

Castro

F. G.

Gottfredson

Kellam

. . . Ji

(2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175. doi:10.1007/s11121-005-5553-y

20.

Fuchs

L.S.

Fuchs

Compton

(2012). The early prevention of mathematics difficulty: Its power and limitations. Journal of Learning Disabilities, 45, 257–269. doi:10.1177/0022219412442167

21.

Gersten

R. M.

Beckmann

Clarke

Foegen

March

Star

J. R.

Witzel

(2009). Assisting students struggling with mathematics: Response to intervention (RTI) for elementary and middle schools (Practice Guide Report No. NCEE 2009-4060). Washington, DC: U.S Department of Education, National Center for Education Evaluation and Regional Assistance.

22.

Ginsburg

Baroody

(2003). Test of Early Mathematics Ability–Third Edition. Austin, TX: Pro-Ed.

23.

Gottfredson

D. C.

Cook

T. D.

Gardner

F. E.

Gorman-Smith

Howe

G. W.

Sandler

I. N.

Zafft

K. M.

(2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926. doi:10.1007/s11121-015-0555-x

24.

Graham

J. W.

(2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. doi:10.1146/annurev.psych.58.110405.085530

25.

Hannan

P. J.

Murray

D. M.

(1996). Gauss or Bernoulli? A Monte Carlo comparison of the performance of the linear mixed model and the logistic mixed model analyses in simulated community trials with a dichotomous outcome variable at the individual level. Evaluation Review, 20, 338–352. doi:10.1177/0193841X9602000306

26.

Harcourt Brace Educational Measurement. (2002). Stanford Achievement Test (SAT-10). San Antonio, TX: Harcourt Brace Jovanovich.

27.

Harcourt Brace Educational Measurement (2003). Stanford Early School Achievement Test (10th ed.). San Antonio, TX: Harcourt Brace Jovanovich.

28.

Hedges

L. V.

(1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128. doi:10.3102/10769986006002107

29.

Hox

J. J.

(2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.

30.

Jamieson

(1999). Dealing with baseline differences: Two principles and two dilemmas. International Journal of Psychophysiology, 31, 155–161. doi:10.1016/S0167-8760(98)00048-8

31.

Jordan

Glutting

Ramineni

(2008). A number sense assessment tool for identifying children at risk for mathematical difficulties. In Dowker

(Ed.), Mathematical difficulties: Psychology and intervention (pp. 45–57). San Diego, CA: Academic Press.

32.

Kromrey

J. D.

Dickenson

W. B.

(1996). Detecting unit of analysis problems in nested designs: Statistical power and Type I error rates of the F test for groups-within-treatment effects. Educational and Psychological Measurement, 56, 215–231. doi:10.1177/0013164496056002003

33.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. doi:10.2307/2529310

34.

Lemons

C. J.

Fuchs

Gilbert

J. K.

Fuchs

L. S.

(2014). Evidence-based practices in a changing world: Reconsidering the counterfactual in education research. Educational Researcher, 43, 242–252. doi:10.3102/0013189X14539189

35.

Lemons

C. J.

King

S. A.

Davidson

K. A.

Berryessa

T. L.

Gajjar

S. A.

Sacks

L. H.

(2016). An inadvertent concurrent replication: Same roadmap, different journey. Remedial and Special Education, 37, 213–222. doi:10.1177/0741932516631116

36.

Little

R. J. A.

Rubin

D. B.

(2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.

37.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty replication in the education sciences. Educational Researcher, 43, 304–316. doi:10.3102/0013189X14545513

38.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research how often do they really occur?. Perspectives on Psychological Science, 7, 537–542. doi:10.1177/1745691612460688

39.

Morgan

P. L.

Farkas

Maczuga

(2015). Which instructional practices most help first-grade students with and without mathematics difficulties? Educational Evaluation and Policy Analysis, 37, 184–205. doi:10.3102/0162373714536608

40.

Murray

D. M.

(1998). Design and analysis of group-randomized trials. New York, NY: Oxford University Press.

41.

National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common Core State Standards for mathematics. Washington, DC: Author. Retrieved from http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf

42.

Okamoto

Case

(1996). Exploring the microstructure of children’s central conceptual structures in the domain of number. In Case

Okamoto

Griffin

Siegler

R.S.

Keating

D. P.

(Eds.), The role of central conceptual structures in the development of children’s thought: Monographs of the Society for Research in Child Development (pp. 27–58). Chicago, IL: Society for Research in Child Development.

43.

Piaggio

Elbourne

D. R.

Altman

D. G.

Pocock

S. J.

Evans

S. W.

for the CONSORT Group. (2006). Reporting of noninferiority and equivalence randomized trials: An extension of the CONSORT statement. Journal of the American Medical Association, 295, 1152–1160. doi:10.1001/jama.295.10.1152

44.

Raudenbush

S. W.

Sadoff

(2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154. 10.1080/19345740801982104

45.

Roberts

S. A.

(2005). Design and analysis of clinical trials with clustering effects due to treatment. Clinical Trials, 2, 152–162. doi:10.1191/1740774505cn076oa

46.

Rosenthal

Rosnow

R. L.

(2008). Essentials of behavioral research: Methods and data analysis (3rd ed.). San Francisco, CA: McGraw-Hill.

47.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701. doi:10.1037/h0037350

48.

Rubin

D. B.

(2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100, 322–331. doi:10.1198/016214504000001880

49.

SAS Institute. (2009). SAS/STAT 9.2 user’s guide (2nd ed.). Cary, NC: Author. Retrieved from the http://support.sas.com/documentation/index.html

50.

Schmidt

(2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. doi:10.1037/a0015108

51.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.

52.

Sood

Jitendra

A. K.

(2013). An exploratory study of a number sense program to develop kindergarten students’ number proficiency. Journal of Learning Disabilities, 46, 328–346. doi:10.1177/0022219411422380

53.

Valentine

J. C.

Biglan

Boruch

R. F.

Castro

F. G.

Collins

L. M.

Flay

B. R.

. . . Schinke

S. P.

(2011). Replication in prevention science. Prevention Science, 12, 103–117. doi:10.1007/s11121-011-0217-6

54.

van Belle

(2008). Statistical rules of thumb (2nd ed.). New York, NY: Wiley.

55.

What Works Clearinghouse. (2014). Procedures and standards handbook (Version 3.0). Washington, DC: U.S. Department of Education, Institute of Education Sciences. Retrieved from http://ies.ed.gov/ncee/wwc/

56.

What Works Clearinghouse. (n.d.). What works in math. Retrieved from http://www.ies.ed.gov

57.

Woodcock

Johnson

M.B.

(1989). Woodcock-Johnson Psycho-Educational Battery—Revised. Chicago, IL: Riverside.