Outcome-Reporting Bias in Special Education Intervention Research Using Experimental and Quasi-Experimental Group Designs: A Conceptual Replication

Abstract

We conducted a conceptual replication of Pigott et al.’s study of outcome-reporting bias, wherein they compared intervention outcomes reported in unpublished education dissertations with corresponding published versions. For our replication, we identified a sample of 40 special education dissertations with matched journal publications and found that statistically significant intervention outcomes from dissertations were 1.48 times more likely to be published compared with nonsignificant outcomes. Significant moderators of this effect included type of intervention outcome (academic), type of research design (randomized controlled trial), participant race (with samples greater than or equal to 50% non-White), and type of disability/exceptionality (high incidence). We found that few dissertation authors published their work, providing further evidence for the much-needed inclusion of dissertations in systematic reviews.

Keywords

quantitative research methodology meta-analysis special education

Get full access to this article

View all access options for this article.

References

Chan

A. W.

Hróbjartsson

Haahr

M. T.

Gøtzsche

P. C.

Altman

D. G.

(2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 2457–2465. https://doi.org/10.1001/jama.291.20.2457

Chow

J. C.

Eckholm

(2018). Do published studies yield larger effect sizes than unpublished studies in education and special education? A meta-review. Educational Psychology Review, 30, 727–744. https://doi.org/10.1007/s10648-018-9437-7

Clarke

Doabler

C. T.

Smolkowski

Turtura

Kosty

Kurtz-Nelson

Fien

Baker

S. K.

(2019). Exploring the relationship between initial mathematics skill and a kindergarten mathematics intervention. Exceptional Children, 85(2), 129–146. https://doi.org/10.1177/0014402918799503

Cook

B. G.

Therrien

W. J.

(2017). Null effects and publication bias in special education research. Behavioral Disorders, 42(4), 149–158. https://doi.org/10.1177/0198742917709473

Dwan

Altman

D. G.

Arnaiz

J. A.

Bloom

Chan

A.-W.

Cronin

Decullier

Easterbrook

P. J.

Elm

E. V.

Gamble

Ghersi

Ioannidis

J. P. A.

Simes

Williamson

P. R.

(2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLOS ONE, 3(8), Article e3081. https://doi.org/10.1371/journal.pone.0003081

Endrew F. v. Douglas County School District RE-1, 580 U. S. (2017).

Fanelli

(2010). “Positive” results increase down the hierarchy of the sciences. PLOS ONE, 5, Article e10068. https://doi.org/10.1371/journal.pone.0010068

Fanelli

(2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. https://doi.org/10.1007/s11192-011-0494-7

Fuchs

L. S.

(2019). On the importance of moderator analysis in intervention research: An introduction to the special issue. Exceptional Children, 85(2), 126–128. https://doi.org/10.1177/0014402918811924

10.

Gage

N. A.

Cook

B. G.

Reichow

(2017). Publication bias in special education meta-analyses. Exceptional Children, 83(4), 428–445. https://doi.org/10.1177/0014402917691016

11.

Geuke

G. G. M.

Maric

Mioˇcevic

Wolters

L. H.

de Haan

. (2019). Testing mediators of youth intervention outcomes using single-case experimental designs. New Directions for Child and Adolescent Development, 2019(167), 39–64. https://doi.org/10.1002/cad.20310

12.

Kuritz

S. J.

Landis

J. R.

Koch

G. G.

(1988). A general overview of Mantel-Haenszel methods: Applications and recent developments. Annual Review of Public Health, 9, 123–160. https://doi.org/10.1146/annurev.pu.09.050188.001011

13.

Lyon

A. R.

Connors

Jensen-Doss

Landes

S. J.

Lewis

C. C.

McLeod

B. D.

Rutt

Stanick

Weiner

B. J.

(2017). Intentional research design in implementation science: Implications for the use of nomothetic and idiographic assessment. TBM, 7, 567–580. https://doi.org/10.1007/s13142-017-0464-6

14.

Maag

J. W.

Losinski

(2015). Thorny issues and prickly solutions: Publication bias in meta-analytic reviews in the social sciences. Advances in Social Sciences Research Journal, 2(3), 242–253. https://doi.org/10.14738/assrj.23.1044

15.

McLeod

B. D.

Weisz

J. R.

(2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72(2), 235–251. https://doi.org/10.1037/0022-006X.72.2.235

16.

Medical Research Council. (2023). Outcome reporting bias in trials. https://outcome-reporting-bias.org/

17.

Pigott

T. D.

Valentine

J. C.

Polanin

J. R.

Williams

R. T.

Canada

D. D.

(2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. https://doi.org/10.3102/0013189X13507104

18.

Polanin

J. R.

Tanner-Smith

E. E.

Hennessy

E. A.

(2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067

19.

Preacher

K. J.

Sterba

S. K.

(2019). Aptitude-by-treatment interactions in research on educational interventions. Exceptional Children, 85(2), 248–264. https://doi.org/10.1177/0014402918802803

20.

Ranganathan

Pramesh

C. S.

Buyse

(2016). Common pitfalls in statistical analysis: The perils of multiple testing. Perspectives in Clinical Research, 7(2), 106–107. https://doi.org/10.4103/2229-3485.179436

21.

Rothstein

H. R.

Bushman

B. J.

(2012). Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods, 17, 129–136. https://doi.org/10.1037/a0027128

22.

Rothstein

H. R.

Hopewell

(2009). Grey literature. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (pp. 103–125). Russell Sage Foundation.

23.

Rothstein

H. R.

Sutton

A. J.

Borenstein

(2005). Publication bias in meta-analysis. In Rothstein

H. R.

Sutton

A. J.

Borenstein

(Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1–7). John Wiley. https://doi.org/10.1002/0470870168.ch1

24.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage Learning.

25.

Talbott

De Los Reyes

(2022). Making sense of multiple data sources: Using single case research design for behavioral decision making. In Farmer

T. W.

Talbott

McMaster

K. L.

Lee

Aceves

T. C.

(Eds.), Handbook of Special Education Research: Theory, Methods, and Developmental Processes: Volume 1 (pp. 231–244). Routledge.

26.

Talbott

Maggin

D. M.

Van Acker

E. Y.

Kumm

(2018). Quality indicators for reviews of research in special education. Exceptionality, 26(4), 245–265. https://doi.org/10.1080/09362835.2017.1283625

27.

Toste

J. R.

Logan

J. A. R.

Shogren

K. A.

Boyd

B. A.

(2023). The next generation of quality indicators for group design research in special education. Exceptional Children, 89(4), 359–378. https://doi.org/10.1177/00144029221150801

28.

Trainor

B. P.

(2015). Incomplete reporting: Addressing the problem of outcome-reporting bias in educational research [Unpublished doctoral dissertation]. Loyola University Chicago. https://ecommons.luc.edu/luc_diss/1656

29.

Vaughn

Roberts

Capin

Miciak

Cho

Fletcher

J. M.

(2019). How initial word reading and language skills affect reading comprehension outcomes for students with reading difficulties. Exceptional Children, 85(2), 180–196. https://doi.org/10.1177/0014402918782618

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB