Perceptions of Single-Case Experimental Design Researchers About Questionable and Improved Research Practices

Abstract

Questionable research practices (QRPs) are methodological decisions motivated and reinforced by publication that can distort conclusions drawn from the scientific literature. This study surveyed 143 single-case experimental design (SCED) researchers from various backgrounds about their perceptions of QRPs and corresponding improved research practices (IRPs). Participants rated the problematic nature, prevalence, validity-enhancing potential, and feasibility of 64 QRP-IRP pairs. We found broad agreement that most QRPs undermine SCED research integrity, and IRPs could further enhance SCED validity. Exploratory analyses revealed a strong correlation between perceived QRP severity and likelihood of IRP adoption. However, variability existed in observed QRP prevalence and feasibility of IRP implementation. We discuss ways to increase awareness about QRPs and IRPs, potential journal editorial policy shifts, and SCED researcher development to strengthen transparency and rigor. Implications for advancing SCED methodology also are discussed with emphasis on fostering a culture of improved SCED transparency and rigor.

Keywords

single-case experimental design metascience questionable research practices improved research practices research transparency methodological integrity

Get full access to this article

View all access options for this article.

References

Agnoli

Wicherts

J. M.

Veldkamp

C. L. S.

Albiero

Cubelli

(2017). Questionable research practices among Italian research psychologists. PLOS ONE, 12(3), e0172792. https://doi.org/10.1371/journal.pone.0172792

Anvari

Lakens

(2018). The replicability crisis and public trust in psychological science. Comprehensive Results in Social Psychology, 3(3), 266-286. https://doi.org/10.1080/23743603.2019.1684822

Baer

D. M.

Wolf

M. M.

Risley

T. R

. (1968). Some current dimensions of applied behavior analysis. Journal of applied behavior analysis, 1(1), 91.

Baker

(2016). Reproducibility crisis. Nature, 533(26), 353–366.

Bakker

B. N.

Jaidka

Dörr

Fasching

Lelkes

(2021). Questionable and open research practices: Attitudes and perceptions among quantitative communication researchers. Journal of Communication, 71(5), 715–738. https://doi.org/10.1093/joc/jqab031

Dellsén

(2024). Disagreement and consensus in science. In Baghramian

Carter

J. A.

Cosker-Rowland

(Eds.), The Routledge handbook of philosophy of disagreement (pp. 286–296). Routledge.

Gopalakrishna

ter Riet

Vink

Stoop

Wicherts

J. M.

Bouter

L. M.

(2022). Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PLOS ONE, 17(2), e0263023. https://doi.org/10.1371/journal.pone.0263023

Horner

R. D.

Baer

D. M.

(1978). Multiple-probe technique: A variation of the multiple baseline. Journal of Applied Behavior Analysis, 11(1), 189–196. https://doi.org/10.1901/jaba.1978.11-189

Huntington

R. N.

Badgett

N. M.

Rosenberg

N. E.

Greeny

Bravo

Bristol

R. M.

Park

M. S.

(2023). Social validity in behavioral research: A selective review. Perspectives on Behavior Science, 46(1), 201–215.

10.

Iannone

Cheng

Schloerke

Hughes

Lauer

Seo

Brevoort

Roy

(2024). gt: Easily create presentation-ready display tables (R Package Version 0.11.1.9000). https://gt.rstudio.com

11.

Ioannidis

J. P.

(2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056

12.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

13.

Johnson

A. H.

Cook

B. G.

(2019). Preregistration in single-case design research. Exceptional Children, 86(1), 95–112. https://doi.org/10.1177/0014402919868529

14.

Kazdin

A. E.

(1982). Single-case research designs. Oxford University Press.

15.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf

16.

Ledford

J. R.

Gast

D. L.

(2024). Single case research methodology: Applications in special education and behavioral sciences (4th ed). Routledge.

17.

Makel

M. C.

Hodges

Cook

B. G.

Plucker

J. A.

(2021). Both questionable and open research practices are prevalent in education research. Educational Researcher, 50(8), 493–504. https://doi.org/10.3102/0013189X211001356

18.

McVay

M. A.

Cooper

K. B.

Carrera Seoane

Donahue

M. L.

Scherer

L. D.

(2021). Transparent reporting of hypotheses and analyses in behavioral medicine research: An audit of publications in 2018 and 2008. Health Psychology and Behavioral Medicine, 9(1), 285–297. https://doi.org/10.1080/21642850.2021.1907186

19.

Morris

E. K.

Smith

N. G.

Altus

D. E

. (2005). BF Skinner’s contributions to applied behavior analysis. The Behavior Analyst, 28(2), 99–131.

20.

Neoh

M. J. Y.

Carollo

Lee

Esposito

(2023). Fifty years of research on questionable research practices in science: Quantitative analysis of co-citation patterns. Royal Society Open Science, 10(10), 230677. https://doi.org/10.1098/rsos.230677

21.

Nosek

B. A.

Hardwicke

T. E.

Moshontz

Allard

Corker

K. S.

Dreber

Vazire

(2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748. https://doi.org/10.1146/annurev-psych-020821-114157

22.

O’Donohue

W. O.

Masuda

(2022). Questionable research practices in clinical psychology. In O’Donohue

Masuda

Lilienfeld

(Eds.), Avoiding questionable research practices in applied psychology (pp. 3–18). Springer.

23.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

24.

Pashler

Wagenmakers

E. J.

(2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530.

25.

Plotly Technologies Inc. (2015). Collaborative data science. Plotly Technologies Inc. https://plot.ly

26.

R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

27.

Rosenthal

(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

28.

Schwartz

I. S.

Baer

D. M.

(1991). Social validity assessments: Is current practice state of the art? Journal of Applied Behavior Analysis, 24(2), 189–204. https://doi.org/10.1901/jaba.1991.24-189

29.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(Eds.). (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.

30.

Sidman

(1960). Tactics of scientific research. Basic Books.

31.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/095679761141

32.

Simmons

J. P.

Simonsohn

(2017). Power posing: P-curving the evidence. Psychological Science, 28(5), 687–693. https://doi.org/10.1177/0956797616658563

33.

Slocum

T. A.

Pinkelman

S. E.

Joslyn

P. R.

Nichols

(2022). Threats to internal validity in multiple-baseline design variations. Perspectives on Behavior Science, 45(3), 619–638. https://doi.org/10.1007/s40614-022-00326-1

34.

Slocum

T. A.

Tincani

Travers

J. C.

Dowdy

(2023, March). Questionable single-case research practices: Preliminary results and discussion [Paper presentation]. Badar-Kauffman Conference on Contemporary Issues in Special Education, Kent, OH.

35.

Soderberg

C. K.

Errington

T. M.

Schiavone

S. R.

Bottesini

Thorn

F. S.

Vazire

Esterling

K. M.

Nosek

B. A

. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990–997.

36.

Tawny

J. W.

Gast

D. L.

(1984). Single subject research in special education. Merrill.

37.

Tincani

Travers

J. C.

(2019). Replication research, publication bias, and applied behavior analysis. Perspectives on Behavior Science, 42(1), 59–75. https://doi.org/10.1007/s40614-019-00191-5

38.

Tincani

Travers

J. C.

(2022). Questionable research practices in single-case experimental designs: Examples and possible solutions. In O’Donohue

Masuda

Lilienfeld

S. O.

(Eds.), Questionable research practices: Designing, conducting, and reporting sound research in clinical psychology. Springer Publications.

39.

Tincani

Travers

J. C.

Dowdy

Slocum

T. A.

Deitrich

(2025). Questionable and improved research practices in single-case experimental design: Initial investigation and findings. Perspectives on Behavior Science, 48(2), 447–473. https://doi.org/10.1007/s40614-025-00441-9

40.

van den Akker

O. R.

van Assen

M. A.

Bakker

Elsherif

Wong

T. K.

Wicherts

J. M

. (2024). Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology. Behavior Research Methods, 56(6), 5424–5433.

41.

What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook (Version 5.0). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Handbooks

42.

Wickham

(2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag. https://ggplot2.tidyverse.org

43.

Wickham

François

Henry

Müller

Vaughan

(2023). dplyr: A grammar of data manipulation (R Package Version 1.1.4). https://dplyr.tidyverse.org

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB