Reliability and the Hidden Translational Gaps in Multidisciplinary Clinical Psychological Science

Abstract

Clinical psychological science combines multiple measurement traditions, which can create hidden translational gaps in which mismatches in methods and constructs go unnoticed. Here, I discuss reliability as an example of a hidden gap between disciplines, drawing from my own career adapting experimental cognitive tasks for clinical research and the personal challenges of confronting methods limitations. Now, as the field moves toward within-person (e.g., idiographic) approaches, researchers face potential new gaps as they adapt measures from experimental and individual differences research. Bridging these gaps will require even broader multidisciplinary collaboration and awareness of the boundaries between methods.

Keywords

cognition(s)individual differences neuropsychology research methods and experimental design test validity

Get full access to this article

View all access options for this article.

References

Brigham

C. C.

(1922). A study of American intelligence. Princeton University Press.

Button

K. S.

Ioannidis

J. P.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S.

Munafò

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.

Clark

Birch-Hurst

Pennington

C. R.

Petrie

A. C.

Lee

J. T.

Hedge

(2022). Test-retest reliability for common tasks in vision science. Journal of Vision, 22(8), Article 18. https://doi.org/10.1167/jov.22.8.18

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957

Cuthbert

B. N.

Insel

T. R.

(2013). Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC medicine, 11(1), 126.

Cyders

M. A.

Coskunpinar

(2011). Measurement of constructs using self-report and behavioral lab tasks: Is there overlap in nomothetic span and construct representation for impulsivity? Clinical Psychology Review, 31(6), 965–982.

Enkavi

A. Z.

Eisenberg

I. W.

Bissett

P. G.

Mazza

G. L.

MacKinnon

D. P.

Marsch

L. A.

Poldrack

R. A.

(2019). Large-scale analysis of test–retest reliabilities of self-regulation measures. Proceedings of the National Academy of Sciences, 116(12), 5472–5477.

Fechner

G. T.

(1860). Elements of Psychophysics (Vol. 2). Breitkopf and Härtel.

Galton

(1883). Inquiries into human faculty and its development. Macmillan.

10.

Gelman

Carlin

(2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651.

11.

Germine

Strong

R. W.

Singh

Sliwinski

M. J.

(2021). Toward dynamic phenotypes and the scalable measurement of human behavior. Neuropsychopharmacology, 46(1), 209–216. https://doi.org/10.1038/s41386-020-0757-1

12.

Germine

L. T.

Hooker

C. I.

(2011). Face emotion recognition is related to individual differences in psychosis-proneness. Psychological Medicine, 41(5), 937–947. https://doi.org/10.1017/S0033291710001571

13.

Green

S. B.

Yang

Alt

Brinkley

Gray

Hogan

Cowan

(2016). Use of internal consistency coefficients for estimating reliability of experimental task scores. Psychonomic Bulletin & Review, 23, 750–763.

14.

Haxby

J. V.

Parasuraman

Lalonde

Abboud

(1993). SuperLab: General-purpose Macintosh software for human experimental psychology and psychological testing. Behavior Research Methods, Instruments, & Computers, 25, 400–405.

15.

Hedge

Powell

Bompas

Sumner

(2020). Self-reported impulsivity does not predict response caution. Personality and Individual Differences, 167, Article 110257. https://doi.org/10.1016/j.paid.2020.110257

16.

Hedge

Powell

Sumner

(2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50, 1166–1186.

17.

Hochman

(2025, March 8). The dot-probe task is probably fine: Bayesian modeling of reliability with brms. CogPsychReserve. https://cogpsychreserve.netlify.app/posts/dotprobereliability/

18.

Miller

G. A.

(2003). The cognitive revolution: A historical perspective. Trends in Cognitive Sciences, 7(3), 141–144.

19.

Molenaar

P. C.

(2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement, 2(4), 201–218.

20.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

21.

Parsons

Kruijt

A.-W.

Fox

(2019). Psychological science needs a standard practice of reporting the reliability of cognitive-behavioral measurements. Advances in Methods and Practices in Psychological Science, 2(4), 378–395.

22.

Passell

Dillon

D. G.

Baker

J. T.

Vogel

S. C.

Scheuer

L. S.

Mirin

N. L.

Rutter

L. A.

Pizzagalli

D. A.

Germine

(2019). Digital cognitive assessment: Results from the TestMyBrain NIMH Research Domain Criteria (RDoC) field test battery report [PREPRINT]. PsyArXiv. https://doi.org/10.31234/osf.io/dcszr

23.

Paulus

M. P.

Thompson

W. K.

(2019). The challenges and opportunities of small effects: The new normal in academic psychiatry. JAMA Psychiatry, 76(4), 353–354.

24.

Price

R. B.

Brown

Siegle

G. J.

(2019). Computational modeling applied to the dot-probe task yields improved reliability and mechanistic insights. Biological Psychiatry, 85(7), 606–612.

25.

Schmukle

S. C.

(2005). Unreliability of the dot probe task. European Journal of Personality, 19(7), 595–605.

26.

Schneider

(1988). Micro experimental laboratory: An integrated system for IBM PC compatibles. Behavior Research Methods, Instruments, & Computers, 20(2), 206–217.

27.

Seguin

(1866). Idiocy: And its treatment by the physiological method. W. Wood & Company.

28.

Sliwinski

M. J.

Mogle

J. A.

Hyun

Munoz

Smyth

J. M.

Lipton

R. B.

(2018). Reliability and validity of ambulatory cognitive assessments. Assessment, 25(1), 14–30.

29.

Spearman

(1904). Reprinted: The proof and measurement of association between two things (2010). International Journal of Epidemiology, 39(5), 1137–1150.

30.

Stern

(1900). On the psychology of individual differences: Ideas toward a “differential psychology”. JA Barth.

31.

Strickland

J. C.

Johnson

M. W.

(2021). Rejecting impulsivity as a psychological construct: A theoretical, empirical, and sociocultural argument. Psychological Review, 128(2), 336–361. https://doi.org/10.1037/rev0000263

32.

Thorndike

E. L.

(1904). Theory of mental and social measurements. Teachers College, Columbia University.

33.

Wundt

(1874). Grundzüge der physiologischen Psychologie [Principles of physiological psychology]. W. Engelmann.

34.

Passell

Strong

R. W.

Grinspoon

Jung

Wilmer

J. B.

Germine

L. T.

(2025). No evidence of reliability across 36 variations of the emotional dot-probe task in 9,600 participants. Clinical Psychological Science, 13(2), 261–277.

35.

Yerkes

R. M.

(Ed.). (1921). Psychological examining in the United States army (Vol. 15). U.S. Government Printing Office.

36.

Zorowitz

Niv

(2023). Improving the reliability of cognitive task measures: A narrative review. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 8(8), 789–797.