The Consequences of Predicting Scientific Impact in Psychology Using Journal Impact Factors

Abstract

An academic journal’s impact factor (hereafter JIF) is an average measure of the citation count of individual articles published in that journal. JIF is used to assess merit, predict impact, and allocate resources, but the actual number of citations to individual articles is only modestly correlated with the JIFs of the journals in which they are published. We counted PsycInfo citations to 1,133 articles published in nine leading psychology journals (1996–2005). Both article length (r =.31) and reference list length (r = .41) predicted log-transformed citation counts better than JIF (r = .27). Articles with fewer graphs and more structural equation models were more frequently cited. Citation count was better predicted by a model based on article length and citation count rather than JIF. When JIF was used to predict citation count, the impact of women authors and social science research was underestimated. These findings distinguish impact in science, as measured by JIF, from actual impact in psychology, and they show the unintended consequences of using a measure of the former to predict the latter.

Keywords

allied field sociology methodology quantitative scientific sex/gender

Get full access to this article

View all access options for this article.

References

Adair

J. G.

Vohra

(2003). The explosion of knowledge, references, and citations: Psychology’s unique response to a crisis. American Psychologist, 58, 15–23.

Adler

R. J.

Ewing

Taylor

(2008). Citation statistics: A report from the International Mathematical Union in cooperation with the International Council of Industrial and Applied Mathematics and the Institute of Mathematical Statistics. Statistical Science, 24, 1–14.

Baldi

(1998). Normative versus social constructivist processes in the allocation of citations: A network-analytic model. American Sociological Review, 63, 829–846.

Bornman

Daniel

H.-D.

(2008). What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation, 64, 45–80.

Cawkell

A. E.

(1976). Documentation note - citations, obsolescence, enduring articles, and multiple authorships. Journal of Documentation, 32, 53–58.

Cleveland

W. S.

(1984). Graphs in scientific publications. American Statistician, 38, 261–269.

Dahwan

S. M.

Gupta

B. M.

(2004). Comparative evaluation of Indian physics research: Impact factor vs. citations frequency. Current Science, 86, 1194–1195.

Diekman

A. B.

Eagly

A. H.

(2000). Stereotypes as dynamic constructs: Women and men of the past, present, and future. Personality and Social Psychology Bulletin, 26, 1171–1188.

European Association of Science Editors. (2008). EASE statement on inappropriate use of impact factors. Retrieved from http://www.ease.org.uk/artman2/uploads/1/EASE_statement_IFs_final.pdf

10.

Gannon

Luchetta

Rhodes

Pardie

Segrist

(1992). Sex bias in psychological research: Progress or complacency? American Psychologist, 47, 389–396.

11.

Garfield

(1972). Citation analysis as a tool in journal evaluation. Science, 178, 471–479.

12.

Garfield

(2006). The history and meaning of the Journal Impact Factor. Journal of the American Medical Association, 295, 90–93.

13.

Haslam

Ban

Kaufman

Loughnan

Peters

Whelan

Wilson

(2008). What makes an article influential? Predicting impact in social and personality psychology. Scientometrics, 76, 169–185.

14.

Haslam

Koval

(2010). Possible research area bias in the Excellence in Research in Australia (ERA) draft journal rankings. Australian Journal of Psychology, 62, 112–114.

15.

Hegarty

(2011). The cognitive science of visual-spatial displays: Implications for design. Trends in Cognitive Science, 3, 446–474.

16.

Herschberger

S. L.

Marcoulides

G. A.

Parramore

M. M.

(2003). Structural equation modelling: An introduction. In Pugesek

B. T.

Adrian VonEye

(Eds.), Structural equation modelling: Applications in ecological and evolutionary biology (pp. 3–41). Cambridge, England: Cambridge University Press.

17.

Holden

Rosenberg

Barker

Onghena

(2006). An assessment of the predictive validity of impact factor scores: Implications for academic employment decisions in social work. Journal of Social Work Practice, 16, 613–624.

18.

Latour

(1990). Visualisation and cognition: Drawing things together. In Lynch

Woolgar

(Eds.), Representation in scientific practice (pp. 19–68). Cambridge, MA: MIT Press.

19.

Lee

I. C.

Crawford

(2007). Lesbian and bisexual women in the eyes of scientific psychology. Feminism & Psychology, 17, 109–127.

20.

McCabe

D. P.

Castel

A. D.

(2008). Seeing is believing: The effects of brain images on judgments of scientific reasoning. Cognition, 107, 343–352.

21.

Nederhof

A. J.

(2006). Bibliometric monitoring of research performance in the social sciences and the humanities: A review. Scientometrics, 66, 81–100.

22.

Norris

Oppenheim

(2003). Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 59, 709–730.

23.

Osherson

Smith

E. E.

Wilkie

Lopez

Shafir

(1990). Category-based induction. Psychological Review, 97, 185–200.

24.

Over

(1982). Research productivity and impact of male and female psychologists. American Psychologist, 37, 24–31.

25.

Park

(2009). Publishing in the psychological sciences: Enhancing journal impact while decreasing author fatigue. Perspectives on Psychological Science, 4, 36–37.

26.

Petty

R. E.

Fleming

M. A.

Fabrigar

L. R.

(1999). The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25, 188–203.

27.

PLoS Medicine Editors. (2006). The impact factor game. PLoS Medicine, 3, e291.

28.

Quiñones-Vidal

López-García

J. J.

Peñaranda-Ortega

Tortosa-Gil

(2004). The nature of social and personality psychology as reflected in JPSP, 1965-2000. Journal of Personality and Social Psychology, 86, 435–452.

29.

Robins

Gosling

S. D.

Craik

K. H.

(1999). An empirical analysis of trends in psychology. American Psychologist, 54, 117–128.

30.

Rossiter

M. W.

(1993). The Matthew Matilda effect in science. Social Studies of Science, 23, 325–341.

31.

Roth

W. M.

(2003). Toward an anthropology of graphing: Semiotic and activity-theoretic perspectives. Berlin, Germany: Springer.

32.

Safer

M. A.

Tang

(2009). The psychology of referencing in psychology journal articles. Perspectives in Psychological Science, 4, 51–53.

33.

Seglen

P. O.

(1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45, 1–11.

34.

Seglen

P. O.

(1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 497–502.

35.

Shadish

W. R.

Tolliver

Gray

Gupta

S. K. S.

(1995). Author judgements about works they cite: Three studies from psychology journals. Social Studies of Science, 25, 477–498.

36.

Smith

L. D.

Best

L. A.

Stubbs

A. D.

Archibald

A. B.

Roberson-Nay

(2002). Constructing knowledge: The role of graphs and tables in hard and soft psychology. American Psychologist, 57, 749–761.

37.

Tang

Safer

M. A.

(2008). Author-rated importance of cited references in biology and psychology publications. Journal of Documentation, 64, 246–272.

38.

Taylor

(2009). Publishing in scientific journals: We’re not just talking to ourselves anymore. Perspectives in Psychological Science, 4, 38–39.

39.

Weisberg

D. S.

Keil

F. C.

Goodstein

Rawson

Gray

J. R.

(2008). The seductive allure of neuroscience explanations. Journal of Cognitive Neuroscience, 20, 470–477.