The vote by TASA members on the ‘Most Influential Book in Australian Sociology’ (MIBAS) provides an opportunity to compare quantitative indicators based on citations with the peer review represented by the MIBAS votes. While it is generally agreed that citations are only a partial indicator of such a complex phenomenon as quality, citations are regarded as a reflection of a publication’s, author’s or research group’s ‘impact’ or influence. This assumption is explored by applying several interpretations and operationalizations of the concept of ‘influence’, by discussing their validity, and by comparing the respective citation rankings to the MIBAS poll. The major lessons of the comparison are that citation-based indicators should not be applied in diachronic comparisons, for evaluating publications on nationally specific topics, or in fields in which books are an important part of the research output.
Albert, M. (2003) ‘Universities and the Market Economy: The Differential Impact on Knowledge Production in Sociology and Economics’ , Higher Education45: 147–182 .
2.
Anderson, R.C. , F. Narin and P. McAllister (1978) ‘Publication Ratings versus Peer Ratings of Universities’ , Journal of the American Society for Information Science3: 91–103 .
3.
Beyer, J.M. (1978) ‘Editorial Policies and Practices among Leading Journals in Four Scientific Fields’ , Sociological Quarterly19: 68–88 .
4.
Bott, D.M. and L.L. Hargens (1991) ‘Are Sociologists’ Publications Uncited? Citation Rates of Journal Articles, Chapters, and Books’ , American Sociologist22: 147–158 .
5.
Butler, L. (2001) Monitoring Australia’s Scientific Research: Partial Indicators of Australia’s Research Performance. Canberra: Australian Academy of Science .
6.
Cole, S. (1983) ‘The Hierarchy of the Sciences?’ , American Journal of Sociology89: 111–139 .
7.
Cole, S. and J.R. Cole (1967) ‘Scientific Output and Recognition, a Study in the Operation of the Reward System in Science’ , American Sociological Review32: 377–390 .
8.
Cozzens, S.E. (1985) ‘Comparing the Sciences: Citation Context Analysis of Papers from Neuropharmacology and the Sociology of Science’ , Social Studies of Science15: 127–153 .
9.
Dieks, D. and H. Chang (1976) ‘Differences in Impact of Scientific Publications: Some Indices Derived from a Citation Analysis’ , Social Studies of Science6: 247–267 .
10.
Hargens, L.L. (2000) ‘Using the Literature: Reference Networks, Reference Contexts, and the Social Structure of Scholarship’ , American Sociological Review65: 846–865 .
11.
Harley, S. and F.S. Lee (1997) ‘Research Selectivity, Managerialism, and the Academic Labor Process: The Future of Nonmainstream Economics in UK Universities’ , Human Relations50: 1427–1460 .
12.
ISI (1998) Social Science Citation Index: 1997 Annual. Philadelphia, PA: Institute for Scientific Information .
13.
Lewison, G. (2001) ‘Evaluation of Books as Research Outputs in History’ , Research Evaluation10: 89–95 .
14.
Lodahl, J.B. and G. Gordon (1972) ‘The Structure of Scientific Fields and the Functioning of University Graduate Departments’ , American Sociological Review37: 57–72 .
15.
MacRoberts, M.H. and B.R. MacRoberts (1989) ‘Problems of Citation Analysis: A Critical Review’ , Journal of the American Society for Information Science40: 342–349 .
16.
MacRoberts, M.H. and B.R. MacRoberts (1996) ‘Problems of Citation Analysis’ , Scientometrics36: 435–444 .
17.
Moed, H.F. , W.J.M. Burger , J.G. Frankfort and A.F.J. van Raan (1985a) ‘The Application of Bibliometric Indicators: Important Field-and Time-Dependent Factors to be Considered’ , Scientometrics8: 177–203 .
18.
Moed, H.F. , W.J.M. Burger , J.G. Frankfort and A.F.J. van Raan (1985b) ‘A Comparative Study of Bibliometric Past Peformance Analysis and Peer Judgement’ , Scientometrics8: 149–159 .
19.
Moed, H.F. , M. Luwel and J. Nederhof (2002) ‘Towards Research Performance in the Humanities’ , Library Trends50: 498–520 .
20.
Najman, J.M. and B. Hewitt (2003) ‘The Validity of Publication and Citation Counts for Sociology and Other Selected Disciplines’ , Journal of Sociology39: 62–80 .
21.
Nederhof, A.J. and R.A. Zwaan (1991) ‘Quality Judgments of Journals as Indicators of Research Performance in the Humanities and the Social and Behavioral Sciences’ , Journal of the American Society for Information Science42: 332–340 .
22.
Nederhof, A.J. , R.A. Zwaan , R.E. de Bruin and P.J. Dekker (1989) ‘Assessing the Usefulness of Bibliometric Indicators for the Humanities and the Social and Behavioural Sciences: A Comparative Study’ , Scientometrics15: 423–435 .
23.
Nock, D.A. (2001) ‘Careers in Print: Canadian Sociological Books and Their Wider Impact (1975–1992)’ , Canadian Journal of Sociology26: 469–485 .
24.
Pfeffer, J. , A. Leong and K. Strehl (1977) ‘Paradigm Development and Particularism: Journal Publication in Three Scientific Disciplines’ , Social Forces55: 938–951 .
25.
Phelan, T.J. (2000) ‘Bibliometrics and the Evaluation of Australian Sociology’ , Journal of Sociology36: 345–363 .
26.
Seglen, P.O. (1992) ‘The Skewness of Science’ , Journal of the American Society for Information Science43: 628–638 .
27.
Solla Price, D. de (1986 [1963]) Little Science, Big Science … and Beyond. New York: Columbia University Press .
28.
Tijssen, R.J.W. (2003) ‘Scoreboards of Research Excellence’ , Research Evaluation12: 91–103 .
29.
van Raan, F.J. (1996) ‘Advanced Bibliometric Methods as Quantitative Core of Peer Review Based Evaluation and Foresight Exercises’ , Scientometrics36: 397–420 .
30.
van Raan, F.J. (2000) ‘The Pandora’s Box of Citation Analysis: Measuring Scientific Excellence – The Last Evil?’, pp. 301–319 in Blaise Cronin and Helen Barsky Atkins (eds) The Web of Knowledge. Medford, NJ: Information Today, Inc.
31.
Whitley, R. (1984) The Intellectual and Social Organization of the Sciences. Oxford: Clarendon Press .
32.
Wiberley, S.E., Jr (2003) ‘A Methodological Approach to Developing Bibliometric Models of Types of Humanities Scholarship’ , Library Quarterly73: 121–159 .
