Citation Metrics in Psychological Science

Abstract

We present norms for faculty citation counts based on 811 faculty members at 30 PhD-granting psychology departments in the United States across the range of the National Research Council rankings. The metrics were highly skewed, with most scientists having a low to moderate number of citations of their work and a few scientists having extremely high numbers. However, the median per-year citation count was 149, showing widespread scientific contributions across scholars. Some individuals in lower ranked departments are more highly cited than the average scholar in higher ranked departments, with enormous variation in citation counts in both the low- and high-ranking departments. Citation counts overall have risen in recent years, and the citations of early-career scholars are increasing at a faster rate than their senior colleagues did at the same point in their careers. We found that citation counts at the beginning of scientists’ careers substantially predict lifetime citation success. Young scholars’ citation counts are associated with obtaining positions at higher ranked universities. Finally, we found no significant differences for subfields of psychology. In sum, although a few highly productive scientists have a very large influence, trends reveal that contributions to psychological science are growing over time, widespread, and not limited to a few stars and elite departments.

Keywords

academic career trajectory academic psychology application: education citation counts intragroup processes

In this article, we provide norms to help evaluate the citation counts of psychological scientists. Using all the regular psychology faculty members in psychology departments at 30 randomly selected universities that offer doctoral degrees in psychology, we present norms for several citation indicators along with age-adjusted norms. The goal of this article is to provide empirically based norms to interpret citation counts, complementing or replacing the intuitive approach that is often used to evaluate such numbers. Citation metrics are one source of information that can be used in hiring, promotion, awards, and funding, and our goal is to help these evaluations. We examined additional questions such as whether early-career counts accurately predict later career numbers and whether citation counts are increasing in the field.

Rushton (1984) argued that citation counts are helpful because they are a quantifiable, objective, and readily available source of information on the esteem in which scientists are held by the field. Likewise, Ruscio et al. (2012) described the advantages of citation metrics in evaluating scientific records. Diener et al. (2014) found that citation metrics converged significantly with other measures of scholarly influence such as textbook coverage and expert ratings. Endler et al. (1978) found that citations correlate more highly with reputational ranks than did numbers of publications. Cacioppo (2008) described the strengths of citation metrics as well as their limitations. He pointed out their strengths in relation to other sources of information such as letters of recommendation. Sternberg (2018) recognized the limitations of citation metrics but also argued that they have many advantages over qualitative evaluations, for example, less bias. However, the advantages and disadvantages of citation counts are not the focus of this article, which is directed at people who would like to consult citation numbers as one source of useful information.

Our work builds on the earlier analysis of citation metrics in psychology by Joy (2006). We seek to update the norms and conclusions of Joy as well as answer additional questions he did not address. We have used a broader database (Google Scholar) and focused only on research institutions. In addition, we offer specific norms for evaluating citation metrics as well as norms for several different citation metrics. We do not cover newer metrics of impact, such as Internet views and downloads, because they are beyond the scope of this article, but readers are alerted that they may be valuable additional sources of information.

Method

During 2018, we collected the names and citation metrics of all regular faculty members at 30 departments of psychology in the United States that offer doctoral degrees in psychology. We randomly selected 20 departments from the 1995 ranking by the National Research Council (NRC) of psychology departments with doctoral programs. The goal was to obtain a broad representative sample of faculty in research departments that offer a doctoral degree in psychology. Because the initial random selection ended up by chance overrepresenting highly ranked departments, we randomly selected 10 additional departments from the lower ranks. This resulted in a broad representation of departments across the entire range, with a mean ranking very close to the midpoint of the rankings (92.8). Departments varied in size from 12 to 58 faculty members (M = 27), with a total of 811 scholars. One measure was removed for several scholars because of inconsistencies or uncertainties about their publications, leaving 810 scholars with an h index and a most-cited article and 806 scholars with a total citation count.

One possible concern is that the NRC rankings are several decades old and therefore do not reflect the current quality of departments. To check on this possibility, we compared the NRC rankings with the scores produced by U.S. News and World Report (“Best psychology schools,” 2017), which were based on survey ratings of the departments by psychologists drawn from doctoral departments. Another concern is that rankings can be misleading because they may inflate small absolute differences in scores. Furthermore, rankings have a flat distribution rather than a normal one. To examine whether these issues are in fact problematic, we correlated at the departmental level both the scores and rankings from the NRC and U.S. News lists. All of the correlations were quite high, suggesting that there has not been dramatic change in the quality of departments over time and that rankings and scores produce essentially the same results. The NRC department scores and ranks were correlated at .99 (p < .001), and the U.S. News ranks and scores were correlated at .97 (p < .001). The NRC rank and U.S. News ranks correlated .90 (p < .001), and the NRC scores and the U.S. News scores correlated .94 (p < .001). Thus, there do not seem to be serious issues regarding using the NRC rankings as an estimate of departmental prestige. Although the correlations allow for some departmental movement in the rankings, the overall norms still represent the entire range of rankings well.

We used Google Scholar citation counts in 2018 as a primary source of citation data. Although various citation search engines have their champions, a defense of Google Scholar as the most inclusive and democratic can be found in Harzing (2017). Empirical work on citation-search computer programs suggests that Google Scholar is reliable and broad. Martin-Martin et al. (2017) found that Google Scholar can effectively identify highly cited articles. Furthermore, Martín-Martín et al. (2018) found that Google Scholar tended to be more inclusive than other search sources and was more likely to find citations in non-English sources, theses, unpublished material, and conference proceedings. However, they found that the various search engines produced counts that are very highly correlated even when they differ in mean numbers of citations uncovered. In a recent article, Gusenbauer (2019) compared 12 citation search engines and found that Google Scholar was the most comprehensive academic search platform.

When faculty members had a Google Scholar profile, we relied on the citation numbers contained in it. When they did not, we used Harzing’s (2016) Publish or Perish website software to compute the citation counts. Data on scholars’ PhD year and university were obtained from websites and other sources. The faculty who were included in our tally were professors with regular appointments; we did not include postdoctoral fellows, emeritus faculty, and visiting faculty. We did include faculty who were listed as instructors, teaching professors, or lecturers.

Results

Descriptive data

The 30 departments of psychology included in the study are shown in Appendix A along with their rankings and number of faculty members.

Table 1 presents descriptive statistics for total citation counts, the h index, and the most cited article as well as age-adjusted numbers for total citations and the h index. The total citations are those the scholar has accumulated across his or her entire career, and the most cited measure is the scholar’s article that has been cited the most across time. The h index is the highest number representing that number of publications that have been cited that number of times or more. The age adjustments were performed by dividing total citations and the h index by the number of post-PhD years. The age-adjusted numbers might be useful in comparing people considering the stage in their careers, whereas the unadjusted norms reflect the absolute influence of scholars. The distributions are highly positively skewed.

Table 1.

Descriptive Statistics for the Citation Metrics for Psychology Faculty

Statistic	M	SD	Skew	Median
Total citations	7,721	17,765	6.54	2,536
h index	29	24	1.83	22
Top citation	1,101	2,959	7.84	378
Age-adjusted total citations	300	509	5.26	149
Age-adjusted h index	1.6	1.2	3.23	1.3
Age-adjusted most-cited publication	45.9	82.8	6.1	22.3

Citation norms

Norms for the citation metrics are presented in Table 2. Readers are cautioned that because citation counts seem to be rising over time, the norms presented here should probably not be used for more than a few years, at which time updated norms will be desirable. Perhaps a scientific organization could create a computer program that would periodically update norms. It also should be possible to create norms that are designed for specific groups, such as fields of psychology, university position, and type of institution. Because by chance among the 30 universities selected there were three Ivy League universities ranked in the top 10 of departments, our percentile norms might be slightly low, especially for higher citation counts. The percentiles for members in the upper half of the distribution might be several points higher in percentile terms than those shown in the table, according to our analyses in which several highly ranked departments were deleted.

Table 2.

Citation Normative Statistics

Percentile	Total citation count	h index	Top cited publication	Age-adjusted total citations	Age-adjusted h index	Age-adjusted most-cited publication
1st	3	1	3	.6	0.05	0.21
5th	79	4	26	8	0.27	2.04
10th	221	6	59	19	0.44	4.00
20th	576	11	120	45	0.68	8.42
30th	1,045	15	188	72	0.92	12.33
40th	1,714	18	277	108	1.13	16.76
50th	2,534	22	378	149	1.33	22.33
60th	3,562	27	543	198	1.55	30.00
70th	5,266	34	775	267	1.82	41.43
80th	8,726	42	1,103	388	2.20	60.33
90th	17,251	58	2,158	727	2.80	96.48
95th	32,716	76	3,945	1,091	3.60	150.82
99th	87,621	113	11,984	2,221	6.20	387.98

A possible concern with age-adjusted norms is whether to use the year the PhD was received or the year of first publication for the adjustment. Some individuals begin publishing years before they earn a PhD, and others publish only after earning a PhD. Although these two different numbers might be considered for scholars whose career trajectories are unusual, we found that total citations age adjusted on the basis of the year the PhD was received and on first publication were correlated (r) at .99 (p < .001). Thus, although in individual cases evaluators may consider whether a particular person published many years before earning a PhD, or only later, in general, this should not make a substantial difference for the field as a whole. However, as we continue to encourage, thoughtful consideration of the individual facts in each case is needed for evaluating the records of individual scholars.

One caveat is that our metrics relied on Google Scholar. Thus, if citation counts are collected from another source, such as the Web of Science, they will not be comparable with our norms.

Age adjustment

One criticism of citation counts to reflect scholarly impact is that such indicators tend to favor older investigators, who have had more time to publish and be cited. We therefore offer age-corrected norms by dividing citations by number of post-PhD years for each scholar to create an age-adjusted score both for total citations and the h index, as suggested by Hirsch (2005) for his h index. This adjustment substantially reduces the correlation between total citations and post-PhD years, although it does not completely do so (r = .36 vs. r = .19 for the age-corrected total-citation index). However, for the h index, we found that the age adjustment gives a clear advantage to young scholars (unadjusted r = .52, adjusted r = −.32).

Age adjustments may make the most sense for scholars in the early stages of their careers; for older scholars, the full career contributions seem more important than an indicator that differentiates between senior scholars and very senior scholars. We found that, for the cohort that had received a PhD more than 20 years ago, age-adjusted total and uncorrected total citations were correlated at .97 and the h index correlation was .91, suggesting that age adjustments are not highly influential once scholars reach senior status. For the youngest cohort, we computed the correlation between our age-adjusted indicator and years for scholars who had received a PhD 10 or fewer years ago, during which hiring and tenure decisions most frequently occur. In this case, the correlation between years and age-adjusted total citations was r(190) = .07 (n.s.). Thus, when evaluating individuals who are early in their career, our simple age adjustment of total citations does reasonably well at eliminating years of scholarship as an influence. Care must be taken in using age adjustment of the h index, however. For the young cohort, this adjustment changed the correlation of years and the index from .42 to –.39, indicating that it penalizes the more senior scholars in the young group.

In sum, age adjustments for total citations seem not to strongly favor a particular age group, especially in comparisons among young scholars. However, age adjustments for the h index should be used cautiously because it seems to penalize older scholars regardless of whether the comparison is made across all professors or only among young scholars. Thus, a possible advantage of total citations compared with the h index is that age adjustments seem to be more able to equate age cohorts using it.

Comparing citation metrics

The correlations between the three citation indicators, both unadjusted and age adjusted, are shown in Table 3. The measures converge substantially with each other, with the exception of the age-adjusted h index, which correlates weaker with the other measures. This finding again raises the question of whether age adjusting the h index is that helpful given that it seems to overcorrect in favor of early-career professors and does not correlate strongly with the other measures.

Table 3.

Correlations Between Citation Indicators

Citation indicators	Total citations	h index	Most cited publication	Age-adjusted total citations	Age-adjusted h-index
h index	.81	—
Most cited publication	.87	.59	—
Age-adjusted total citations	.94	.80	.79	—	—
Age-adjusted h index	.28	.36	.18	.49
Age-adjusted most-cited publication	.79	.54	.93	.82	.35

Note: All correlations p < .001.

Different citation metrics have been championed by different people, but note that each can provide new information and be used for different purposes (Cacioppo, 2008). The most-cited-article indicator has the advantage of capturing the single most influential single contribution of a researcher. The h index has the advantage of signifying a broad set of important contributions to a field, for example, representing many influential empirical articles. In contrast, total citations can be based on a few very influential contributions, such as books and review articles. Thus, the citation indicators often reflect different types of scholarship.

An argument can be made that the h index is superior to total citation counts because it includes both the number of influential publications and the numbers of citations of them. Ruscio et al. (2012) reported that the h index predicted departmental rankings and faculty status (professor, associate, assistant) more strongly than total citations. We correlated both total citations and the h index with departmental rank and faculty status. The h index had a stronger association with both dependent variables, although both had statistically significant associations with department ranks (rs = .32 and .46) and faculty status (rs = .27 and .49). However, for age-adjusted scores, the total citations produced stronger associations; although, again, both associations were significant: department rank (rs = .38 and .30) and faculty status (rs = .23 and –.08). The inverse figure for the prediction of faculty status suggests that perhaps age adjustments to the h index may favor younger scholars.

Citation metrics and career trajectories

What factors predict hiring at prestigious departments? Hiring in highly ranked departments seems to depend in part on having received a PhD from a highly ranked department as well as one’s early-career scholarly impact. We predicted the rank of scholars’ departments from the rank of the PhD-granting department and from scholars’ citation count summed across the first 3 years after they had received the PhD. Both the rank of the PhD-granting department and the early citation count were significantly associated with the rank of scholars’ faculty departments (rs = .48 and .34, ps < .001). When combined in a regression analysis together, both significantly predicted the rank of the professors’ departments (βs = 0.46 and 0.26, both ps < .001). Of our total sample, approximately 23% moved up in rank from their PhD program to their current position, and 68% moved down.

Given the apparent influence of early citation count on being hired by a prestigious department, how informative is a person’s early-career impact in actually accurately predicting long-term impact? Should hiring and tenure decisions be influenced by the early-career record? For scholars who had received PhDs more than 20 years ago, Table 4 presents the correlations of career citation counts with three early-career periods. There is a very large amount of stability in the early years in terms of citations. For example, post-PhD Years 5 and 10 correlate extremely highly. Furthermore, post-PhD Years 1 through 3 do about as well as post-PhD Years 5 or 10 in predicting lifetime citations. The associations suggest that early-career citations do a reasonably good job in predicting later citation success and that there is substantial stability in citations through scholarly careers. One large caveat is that these associations are based on only 29 individuals because early citations counts were not available for most older scholars.

Table 4.

Early-Career Metrics and Long-Term Influence

Measure	Citations
Measure	Post-PhD Years 1–3	Post-PhD Year 5	Post-PhD Year 10	Total career
Citations
Post-PhD Year 5	.93
Post-PhD Year 10	.91	.96
Total career	.68	.67	.68
Career h index	.71	.68	.64	.94

Note: N = 29. All ps < .001.

Citations and NRC departmental rankings

The NRC ranks of departments correlated with the citation metrics, which is not surprising given the differences in resources and priorities between departments. However, it is worth noting that there is a large variation in citations of individuals within departments such that a highly cited person in a department with a low ranking often outperformed many people in highly ranked departments. We divided the departments into those ranked in the top 50, the next 50, and the remaining 85. The mean age-adjusted total citations did, indeed, vary among the three departmental rank categories (M = 541.8, SD = 709.7; M = 208.7, SD = 308.9; M = 124.7, SD = 169.2), F(2, 790) = 66.8, p < .001 (see Fig. 1). However, the highest age-adjusted individuals in the two lower categories (2,829.2 and 1,324.7) greatly exceeded the mean for the highest department rank category. The bar chart in Figure 1 makes clear that the age-adjusted means vary substantially among the three categories. However, it is also evident that the standard deviations are extremely large within categories, indicating a huge amount of overlap in age-adjusted citations among them.

Fig. 1.

Age-adjusted total citations for department rank categories. Error bars indicate ±1 SD.

Although the mean h index also varied among department rank categories (41.9, 25.8, and 18.9), the highest individual in each of the two lower two categories (100 and 101) greatly exceeded the mean for the upper category. In the second departmental rank category is a professor who has been cited 101,852 total times, and in the lowest category is a scholar who has been cited 37,091 times. Of the top five most highly cited scholars in our sample, three were not in departments ranked in the top 25.

In the seven departments we included that have NRC rankings lower than 150, we found that the faculty have metrics that do point to scientific contributions. For example, the total citations for these professors have a median of 691 and an age-adjusted median of 44. Thus, the average professor in these departments is cited a nontrivial amount. Only three faculty members had no citations. Thus, it must not be forgotten that although highly ranked departments have more stars, there are many people in lesser ranked departments who have significantly contributed to psychological science.

Overall, how important is an individual’s institution in determining scholarly impact? We examined the amount of variance due to differences between departments in explaining the overall variances in age-adjusted total citations. When age-adjusted total citations was predicted by departmental rank scores, the adjusted R² was .14 and the age-adjusted h index was 0.09. These estimates for amount of variance predicted by department rank indicate that although the university is influential, individual differences within departments appear to be much more so.

Citations metrics across subfields of psychology

Is publishing and gaining a high citation count more difficult in some areas of psychology than in other subdisciplines? Our analyses of four subfields of psychology (neuroscience and cognitive, developmental, clinical, social/community/organizational/personality, and omitting the “other” category) did not indicate significant differences in citations between the subfields. Two citation metrics are shown in Table 5 for the four subfields. A multivariate analysis of variance (ANOVA) on the two metrics did not yield a significant difference, nor was the difference significant across fields for either of the two citation metrics. When individual comparisons were made between each of the subfields, none were significant.

Table 5.

Citation Metrics for Subfields of Psychology

Metric and subfield	M	SD
Age-adjusted total citations
Neuroscience	326.5	500.5
Developmental	231.8	270.6
Clinical	314.7	610.3
Social, personality, community, and organizational	325.1	524.9
Age-adjusted h index
Neuroscience	1.58	1.04
Developmental	1.44	1.04
Clinical	1.77	1.69
Social, personality, community, and organizational	1.57	1.19

Although there do appear to be nontrivial differences between subdisciplines (with the lowest numbers being for developmental psychology), the very large variance within each subfield resulted in nonsignificant mean differences. Thus, although there might be differences in citation metrics between subfields of psychology, these appear to be overshadowed by the huge variability within them. However, note that if one were to examine the areas of psychology in a more fine-grained way, one would be more likely to find differences in citation metrics. For example, a scholar doing research on taste aversion might be challenged to achieve the same citation count as another scholar conducting research on prejudice given the numbers of scholars working in those two areas. Although one cannot automatically assume that some broad fields of psychology produce much higher citation counts than other broad fields, this does not mean that careful thought is not required in judging citation counts, especially for scholars in smaller and less popular research areas.

Rising citation metrics

Citation counts for new scholars appear to be rising over time (Joy, 2006). We analyzed citation counts in the early years for the middle cohort (PhD 1981–2000) compared with the younger cohort (PhD 2001–2015). Individual year citation counts were not available for the early careers of the oldest cohort, and so they could not be included in this comparison. We compared the two cohorts for their summed citations counts for post-PhD Years 1 through 3, Year 5, and Year 10. A multivariate ANOVA for the three metrics showed that the two cohorts differed marginally significantly, F(3, 142) = 2.64, p < .053. For post-PhD Years 1 to 3 summed, the middle cohort averaged a total citation count of 171.0, and the younger cohort count was 248.1, F(1, 144) = 4.28, p < .05. For post-PhD Year 5, the comparison was 114.1 compared with 185.5, F(1, 144) = 7.19, p < .01, and for post-PhD Year 10, it was 246.4 compared with 431.2, F(1, 144) = 7.40, p < .01. It appears that both soon after graduate school and in their early career, faculty members in the newer cohort had considerably more citations than their predecessors. Furthermore, in a repeated measures ANOVA that included post-PhD Years 1 through 3 and Year 10, there was a significant interaction between year and cohort, F(1, 146) = 5.01, p < .05, indicating that the younger cohort’s citations grew more rapidly over the early-career years.

The trends of increasing citations could be due to increased productivity, the use of computers for statistical analyses and literature searches, a greater number of authors per article, or increasing numbers of journals and researchers in the field. Whatever the explanation, citation counts are rising over time.

Other citation metric adjustments

There are countless corrections and other aspects of citations that people might consider, such as the number of authors who contributed to a particular article or the number of publications in which a particular author has (a) cited themselves, (b) been the sole author, (c) been the first author, and (d) been the last author. Our norms are based on simple metrics and do not adjust for the large number of other variables that can be considered in interpreting citation metrics. Ioannidis et al. (2019) discussed various adjustments for citation metrics and presented a list of the most highly cited scientists, including more than 3,000 behavioral scientists, based on various adjustments, such as for self-citations. When we examined their adjusted metrics, the figures with and without self-citations were virtually identical. For total citations, the two were correlated at .996 and .99 for the h index. The total citations and the h index were correlated at .84 for all citations and .83 for those without self-citations. Thus, general concerns about inflation from self-citations seems unwarranted, although they might make a difference in particular cases. Citations from publications in which an author was the only author, the first author, or the last author combined were correlated at .93 with total citations. The conclusion would be that although in a few cases adjusting citation metrics for various factors will alter conclusions, in the majority of cases, adjustments for authorship order and self-citations are not necessary. This conclusion converges with the findings of Ruscio et al. (2012), who found that citation counts adjusted for self-citations correlated very highly with unadjusted scores.

Note that people comparing metrics between scholarly fields may find metric adjustments are important because our analyses above are not necessarily applicable in comparing across disciplines. It may also be that despite the high correlations, there are a few individuals who have high citation counts primarily because they publish with a highly cited scientist or helped author a publication with a huge number of authors.

Future metrics

Because citation counts across psychology seem to be increasing, it will be important to have norms that are updated every few years, and this should become increasingly easier with computer software. In addition, Altmetric attention scores may become helpful, such as the reads, views, and downloads of publications (Trueger et al., 2015). Additional types of scores can also assess coverage, such as in Wikipedia and the popular press. Besides Wikipedia coverage, Diener et al. (2014) also examined measures such as textbook coverage and awards. Each of the different measures of impact has strengths and a particular focus but are beyond the scope of the current article, which is focused on providing citation information.

Discussion

This article is intended as a resource for people who would like to use citation counts as a metric of scholarly influence in academia. The merits and demerits of citation counts are beyond the scope of the current article. Some scholars prefer their own and their colleagues’ judgments over objective metrics. However, personal judgments have shortcomings, such as the biases of individuals, the time required to arrive at informed judgments, and the inability of scholars to be fully familiar with all of the subdisciplines in which judgments are required. In addition, a common finding in psychology is that actuarial judgments often outperform individual judgments.

The argument in favor of personal judgments overlooks the fact that citation counts are also based on judgments by scholars. In the case of citation counts, however, those judgments are broadly derived from the whole scholarly community and are weighted by the scholars who are publishing about the topic of the cited publications. Thus, there is much to recommend citation numbers in evaluating scholarly records.

Although personal judgments have shortcomings and objective metrics have strengths, there are also limitations to citation metrics. For one thing, they tend to reward articles that present a statistic or a measurement scale, which may be very highly cited. For example, textbooks that present standard statistics may be cited tens of thousands of times. Thus, using citation counts alone for evaluations could have the undesirable effect of overly rewarding the writing of review articles instead of original empirical discoveries. Problems of citation counts related to issues of self-citation and differing numbers of authors on articles do not seem in general to be major problems. This does not mean, however, that these factors might not be problematical in individual cases. To take an extreme example, the articles confirming the existence of the Higgs boson had over 5,000 authors (ATLAS Collaboration et al., 2012). Thus, investigators could have very highly cited articles on their scholarly record, but this might be misleading as to the importance of their contribution. Thus, like all statistics, citation metrics should be used thoughtfully.

A number of conclusions can be drawn from our citation data.

First, virtually all faculty members at all research universities have been an author of publications that have been cited. Faculty members in doctoral departments with no cited publications are extremely rare. Only three scholars out of 811 had no citations at all. At the same time, a few faculty members contribute a disproportionate share of the influential psychology literature. This finding replicates the fact noted by J. R. Cole and Cole (1972) in the physics literature that a few scientists contribute a very substantial amount to scientific progress. The field needs to encourage such productive individuals but also ensure that inequalities in scientific productivity are not due to external impediments for some scholars. Although a few scientists contribute hugely to the scientific literature, we also found that less productive scientists also make significant contributions. The differences in citation counts between departments, as well as the large variability of ranks within departments, are consistent with Simonton’s (2003) claim that both personal attributes and situational resources contribute to scholarly creativity.

Second, reasons have been advanced to favor various citation metrics over others, suggesting that certain metrics are superior. However, our data reveal that these various metrics are very highly correlated and also that at times they predict incrementally—suggesting that each adds information beyond the other. Furthermore, in a reanalysis of the data reported by Diener et al. (2014), we also found that both total citations and the h index were significantly associated with both awards and textbook coverage.

As we noted above, citation counts can be achieved in different ways. S. Cole and Cole (1967) pointed out that citation counts can be achieved by publishing many articles that are moderately cited or a few articles that are highly cited. For this reason, it can be very helpful to examine additional metrics beyond total citations, such as the h index. Evaluators need not pit the various measures against each other, asking which is the correct one to use, but can consult each of them. In some cases, the information across the metrics will be largely redundant, but in other cases, one metric will reveal a larger influence for a particular scholar compared with another metric.

Third, Early citation count is important in determining the ranking of a university. Although there is a tendency for professors in highly ranked programs to have come from highly ranked PhD programs themselves, there are clear examples of people who have risen from lower ranked programs to higher ranked ones. Thus, a person’s PhD program can influence place of employment as a faculty member, but a very strong publication record will also be helpful.

Fourth, the early publication record, reflected in the citations in post-PhD Years 1 to 3, is strongly predictive of the age-adjusted career citation record, in this case assessed in professors who had received a PhD more than 20 years ago. Thus, in most cases, it is possible for departments to make reasonably sound judgments about hiring and tenure of young scholars on the basis of the early record.

Fifth, we did not find significant differences in citations among subfields of psychology. However, this does not mean that a more fine-grained analysis of the subfields would not indicate differences.

Sixth, we found that citation counts are rising over time so that young scholars tend to have higher citation counts than their older colleagues did at the same time in their careers. The younger cohort also showed a significantly steeper rise in their counts from post-PhD Years 1 through 3 to Year 10. We do not know whether this is due to the rising numbers of journals and publications, greater productivity in general compared with the past, a greater number of authors per article, or greater competition for professorships in doctoral departments.

Finally, we discovered that adjusting for post-PhD years left a very modest advantage for older scholars for total citations but gave a very clear advantage to younger scholars for the h index. When adjusting for age, some junior scholars had an extremely high h index. Although the advantage shown for younger scholars might reflect growing productivity over the years, it could also reflect the fact that dividing by only a small number of years can sometimes result in quite high values. Thus, we caution that comparing across age cohorts—comparing young scholars with senior scholars, for example—must be done with extreme caution.

Our analysis of citation metrics follows that of Joy (2006). Many of our findings replicate those of the earlier analysis. For example, the immediate post-PhD citation metrics were also found by Joy to be predictive of later career productivity. Joy also found that a few extremely highly cited individuals account for a large percentage of total citations.

We cannot confirm that rising citation counts are beneficial for the science of psychology. However, if one assumes that the science is sound, it seems reasonable to assume that more citations could indicate a faster increase in knowledge, which would be consistent with Pinker’s (2018) claim that science and technology are increasing rapidly as well as Haslam et al.’s (2016) finding that there has been a steep rise in citation counts in psychology in Australia. However, our findings on rising metrics are different from the conclusions drawn earlier by Joy (2006), and we are not certain why. It could be that since 2006, there has been an uptick in scholarly output by young scholars, it might be due to our focus exclusively on doctoral-degree-granting institutions, or it might be that Google Scholar has yielded a more complete picture. For whatever reason, we found a greater rate of publications for young compared with older scholars at the same points in their careers.

What accounts for the consistency of citation counts for most scholars through their careers and the huge range of counts between different scholars? Others have also found that research productivity for most psychologists shows high stability throughout their careers (Joy, 2006). According to the cumulative advantage hypothesis, also known as the Matthew effect (Allison & Stewart, 1974; Bol et al. 2018), people who start out productively receive more incentives for productivity through funding and are more likely to gain professorial positions and to stay in them, and the oldest cohorts of professors are those who have been most motivated to publish and who gained the highest recognition for their publications, accounting for the positive correlation between age and publication rate. Thus, the most productive individuals are given more resources because of the quality of their early work, which in turn makes them even more productive.

Our findings are encouraging in showing that most psychology faculty have been involved in research that has been cited and also that citation counts are rising over time. The field does rely on very productive scholars, but in addition, a large number of faculty are authors of articles that are being cited. Indeed, professors at the median have been cited 148 times a year, have 22 publications that have been cited 22 or more times, and have a top-cited article with 378 citations. More than 90% of our sample had a publication that has been cited more than 100 times. Scientific stars are very important to the field, but we must not lose sight of the fact that a large number of faculty members are contributing to the psychology literature, even faculty who might be primarily involved in teaching, administration, or the delivery of services. The involvement of most psychologists in research not only pays research dividends but also can be helpful in terms of teaching and mentoring as well as helping professors stay current in their knowledge. Thus, we can appreciate the contributions made by eminent scientists and at the same time be grateful that there are scholarly contributions from a very broad set of scholars and that citations in the field are growing over time.

Footnotes

Appendix A

Table A1.

Departments, Rankings, and the Number of Faculty

Doctoral department	Departmental NRC ranking	Number of faculty included
Akron	143	19
American University	129	24
Arkansas	153	19
Biola	179	23
Brandeis	75	12
Clark	52	14
Columbia	17	18
Farleigh Dickinson	170	20
Fordham	148	27
George Mason	104	42
Georgia Institute of Technology	75	23
Harvard	6	27
Louisiana State	126	26
Louisville	111	25
Mississippi	137	25
Notre Dame	90	39
Oklahoma State	166	24
Pennsylvania State	30	51
Princeton	13	31
Rochester	30	18
Stanford	1	28
Texas at Austin	17	58
Texas Woman’s University	184	16
Tulsa	111	12
Nebraska–Lincoln	97	31
North Texas	159	27
Northern Colorado	172	16
University of Washington	13	48
Wayne State	83	41
Yale	4	27

Note: Tied departments’ rank values rounded to next highest integer above. NRC = National Research Council.

Acknowledgements

Our sample presents an appropriate representation of academic psychologists both in size and in distribution. The presented article was not formally preregistered. Neither the data nor the materials have been made available on a permanent third-party archive. Requests for the data or materials can be sent via email to the corresponding author. The current data are archival in nature and did not require obtaining novel information from human participants. The current research follows the American Psychological Association Code of Ethics and the World Medical Association’s Declaration of Helsinki.

Transparency

Action Editor: Dean Simonton

Editor: Laura A. King

ORCID iD

Nina Radosic

References

Allison

P. D.

Stewart

J. A.

(1974). Productivity differences among scientists: Evidence for accumulative advantage. American Sociological Review, 39, 596–606.

ATLAS Collaboration, Aad

Abajyan

Abbott

Abdallah

Abdel Khalek

Abdelalim

A. A.

Abdinov

Aben

Abi

Abolins

AbouZeid

O. S.

Abramowicz

Abreu

Acharya

B. D.

Adamczyk

Adams

D. L.

Addy

T. N.

Adelman

. . . Zinonos

(2012). Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Physics Letter B, 716(1), 1–29. https://doi.org/10.1016/j.physletb.2012.08.020

Best psychology schools. (2017). U.S. News and World Report. https://www.usnews.com/best-graduate-schools/top-humanities-schools/psychology-rankings

Bol

de Vaan

van de Rijt

(2018). The Matthew Effect in science funding. Proceedings of the National Academy of Sciences, USA, 115, 4887–4890.

Cacioppo

J. T.

(2008). Metrics of science. APS Observer, 21(1). https://www.psychologicalscience.org/observer/metrics-of-science

Cole

J. R.

Cole

(1972). The Ortega hypothesis. Science, 178, 368–375.

Cole

J. R.

(1967). Scientific output and recognition: A study in the operation of the reward system in science. American Sociological Review, 32, 377–390.

Diener

Oishi

Park

(2014). An incomplete list of eminent psychologists of the modern era. Archives of Scientific Psychology, 2, 20–31.

Endler

N. S.

Rushton

J. P.

Roediger

H. L.

(1978). Productivity and scholarly impact (citations) of British, Canadian, and U.S. departments of psychology (1975). American Psychologist, 33, 1064–1082.

10.

Gusenbauer

(2019). Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics, 118, 177–214.

11.

Harzing

A-W.

(2016). Publish or perish. Harzing.com. https://harzing.com/resources/publish-or-perish

12.

Harzing

A.-W.

(2017, February 28). Google Scholar is a serious alternative to Web of Science. Harzing.com. https://harzing.com/blog/2017/02/google-scholar-is-a-serious-alternative-to-web-of-science

13.

Haslam

Stratemeyer

Vargas-Saenz

(2016). Scholarly productivity and citation impact of academic psychologists in Group of Eight universities. Australian Journal of Psychology, 69, 162–166.

14.

Hirsch

J. E.

(2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, USA, 102, 16569–16572.

15.

Ioannidis

J. P. A.

Baas

Klavans

Boyack

K. W.

(2019). A standardized citation metrics author database annotated for scientific field. PLOS Biology, 17(8), Article e3000384. https://doi.org/10.1371/journal.pbio.3000384

16.

Joy

(2006). What should I be doing, and where are they doing it? Scholarly productivity of academic psychologists. Perspectives on Psychological Science, 1, 346–364.

17.

Martin-Martin

Orduna-Malea

Harzing

A.-W.

López-Cózar

E. D.

(2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11, 152–163.

18.

Martín-Martín

Orduna-Malea

Thelwall

López-Cózar

E. D.

(2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12, 1160–1177.

19.

National Research Council. (1995). Research doctorate programs in the United States: Continuity and change. The National Academies Press. https://doi.org/10.17226/4915

20.

Pinker

(2018). Enlightenment now. Penguin Books.

21.

Ruscio

Seaman

D’Oriano

Stremlo

Muhalchik

(2012). Measuring scholarly impact using modern citation-based indices. Measurement, 10, 123–146.

22.

Rushton

J. P.

(1984). Evaluating research eminence in psychology: The construct validity of citation counts. Bulletin of the British Psychological Society, 37, 33–36.

23.

Simonton

D. K.

(2003). Scientific creativity as constrained stochastic behavior: The integration of product, process, and person perspectives. Psychological Bulletin, 129, 475–494.

24.

Sternberg

(2018). Evaluating merit among scientists. Journal of Applied Research in Memory and Cognition, 7, 209–216.

25.

Trueger

N. S.

Thoma

Hsu

C. H.

Sullivan

Peters

Lin

(2015). The altmetric score: A new measure for article-level dissemination and impact. Annals of Emergency Medicine, 66, 549–553.