Faculty Performance Evaluation Using Citation Analysis

Abstract

This article provides a citation analysis for faculty from Association of Collegiate Schools of Planning (ACSP) member schools. The article argues that Google Scholar data is a particularly valuable source of citation data for urban planning because its coverage extends beyond traditional peer-reviewed publications. The analysis reports the level of scholarly activity within the urban planning discipline. The results show citation patterns for planning faculty, departments, and universities along with discussing the distribution of citation activity across the discipline. The article concludes by encouraging planning scholars and administrators to undertake more analysis of planning scholarship to understand scholarly performance and impact.

Keywords

citation analysis bibliometrics research evaluation scholarship

Introduction

Academic discovery occurs through new thoughts, ideas, and perspectives developed through the scholarly process. These discoveries are then expressed in tangible ways so they can be communicated to interested academics and practitioners. Marchionini (2010) describes this process as converting the mental to the physical in the form of actionable information. For social scientists, we commonly see the physical expression of these “artifacts” as books, chapters, journal articles, and other types of published reports and documents. More recently, these artifacts have started to take digital form as blog entries, online articles, and other electronic multimedia. Part of the research process involves search and review of these artifacts, selecting and referencing (i.e., citing) influential pieces that lend support to statements made by authors. This generates a network of citations that can be both quantitatively and qualitatively evaluated with data from citation indexes such as Scopus, Web of Science, and Google Scholar. The connections in the network (the citations) represent the influence and flow of information over time, from one “artifact” to another.

Citing credible sources shows that authors have investigated the topics being discussed and therefore are making informed and reliable statements. Citations are considered valid based on the reputation of the cited authors or the reputation of the publications in which they appear (Garfield and Merton 1979; Moed 2006). References to non-published materials such as speeches, web sites, and government publications, also rely on the reputation and authority of the source and are often cited. Researchers have found several motivations for citation activity (see Bornmann and Daniel 2008; Campbell 2011). Some concern has been expressed about self-citation as a form of self-promotion and to increase citation counts elevating the perceived importance of an author’s publication (MacRoberts and MacRoberts 1996). While some worry about inflated citation metrics resulting from self-citation, Harzing (2010) explains that manipulation is less a concern for individual authors as much as it is for journal impact factors. Journal impact factors are more susceptible to self-citation because relatively small numbers of citations can produce a significant change compared to that of an individual, especially for those with large numbers of citations (see Stevens 1990). Nevertheless, as planners are well aware, few measures are immune from being manipulated.

Quantitative measures of academic output and impact have been used to assess performance, especially for academic promotion and tenure. The pressure to “publish or perish” within academia stresses the importance of scholarship, especially during the academic probationary process. Productivity can be a critical factor when arguing for scarce resources, when comparing academic programs and universities that are competing in global education and research markets (Goldstein and Maier 2010; Arimoto 2011; Linton, Tierney, and Walsh 2011). Productivity and impact measures are often debated, as are perspectives about the meaning of citations, and the weight given to journal impact factors (see Goldstein and Maier 2010; Sawicki 2011; Goldstein and Maier 2011; Freestone 2011; Salet and Boer 2011; Webster 2011; Campbell 2011; and Stiftel 2011 in this journal for an excellent discussion about journal impact factors in evaluating planning scholarship). Assessment of scholarship remains a challenge with arguments made for both quantitative and qualitative approaches. This article hopes to invigorate further discussion around research evaluation for urban planning, which intersects academic and professional realms. For university faculty and administrators, the question is often about how to evaluate the significance or impact of scholarly output.

Citation analysis for scholarly evaluation has an extensive literature that weighs appropriateness within and across disciplines as well as offering nuanced discussion of a range of metrics (see, e.g., Garfield 1972; Garfield and Merton 1979; MacRoberts and MacRoberts 1989, 1996; Adam 2002; Moed 2006). Citation analysis is one way to evaluate scholarly activity, but often is limited to assessing only productivity or output rather than other dimensions of the research ecosystem, which includes aspects of visibility, reputation, and impact (see Sanchez 2014). This article discusses how Google Scholar (GS) is a useful source of data for urban planning and provides results of a descriptive analysis of citation activity for urban planning faculty in the United States. The following three sections provide a discussion about citation data, types of planning scholarship, and citation analysis. This is followed by a description of the methodology, the results of the analysis, and finally, a discussion with conclusions.

Some argue that citation analysis overemphasizes peer-reviewed publications in evaluating faculty performance. Planning faculty are expected to contribute to professional practice, while universities and colleges do not consistently value these types of research and publication (Wachs 1994). Planning students seek practical training, and planning departments are expected to play a role in connecting directly with local communities through service and outreach (Spain 1992; Wiewel, Carlson, and Friedman 1996). But assessing public service may be even more difficult than traditional scholarship, despite cases where it is adequately documented (Frank 2008; Checkoway 1997). The emphasis on “a productive history of publication in refereed journals” above teaching and service or outreach in planning programs resulting from institutional preferences for planning faculty with PhDs, rather than professional degrees, is seen as contributing to the disconnect between planning research and practice (Krumholz 1986, 63).

Citation Data

This analysis uses Google Scholar as a source of citation data. There are hundreds of articles that discuss the application of Google Scholar to citation analysis, and make comparisons to Elsevier’s Scopus and Thomson Reuter’s Web of Science (WoS; previously, ISI Web of Knowledge). With its release in 2004, one question about GS is whether its coverage of scholarly publications can match that of Scopus or WoS (Yang and Meho 2006; Meho and Yang 2007; Falagas et al. 2008; Li et al. 2010; Harzing 2013). Acknowledging that coverage issues are discipline specific, there are many examples of GS-based citation analyses for particular fields ranging from oncology and condensed matter physics (Bakkalbasi et al. 2006), to business and economics (Levine-Clark and Gil 2008), to health and medical research (Kulkarni et al. 2009). Most comparisons focus on citation counts for small samples of academics while others compare citation rates for journals (see, e.g., Bauer and Bakkalbasi 2005; Jacsó 2005a; Moussa and Touzani 2010). Other meta-analyses are helpful in understanding patterns in bibliometric differences among data sources and analysis methods (see Schroeder 2007 and Franceschet 2010).

There are particular aspects of citation databases that emerge from comparative analyses including the age range of cited publication materials, languages included, types of materials cited, and disciplinary coverage (Mayr and Walter 2007; Shultz 2007; Kousha and Thelwall 2007; Harzing 2013). It is interesting to note that most of the analyses comparing GS with Scopus and WoS concentrate on citation totals and not on the accuracy of these data sources at the publication or author level. In other words, to determine how accurate citation totals are for an individual, the actual list of scholarly products (i.e., those listed in a curriculum vitae) should be compared to the results extracted from the citation databases for an author. This is currently impossible because there are no comprehensive sources of CV data that can be matched to publication records in Scopus, WoS, or GS. Across all of the research on bibliometrics in relation to Scopus, WoS, or GS (and others including PubMed, SciFinder, Microsoft Academic Search), the overall theme is that there is a difference in coverage among these sources, which results in different metrics.

Frequently mentioned is that Google Scholar differs from Scopus and WoS because it includes “non-traditional” publications that are not from scholarly, peer-reviewed sources (Noruzi 2005; Yang and Meho 2006; Walters 2007; Ortega and Aguillo 2014). This is a very significant question in the debate—whether non–peer reviewed or otherwise nontraditional scholarship should be recognized as valid sources for citations. In the case of Google Scholar, much of this qualifies as gray literature, which has been argued to reflect greater reach and impact compared to closed, pay-wall-protected publication and citation data like Elsevier and Thomson Reuters. For disciplines like urban planning, the gray literature produced by faculty is often research-based and reflects scholarly processes, worthy of inclusion in measuring academic reputation and impact (see Kousha and Thelwall 2007; Pomerantz 2013; Harzing and Van der Wal 2007). Critics such as Jacsó (2005b, 2008) and Giustini and Boulos (2013) disagree and discount non–peer reviewed publications. However, referring to the trend toward inclusion of non–peer reviewed sources, Cronin remarked that “Pandora’s box has been opened” (Cronin 2014, 16).

Pauly and Stergiou (2005) compared GS and ISI (predecessor of WoS) finding little difference between the two for a range of disciplines using small samples of authors from each. They state that “free access to these data provided by Google Scholar offers an avenue for more transparency in tenure reviews, funding and other science policy issues, as it allows citation counts, and analyses based thereon, to be performed and duplicated by anyone” (2). Along these same lines Harzing and Van der Wal (2007) discuss potential issues with GS because of its inclusion of nontraditional citation materials. Despite its differences, GS is seen as a viable source for citation analysis because results frequently correlate with other sources like Scopus and WoS even though being more inclusive means that GS can lead to higher citation counts. But as Harzing and Van der Wal (2007) state, it is very unlikely that high citation counts occur for academics who are not significant in their field.

Still more evidence of the evolving openness of citation analysis include mentions of scholarly products on the web, including those referencing scholarly publications and nonscholarly publications. In the spirit of open access and evolving citation processes, “webometrics” and “altmetrics” have the potential to assess a broader range of scholarly impact (see, e.g., Kousha and Thelwall 2007; Aguillo 2011). Kousha, Thelwall, and Rezaie (2010) refer to formal and informal online impact, with formal impact being that measured by sources such as GS and Google Books for citations and informal impact being associated with educational impact (e.g., citations in course syllabi posted online), scholarly presentations (conference or seminar presentations), and blog impact. They also conclude that informal online impact is significant and increasing in several disciplines. Along with webometrics, altmetrics is “the creation and study of new metrics based on the social web for analyzing, and informing scholarship” (Priem et al. 2010, 1) to assess scholarly impact, including measures of usage (downloads and views), peer review (expert opinion), citations, and altmetrics (storage, links, bookmarks, conversations). Kousha, Thelwall, and Rezaie (2010), Priem et al. (2010), and Bollen, Rodriguez, and Van de Sompel (2007) make strong cases for usage-based metrics, but do not emphasize the full range of academic outputs such as gray literature that planning academics frequently produce.

Types of Planning Scholarship

Recent trends in bibliometrics suggest that including a wider variety of scholarship is especially applicable to the field of urban planning. This means that measuring scholarly impact should include books, book chapters, and journal articles, and also be extended to gray literature that is produced and consumed by both planning academics and planning practitioners. This includes the rest of the academic footprint such as dissertations, research reports, conference presentations, conference proceedings, and grant-funded research output. Course syllabi are an additional source that are available on the web and often cite not only academic work but also gray literature on planning topics. Other examples of gray literature for planning academics include studio or workshop projects that are posted to the web and often take the form of professional consulting reports.

Some argue that blog posts and mentions will become recognized gray literature while potentially becoming accepted as academic products to be evaluated along with other scholarly artifacts. In their discussion about blogging for untenured professors, Hurt and Yin (2006, 15) mention that blogging represents a form of “pre-scholarship” where the contents may be the kernels of future articles. In the case of planning, the legitimacy of these postings is evidenced by the number of views, shares, and amount of forum discussion that appears from a mix of planning academics and practitioners. The 50 percent of “nonpublishing” planning academics that Stiftel, Rukmana, and Alam (2004a) refer may be producing gray literature that could be a valuable part of planning pedagogy but goes unnoticed by traditional citation analysis and bibliometrics.

Urban Planning Citation Analysis

To illustrate the application of Google Scholar data to a particular discipline, this article provides a descriptive analysis of citation activity for North American urban planning academics based on Association of Collegiate Schools of Planning (ACSP) membership. There were ninety-nine ACSP member schools as of 2014 with member schools including virtually all urban planning degree granting programs in the US and seven non-US programs, most of which are in Canada. ACSP’s sister organization in Canada is the Association of Canadian University Planning Programs (ACUPP), which has eighteen member universities. ACSP member schools are a mix of both accredited and nonaccredited planning programs that grant undergraduate and master’s degrees. ACSP member schools define the population of faculty members that were included in the analysis. For the purposes of this analysis, only tenure-track faculty from each planning program (only faculty members listed as assistant professor, associate professor, and professor) because they represent faculty with a full-time, long-term presence in these programs. Using publication citation data for individual academics means that data can then be aggregated by program and university, as well as by rank, years of service, and other professional characteristics.

While this analysis does not explicitly test any hypotheses, it provides the basis for subsequent questions and analyses about scholarly evaluation in planning. The analysis highlights certain fundamental approaches that go beyond citation counts and provides baseline metrics that will be of interest to planning scholars and administrators. The metrics from the analysis are open to interpretation and debate; however, the analytical process employed can be easily replicated so that trends, instead of just static measurements, can be tested in the future. Given the role that scholarly productivity and impact play in the promotion and tenure review process, the results of the analysis are relevant to ongoing debates about faculty evaluation.

This article performs similar analyses as that of Stiftel, Rukmana, and Alam (2004a) who examined faculty quality of US graduate planning schools. They discussed several performance measures for planning faculty and planning programs using data from ACSP schools, including faculty size, faculty seniority, percentage of faculty publishing, total publications, publication density, publication distribution within schools, total citations, citation density, and citation distribution within schools. Their study adapted methods from the National Research Council, which included other measures of faculty quality, with a focus on faculty scholarship. While similar, the results of the current analyses are not directly comparable to the Stiftel, Rukmana, and Alam study because of significant differences in data sets (WoS vs. GS), the period of publications (five-year vs. whole career), and less comparisons of planning programs and program characteristics.

Since the Stiftel, Rukmana, and Alam article in 2004, there has not been an update on planning faculty, although there is occasional discussion about faculty performance evaluation as well as journal quality and impact indicators (see Anselin, Nasar, and Talen 2011; Zanon 2014). The following analysis is similar in approach to that of Stiftel, Rukmana, and Alam (2004a) and provides an update on faculty quality as assessed by publications and citation activity. However, the two analyses are not directly comparable because they use different data sources (WoS vs. GS) and focus on somewhat different outcome measures. The use of Google Scholar citations is argued to be an improvement and better fit for planning compared to Web of Science because of the shortcomings mentioned by Stiftel, Rukmana, and Alam (2004a, 15). Particular criticisms of the article were related to a lack of understanding about the Web of Science database, what it includes, and how it can be used (see Stiftel, Rukmana, and Alam 2004b).

Methodology

Data for 894 faculty from the ACSP Guide were matched with publication records primarily obtained using Harzing’s Publish or Perish (see http://www.harzing.com/pop.htm). Because of name disambiguation, misspellings, and other inaccuracies, the resulting data from these searches required cleaning to correctly match authors with their publications. The original data set consisted of more than 62,000 records (publications indexed by Google Scholar) associated with approximately 975,100 citations. Automated and manual processes to clean the data for planning faculty reduced these numbers to 25,410 publications (having at least one citation) with 763,559 citations.

This was the second iteration of the analysis, with the first being posted on a blog (see http://tomwsanchez.com/2014-urban-planning-citation-analysis/). Through the Planet listserve, faculty were invited to review their individual results and provide feedback. There were approximately thirty-five questions and comments generally related to citation analysis and specifically about Google Scholar. The results presented here benefited from the corrections to the data provided by approximately 190 faculty. The corrections to the individual analyses did not result in dramatic changes to the overall outcomes with a net change of 3,423 fewer citations (or 0.49 percent). In some cases publications were added and in other cases they were deleted from individual authors. In most cases, this was a function of name similarities where publications were inaccurately matched. In addition, 202 of the faculty maintained Google Scholar Citation profiles that were assumed to be correct and contain only publications they authored or coauthored (see Google Scholar at http://scholar.google.com/ for more information).

The following are the results for seven particular citation metrics based on the analysis of Google Scholar data:

Citations per faculty for planning programs

Citations per years of service

Citations based on school where degree was obtained

Citations and years since degree was obtained

Citations by academic rank

Top twenty-five planning academics

Distribution of citations

The focus of the analysis is on citation activity instead of publication output. While productivity is often considered an important dimension of academic performance, both visibility and impact are the essence of building professional reputation. Finding measures of quality have remained elusive.

Results

Table 1 shows the median citations per faculty member for the top twenty-five planning programs derived from faculty-level citations. The average per faculty citation rates were found to be skewed by high levels of activity from small numbers of faculty, especially for relatively small programs. As shown, only five of the top twenty-five schools (NYU, University of Maryland, University of Louisville, The New School, and SUNY Buffalo) listed do not have substantial skewness for citation activity among faculty (i.e., skewness below 1.0). This means the distribution of citations across these faculties are relatively even and not overly impacted by particular faculty. It is interesting to note that six of the top ten schools shown are also on Planetizen’s 2015 list of Top 10 Graduate Urban Planning Programs. NYU, Tufts, the University of Minnesota, and the University of Maryland are the remaining four not in the Planetizen ranking. Georgia Tech (ranked fifth) and Cornell (ranked seventh) are the only Planetizen Top 10 programs that do not appear on the Top 25 list of citations by program.

Table 1.

Total Citations per Faculty Member, 2014.

			Total	Per Faculty
Rank	School	Faculty	Citations	Median	Mean	Skewness
1	UCLA	16	47,802	1,472.5	2,987.6	3.574
2	Harvard University	15	32,638	1,280.0	2,175.9	2.242
3	UC Berkeley	20	65,031	1,275.0	3,251.6	2.161
4	New York University	7	8,368	1,219.0	1,195.4	0.239
5	USC	16	42,117	1,098.5	2,632.3	2.284
6	Tufts University	8	10,682	1,074.5	1,335.3	1.690
7	University of Minnesota	11	20,866	1,074.0	1,896.9	1.986
8	MIT	27	33,933	929.0	1,256.8	1.934
9	University of Maryland	8	7,904	881.5	988.0	0.950
10	Rutgers University	17	25,198	844.0	1,482.2	1.729
11	UNC	12	14,938	793.5	1,244.8	1.176
12	University of Toronto	8	11,860	771.0	1,482.5	2.757
13	Arizona State University	16	56,312	737.0	3,519.5	3.744
14	Wayne State University	6	11,611	722.5	1,935.2	2.179
15	University of Louisville	6	6,778	681.0	1,129.7	0.759
16	University of British Columbia	12	20,456	583.0	1,704.7	2.337
17	UIUC	18	20,619	509.5	1,145.5	1.832
18	UC Irvine	14	17,760	499.5	1,268.6	3.522
19	Ohio State University	11	8,019	437.0	729.0	2.924
20	The New School for Social Research	9	6,404	420.0	711.6	0.752
21	Columbia University	7	7,753	406.0	1,107.6	1.110
22	University of Wisconsin, Madison	8	4,260	393.0	532.5	1.947
23	University of Massachusetts–Amherst	10	4,902	363.5	490.2	1.486
24	Texas A&M University	18	40,634	362.5	2257.4	3.277
25	University at Buffalo, SUNY	14	5,441	359.5	388.6	−0.221

The number of citations per years of service is used to control for the age of faculty members (see Table 2). Programs with older faculty are expected to have greater numbers of citations by virtue of having more time to publish, and more time for these publications to be cited. The exact year that a faculty member started their academic career is not easily obtained, so the year they obtained their terminal degree (usually a PhD) is used as a proxy. For example, planning programs like Harvard and MIT have higher average years of service (22.4 and 25.2 years, respectively) compared to UNC and Arizona State University (which average 15.1 and 18.2 years, respectively), because of differences in faculty ages. Like the total citation metrics for planning programs, the citations generated per year for individual faculty exhibits substantial skewness. This means that particular faculty citation rates are having an influence on annual citation activities. There were no data available on postdoctoral appointments prior to tenure track positions, leaves of absence, or other time off that could effect the number of years of service.

Table 2.

Citations per Year per Faculty Member, 2014.

		No. of Faculty	Cites/Year per Faculty
Rank	Current School	No. of Faculty	Median	Mean	Skewness
1	UNC	12	80.8	83.7	1.092
2	USC	16	78.8	93.7	1.481
3	UCLA	16	61.0	103.8	3.539
4	University of Minnesota	11	48.8	97.1	1.111
5	University of Maryland	8	48.7	47.8	−0.072
6	Arizona State University	16	47.7	134.0	3.303
7	UC Berkeley	20	42.5	118.0	2.664
8	Columbia University	7	41.9	47.4	0.654
9	Tufts University	8	39.7	61.6	2.432
10	New York University	7	38.7	49.8	2.055
11	Harvard University	15	38.5	93.8	3.531
12	University of British Columbia	12	37.7	76.4	2.874
13	Rutgers University	17	37.3	60.1	1.707
14	UIUC	18	35.8	53.7	1.168
15	University of Toronto	8	34.6	62.8	2.514
16	MIT	27	33.0	54.1	1.671
17	UC Irvine	14	32.9	49.2	3.489
18	University of Louisville	6	31.4	42.3	1.226
19	University of Oregon	5	30.3	37.2	0.283
20	Portland State University	17	29.2	42.2	1.508
21	Wayne State University	6	27.1	55.5	2.051
22	State University of New York at Albany	4	26.8	30.4	0.312
23	Virginia Tech	15	26.7	61.9	2.204
24	The New School for Social Research	9	25.7	31.1	1.143
25	Ohio State University	11	24.4	32.0	1.775

Table 3 shows the top twenty-five universities based on number of citations aggregated by the school where faculty received their degrees. It is interesting to note that the three top-ranked schools, UCSB, University of Chicago, and University of Oregon, do not have doctoral programs in planning, but produce geographers, economists, and political scientists currently on planning faculties elsewhere. This reflects the multidisciplinary nature of urban planning programs. Among the top ten schools, four (UCSB, University of Oregon, Princeton, and University of Minnesota) are less affected by alumni with high levels of citation activity. This is a factor that is apparent in Tables 1 and 2 where small numbers of faculty activity produce the majority of citations among their programs.

Table 3.

Citations per Faculty Member (School of Degree), 2014.

		No. of Faculty	Cites Per Faculty		Skewness
Rank	PhD School	No. of Faculty	Median	Mean	Skewness
1	UCSB	5	3,485.0	2,900.4	0.057
2	University of Chicago	5	1,687.0	3,593.0	1.891
3	University of Oregon	5	1,176.0	1,570.4	0.500
4	Portland State University	8	642.5	1,186.5	1.848
5	UNC	32	551.0	807.4	2.097
6	Princeton University	6	475.5	1,934.5	0.901
7	Northwestern University	6	439.5	1,274.3	1.827
8	MIT	54	416.0	1,662.2	3.349
9	Columbia University	16	392.0	607.7	1.254
10	University of Minnesota	5	373.0	606.0	0.903
11	UC Berkeley	79	348.0	1,318.1	5.192
12	University of Illinois, Chicago	10	313.0	998.8	2.785
13	Georgia Tech	8	298.5	478.5	0.580
14	Cornell University	46	278.0	1,336.7	6.546
15	Rutgers University	26	270.5	657.1	1.279
16	University of British Columbia	8	247.0	306.6	0.961
17	Harvard University	31	233.0	888.8	2.944
18	University of Pennsylvania	35	221.0	1,021.0	2.564
19	Syracuse University	6	220.0	325.7	0.857
20	UC Irvine	10	191.0	643.6	3.037
21	Texas A&M University	12	186.0	585.5	2.309
22	University of Florida	12	185.0	215.8	1.412
23	University of Michigan	27	176.0	387.9	3.590
24	UCLA	44	165.0	1,293.0	4.361
25	University of Wisconsin, Madison	16	161.5	804.8	3.774

Years since degree and current rank were examined (see Figures 1 and 2) and, as would be expected, both are positively correlated with citation counts. The median number of GS citations for assistant professors was 53.0, with 204.5 for associate professors and 718.0 for full professors. As mentioned earlier, the mean numbers of citations for each rank are skewed by outliers at each level, producing significant differences between median and mean values. These differences point to the variation in planning program types, sizes, and composition explored by Stiftel, Rukmana, and Alam (2004a).

Figure 1.

Total citations by years since degree for planning faculty, 2014.

Figure 2.

Mean/median citations within rank for planning faculty, 2014.

The analysis also looked at individual faculty citation counts. The top twenty-five are shown in Table 4 along with their terminal degree. Degrees in Planning (eight of the twenty-five), Geography (four of the twenty-five), and Political Science (four of the twenty-five) made up more than half of the degrees held, followed by Regional Science, Economics, Psychology, and Public Policy/Affairs (two each). Because urban planning is a relatively small and specialized field, these totals are not as high as those in other larger disciplines such as those in the sciences (Harzing 2010). As mentioned earlier the skewness in program level citation output is a function of small numbers of faculty generating high levels of citations compared to their colleagues. The top twenty-five individuals shown in Table 4 account for about 35 percent of citations, from planning programs in the analysis.

Table 4.

Top Twenty-five Cited Planning Faculty, 2014.

Rank	Name	Total Citations	Terminal Degree
1	Luc Anselin	35,470	Regional Science
2	Michael Storper	23,431	Geography
3	James W. Varni	20,209	Psychology
4	AnnaLee Saxenian	17,771	Political Science
5	Robert Cervero	14,987	Urban Planning
6	Harry W. Richardson	12,370	Economics
7	Neil Brenner	11,318	Political Science
8	Martha Feldman	10,472	Political Science
9	John Forester	10,125	City and Regional Planning
10	Reid Ewing	9,386	Urban Planning
11	Lawrence D. Frank	8,954	Urban Design and Planning
12	Michael K. Lindell	8,546	Social/Quantitative Psychology
13	John M. Bryson	8,219	Urban and Regional Planning
14	George Galster	8,006	Economics
15	Michael Dear	7,729	Regional Science
16	Meric Gertler	7,162	Geography
17	Susan Fainstein	6,927	Political Science
18	Nik Theodore	6,414	Public Policy Analysis
19	Alan Murray	6,086	Geography
20	Lawrence Susskind	5,988	Urban Planning
21	John R. Pucher	5,619	Urban Studies and Planning
22	Jennifer Wolch	5,329	Urban Planning
23	Marlon Boarnet	5,321	Public Affairs
24	Geoffrey Hewings	5,316	Economic Geography
25	Rob Shields	5,163	Urban and Regional Studies

Figure 3 shows the concentration of citation activity among relatively small numbers of planning faculty. For all faculty, the top 20 percent produces nearly 80 percent of all citations. For full professors, the top 20 percent produces approximately 70 percent of citations within the group, dropping to 60 percent for associate professors and approximately 70 percent for assistant professors.

Figure 3.

Cumulative citation counts for all planning faculty, 2014.

Discussion

While this analysis is primarily descriptive, the intent is to highlight the uses of citation analysis to evaluate the planning discipline at the program and individual faculty levels. Using Google Scholar data, the citation analysis presented here illustrates an approach for evaluating planning scholarship. First, the results show which planning programs have highly productive and impactful scholars, which can be one factor in rating and comparing programs. Further analysis could look at the relationship of faculty scholarly output and program quality, such as student outcomes and satisfaction. Second, this information can be used at the individual level to assess the level of scholarly faculty contributions. This may be most appropriate for promotion and tenure evaluation so that faculty can be compared to their peers based on rank and years of service as shown in Figures 1 and 2. In addition to the individual faculty characteristics, the school type or ranking can be criteria to identify peers for more accurate comparisons. Further mining of the publication data to extract keywords and topics for each individual can also be used to identify peers based on research areas. Finally, the point about further mining the publication data leads to a much broader discussion and a richer analysis about the influence within urban planning scholarship.

It is important to distinguish between the aggregate snapshot this analysis provides and the individual-level analysis. These scales differ in dynamics and application, even though one draws upon the other (individual to aggregate). This article provides a descriptive analysis and overview of US urban planning scholarship while clarifying the dialogue between the two by providing a set of benchmarks using open and accessible data and methods. The aggregate results reflect on general patterns of planning scholarship appropriate for comparison to other disciplines as well as for trend analyses within planning. The individual-level analyses are well suited to better understand impacts of particular faculty and performance evaluation.

While not unique to planning, the analysis highlights the fact that much of the scholarly output is generated by a relatively small number of faculty. The results show that at all ranks, the top 20 percent produces most of the cited work. This has meaning for planning academics as well as practice. In reflecting on JPER’s twenty-five-year anniversary, Stiftel recounts that one motivation of the journal was to increase planning scholarship (Stiftel 2005). He notes that there has been relative success but that readership outside JPER appears low, and that the impact on planning practice has not been significant. Given that Stiftel reported this more than ten years ago, has anything changed? The results of this analysis suggest that scholarly activity has not increased. However, this question also requires further investigation into whether patterns of research activity between scholarly publication and gray literature have shifted in either direction and whether the scholarly publications by academic planners is reaching a wider audience outside of planning journals.

To better understand research dynamics among planners, more detailed faculty data would allow greater depth of analysis regarding other dimensions such as specializations and collaboration patterns, which may impact research trajectories (Leahey 2006; Maske, Durden, and Gaynor 2003; van Arensbergen, van der Weijden, and Van den Besselaar 2012). It is likely that faculty with administrative roles and other academic responsibilities (department chairs, coordinators, advisors, etc.) sacrifice time that could be spent on research and publication activities. Programs and universities place different levels of emphasis on publication output, especially in cases with higher teaching loads, larger program sizes, and university research classification (e.g., Carnegie Classification). Faculty expectations are different between bachelor’s, master’s, and PhD-granting programs, which impacts levels of effort faculty devote to scholarly output. This can also be reflected in different levels of research support, including internal funding, summer support, course releases, sabbaticals, and student research assistance.

Conclusions

An objective of this article is to revive the discussion around faculty evaluation using citation analysis. The article also argues that Google Scholar is well suited to planning because it is readily available and indexes gray literature, which includes research and professional reports, in addition to standard types of scholarship from refereed sources. Unfortunately, the current data do not reliably classify publications as scholarly versus gray literature to allow for deeper analysis. Based on the citation data and analysis, the results show which planning programs have relatively high levels of scholarly activity, as well as identifying the planning academics generating these citations. Noteworthy in the results is the proportion of faculty that contribute the majority of citations. These results are presented in a traditional form, with many opportunities for further mining of these data remaining. For instance, the bibliographic data for each of the reported citations can illustrate the reach of an article’s impact through its citations in academic journals. Further examination of these data can shed light on patterns of collaborations among coauthors, their universities, departments, and disciplines. Mapping these attributes to show connections and resulting clusters of research activity can also help to estimate scholarly reach and impact. Co-citation across disciplines can extend reach, understanding, and knowledge that typically remains isolated among academic cliques.

This article argues that citation analysis is a valuable aspect of faculty performance evaluation. As noted by others, teaching and service are inherent to not only most academic positions but also to the planning discipline and profession. Different types of universities and planning programs weight scholarship, teaching, and service in different ways, which makes universal faculty performance criteria nearly impossible (ACSP 1986). It is true that most planning programs are relatively small. This means that administrative responsibilities fall on a greater proportion of faculty compared to larger academic programs. As mentioned earlier, faculty time and effort spent on research reports and nonscholarly activities (i.e., gray literature) may not generate citations at an equivalent pace compared with peer-reviewed publications appearing in journals. This needs to be further explored to better understand research activities and scholarly productivity by urban planning faculty.

A recent commentary on research metrics in Nature outlines important elements for balancing quantitative and qualitative assessment (Hicks et al. 2015). The ten principles of the “Leiden Manifesto” reiterates how “researchers can hold evaluators to account, and evaluators can hold their indicators to account” (430). The principles urge (1) peer/expert review in conjunction with quantitative metrics, (2) metrics that match the mission of the particular institution, (3) acknowledge regionally important, non–English language scholarship, (4) open and transparent metrics, (5) researcher access to data for verification purposes, (6) sensitivity to variation between disciplines, (7) use of multiple metrics, (8) account for lack of precise measurement, (9) anticipate institutional impacts of more widely used measures, and (10) continually update and evaluate metrics. Many of the principles relate to concerns previously stated by planning academics in JPER but serve as a useful distillation of the caution needed. While the suggestions within this article do not address all of the issues encompassed by the manifesto, hopefully the article will contribute to what Hopkins (2001) referred to as “conversation,” which can serve to advance the field of planning, help to maintain relevance, and ask new questions.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Thomas W. Sanchez is a professor of Urban Affairs and Planning at Virginia Tech. He conducts research in the areas of transportation, technology, and the social aspects of planning and policy.

References

Adam

2002. “Citation Analysis: The Counting House.” Nature 415 (6873): 726–29.

Aguillo

I. F.

2011. “Is Google Scholar Useful for Bibliometrics? A Webometric Analysis.” Scientometrics 91 (2): 343–51.

Anselin

Nasar

J. L.

Talen

2011. “Where Do Planners belong? Assessing the Relationship between Planning and Design in American Universities.” Journal of Planning Education and Research 31 (2): 196–207.

Arimoto

2011. “Reaction to Academic Ranking: Knowledge Production, Faculty Productivity from an International Perspective.” In University Rankings: The Changing Academy—The Changing Academic Profession in International Comparative Perspective, edited by Shin

Jung Cheol

Toutkoushian

Robert K.

Teichler

Ulrich

229–58. Netherlands: Springer.

Association of Collegiate Schools of Planning. 1986. “Guidelines on Evaluation of Planning Faculty for Promotion and Tenure.”

Bakkalbasi

Bauer

Glover

Wang

2006. “Three Options for Citation Tracking: Google Scholar, Scopus and Web of Science.” Biomedical Digital Libraries 3 (1): 7.

Bauer

Bakkalbasi

2005. “An Examination of Citation Counts in a New Scholarly Communication Environment.” D-Lib Magazine.

Bollen

Rodriguez

M. A.

Van de Sompel

June 2007. “MESUR: Usage-Based Metrics of Scholarly Impact.” Paper presented at the proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, Vancouver, Canada, 474. New York: ACM.

Bornmann

Daniel

H. D.

2008. “What Do Citation Counts Measure? A Review of Studies on Citing Behavior.” Journal of Documentation 64 (1): 45–80.

10.

Campbell

2011. “Commentary: Bibliometrics—The Science of Measuring Anything and Knowing the Quality of Little.” Journal of Planning Education and Research 31 (1): 101–2.

11.

Checkoway

1997. “Reinventing the Research University for Public Service.” Journal of Planning Literature 11 (3): 307–19.

12.

Cronin

2014. “Scholars and Scripts, Spoors and Scores.” In Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, edited by Cronin

Sugimoto

C. R.

3–21. Cambridge: MIT Press.

13.

Falagas

M. E.

Pitsouni

E. I.

Malietzis

G. A.

Pappas

2008. “Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and Weaknesses.” FASEB Journal 22 (2): 338–42.

14.

Franceschet

2010. “A Comparison of Bibliometric Indicators for Computer Science Scholars and Journals on Web of Science and Google Scholar.” Scientometrics 83 (1): 243–58.

15.

Frank

2008. “Measuring Public Service Assessment and Accountability—To Ourselves and Others.” Journal of Planning Education and Research 27 (4): 499–506.

16.

Freestone

2011. “Commentary: Ranking Planning Journals in the Australian Research Assessment Exercise, 2008-2010.” Journal of Planning Education and Research 31 (1): 88–94.

17.

Garfield

1972. “Citation Analysis as a Tool in Journal Evaluation.” American Association for the Advancement of Science. Science 178 (4060): 471-479.

18.

Garfield

Merton

R. K.

1979. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities, Vol. 8. New York: Wiley.

19.

Giustini

Boulos

M. N. K.

2013. “Google Scholar Is Not Enough to Be Used Alone for Systematic Reviews.” Online Journal of Public Health Informatics 5 (2): 214.

20.

Goldstein

Maier

2010. “The Use and Valuation of Journals in Planning Scholarship: Peer Assessment versus Impact Factors.” Journal of Planning Education and Research 30 (1): 66.

21.

Goldstein

Maier

2011. “Response to ‘Letter to the Editors: Inside Baseball.’” Journal of Planning Education and Research 31 (1): 87–87.

22.

Harzing

A.-W.

2010. The Publish or Perish Book. Melbourne: Tarma Software Research.

23.

Harzing

A.-W.

2013. “A Longitudinal Study of Google Scholar Coverage between 2012 and 2013.” Scientometrics 98 (1): 565–75.

24.

Harzing

A.-W.

Van der Wal

2007. “Google Scholar: The Democratization of Citation Analysis.” Ethics in Science and Environmental Politics 8 (1): 61–73.

25.

Hicks

Wouters

Waltman

de Rijcke

Rafols

2015. “Bibliometrics: The Leiden Manifesto for Research Metrics.” Nature 520 (7548): 429–31.

26.

Hopkins

L. D.

2001. “Planning as Science Engaging Disagreement.” Journal of Planning Education and Research 20 (4): 399–406.

27.

Hurt

Yin

2006. “Blogging While Untenured and Other Extreme Sports.” Washington University Law Review 84:1235.

28.

Jacsó

2005a. “As We May Search—Comparison of Major Features of the Web of Science, Scopus, and Google Scholar Citation-Based and Citation-Enhanced Databases.” Current Science-Bangalore 89 (9): 1537.

29.

Jacsó

2005b. “Google Scholar: The Pros and the Cons.” Online Information Review 29 (2): 208–14.

30.

Jacsó

2008. “Google Scholar Revisited.” Online Information Review 32 (1): 102–14.

31.

Kousha

Thelwall

2007. “Sources of Google Scholar Citations Outside the Science Citation Index: A Comparison between Four Science Disciplines.” Scientometrics 74 (2): 273–94.

32.

Kousha

Thelwall

Rezaie

2010. “Using the Web for Research Evaluation: The Integrated Online Impact Indicator.” Journal of Informetrics 4 (1): 124–35.

33.

Krumholz

1986. “From Planning Practice to Academia.” Journal of Planning Education and Research 6 (1): 60–65.

34.

Kulkarni

A. V.

Aziz

Shams

Busse

J. W.

2009. “Comparisons of Citations in Web of Science, Scopus, and Google Scholar for Articles Published in General Medical Journals.” JAMA 302 (10): 1092–96.

35.

Leahey

2006. “Gender Differences in Productivity Research Specialization as a Missing Link.” Gender & Society 20 (6): 754–80.

36.

Levine-Clark

Gil

E. L.

2008. “A Comparative Citation Analysis of Web of Science, Scopus, and Google Scholar.” Journal of Business & Finance Librarianship 14 (1): 32–46.

37.

Burnham

J. F.

Lemley

Britton

R. M.

2010. “Citation Analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar.” Journal of Electronic Resources in Medical Libraries 7 (3): 196–217.

38.

Linton

J. D.

Tierney

Walsh

S. T.

2011. “Publish or Perish: How Are Research and Reputation Related?” Serials Review 37 (4): 244–57.

39.

MacRoberts

M. H.

MacRoberts

B. R.

1989. “Problems of Citation Analysis: A Critical Review.” Journal of the American Society for Information Science 40 (5): 342–49.

40.

MacRoberts

M. H.

MacRoberts

B. R.

1996. “Problems of Citation Analysis.” Scientometrics 36 (3): 435–44.

41.

Marchionini

2010. “Information Concepts: From Books to Cyberspace Identities.” Synthesis Lectures on Information Concepts, Retrieval, and Services 2 (1): 1–105.

42.

Maske

K. L.

Durden

G. C.

Gaynor

P. E.

2003. “Determinants of Scholarly Productivity among Male and Female Economists.” Economic Inquiry 41 (4): 555–64.

43.

Mayr

Walter

A. K.

2007. “An Exploratory Study of Google Scholar.” Online Information Review 31 (6): 814–30.

44.

Meho

L. I.

Yang

2007. “Impact of Data Sources on Citation Counts and Rankings of LIS Faculty: Web of Science versus Scopus and Google Scholar.” Journal of the American Society for Information Science and Technology 58 (13): 2105–25.

45.

Moed

H. F.

2006. Citation Analysis in Research Evaluation, Vol. 9. Dordrecht: Springer.

46.

Moussa

Touzani

2010. “Ranking Marketing Journals Using the Google Scholar-Based hg-Index.” Journal of Informetrics 4 (1): 107–17.

47.

Noruzi

2005. “Google Scholar: The New Generation of Citation Indexes.” Libri 55 (4): 170–80.

48.

Ortega

J. L.

Aguillo

I. F.

2014. “Microsoft Academic Search and Google Scholar Citations: Comparative Analysis of Author Profiles.” Journal of the Association for Information Science and Technology 65 (6): 1149–56.

49.

Pauly

Stergiou

K. I.

2005. “Equivalence of Results from Two Citation Analyses: Thomson ISI’s Citation Index and Google’s Scholar Service.” Ethics in Science and Environmental Politics 2005:33–35.

50.

Pomerantz

2013. “Google Scholar and 100 Percent Availability of Information.” Information Technology and Libraries 25 (2): 52–56.

51.

Priem

Taraborelli

Groth

Neylon

2010. “Altmetrics: A Manifesto.” October 26, 2010. http://altmetrics.org/manifesto.

52.

Salet

Boer

2011. “Commentary: Comparing the Use and Valuation of Journals between US and European Planning Scholars.” Journal of Planning Education and Research 31 (1): 95–97.

53.

Sanchez

T. W.

2014. “Academic Visibility and the Webometric Future.” Journal of the World Universities Forum 6 (2): 37–52.

54.

Sawicki

2011. “Letter to the Editors: Inside Baseball.” Journal of Planning Education and Research 31 (1): 86–86.

55.

Schroeder

2007. “Pointing Users toward Citation Searching: Using Google Scholar and Web of Science.” Portal: Libraries and the Academy 7 (2): 243–48.

56.

Shultz

2007. “Comparing Test Searches in PubMed and Google Scholar.” Journal of the Medical Library Association 95 (4): 442–45.

57.

Spain

1992. “Creating and Defending Links between Teaching, Research, and Public Service.” Journal of Planning Education and Research 12 (1) 77–79.

58.

Stevens

1990. “An Alliance Confirmed Planning Literature and the Social Sciences.” Journal of the American Planning Association 56 (3): 341–49.

59.

Stiftel

2005. “Supply Side Planning Scholarship.” Journal of Planning Education and Research 25 (1): 7–8.

60.

Stiftel

2011. “Commentary: Of What Meaning—Journal Prominence?” Journal of Planning Education and Research 31 (1): 103–4.

61.

Stiftel

Rukmana

Alam

2004a. “A National Research Council-Style Study.” Journal of Planning Education and Research 24 (1): 6–22.

62.

Stiftel

Rukmana

Alam

2004b. “A National Research Council-Style Study—Response.” Journal of Planning Education and Research 24 (2): 128–30.

63.

van Arensbergen

van der Weijden

Van den Besselaar

2012. “Gender Differences in Scientific Productivity: A Persisting Phenomenon?” Scientometrics 93 (3): 857–68.

64.

Yang

Meho

L. I.

2006. “Citation Analysis: A Comparison of Google Scholar, Scopus, and Web of Science.” Proceedings of the American Society for Information Science and Technology 43 (1): 1–15.

65.

Wachs

1994. “The Case for Practitioner Faculty.” Journal of Planning Education and Research 13 (4): 290–96.

66.

Walters

W. H.

2007. “Google Scholar Coverage of a Multidisciplinary Field.” Information Processing & Management 43 (4): 1121–32.

67.

Webster

2011. “Commentary: On the Differentiated Demand for Planning Journals.” Journal of Planning Education and Research 31 (1): 98–100.

68.

Wiewel

Carlson

Friedman

1996. “Planning the New Urban University: The Role of Planning Departments.” Journal of Planning Education and Research 16 (2): 127–35.

69.

Zanon

2014. “Research Quality Assessment and Planning Journals.” Italian Journal of Planning Practice II (2): 96–123.